Computation and Language
Evaluating the Semantic Specificity of Representation Steering in Language Models
Zhangdie Yuan and Andreas Vlachos
Localized Representation Steering (LRS) is widely used to correct
reasoning pathologies in large
language models. However, standard
benchmark evaluations can easily be fooled by superficial label overrides, creating a false impression of
reasoning circuit repairs. In this work, we propose Cross-Rule Transfer (CRT), a diagnostic framework that audits representational interventions by evaluating them on rule families where the model is natively competent. Evaluating late-layer LRS for a widespread logical failure, contradiction blindness, reveals that the intervention merely injects a global label bias: applying the steering vector to rules the model already handles correctly (99.6% baseline) degrades performance to 40.4% by forcing false contradiction predictions. We support this diagnosis with four complementary controls (direct logit bias equivalence, control vector label-flipping, cross-model grafting, and early-layer steering checks), providing a rigorous methodology to distinguish genuine
reasoning repairs from superficial label overrides.
Read the full abstract ▾
Computation and Language
AlgoWorlds: Benchmarking Tool Use for Global Optimization in Algorithmic Worlds
Zixiang Xu, Jiaan Wang and Fandong Meng
Tool-use
benchmarks generally evaluate whether an
agent completes a workflow using appropriate tools and valid arguments. However, feasibility alone is insufficient in real-world decision settings such as route planning and fleet dispatch. Individual choices interact through shared constraints and costs, so a feasible solution may still be substantially suboptimal. This raises a harder question: can an
agent turn information gathered through tools into a globally optimal decision? We introduce AlgoWorlds, a
benchmark that transforms formally specified combinatorial optimization problems into partially observed decision environments with verifiable global optima. Each environment contains a hidden instance observed only through task-specific information tools, after which the
agent commits to one structured decision evaluated for feasibility and optimality. AlgoWorlds contains 240 environments covering ten combinatorial optimization families and four workload levels. Family-specific deterministic programs generate the instances, exact
algorithms certify their optima and determine workload levels, and two structurally different tool interfaces present each underlying instance. We evaluate seven leading LLMs, including Claude Opus 4.8 and GPT-5.6 Sol. Achieving global optimality remains highly challenging: although leading models produce feasible decisions in most cases, the best-performing model reaches exact optimality in only 38.61% of cases. Even when
agents collect sufficient information to reconstruct the hidden instance, most failures end in feasible but suboptimal decisions. The challenge therefore extends beyond information acquisition to information integration, global constraint
reasoning, and decision verification. The project homepage is available at https://xzx34.github.io/AlgoWorlds/, and the code is available at https://github.com/xzx34/AlgoWorlds.
Read the full abstract ▾
Computation and Language
Arabic Safety Alignment as Selective Refusal: An Empirical Study of SFT, DPO, and Guard Calibration
Mohamad Zbib and Ammar Mohanna
Arabic large
language models must refuse harmful
prompts without over-refusing benign or sensitive
prompts, yet a single refusal rate hides this trade-off. We evaluate it using benign refusal B and harmful-
prompt refusal H, where H measures refusal rather than harmful compliance. Across five Arabic-capable models and 130 runs on the full human-written AraSafe set, refusal-only supervised fine-tuning (SFT) collapses toward blanket refusal, whereas selected mixed-SFT configurations reach H = 90% to 93% at B = 14% to 23%; four selected configurations exceed the H = 90% target in all three runs, while Fanar does so in two of three. Direct Preference Optimization (DPO) and inference guards change B and H differently across models rather than acting as uniform upgrades. In a blinded 300-response audit, annotator binary-refusal agreement is 89.0% (kappa = 0.78); Qwen3Guard and Aya Expanse 32B reach 88.7% and 91.0% accuracy, respectively, with no conclusive paired difference. Selected SFT raises H on Arabizi for all five models, but none reaches 90%, showing only partial transfer from Modern Standard Arabic. Overall, the results support model-specific operating-point selection: set a deployment target and retain only interventions that improve it.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.AI
BIRD-History: A Benchmark for History-Driven Text-to-SQL with Fine-Grained Knowledge Annotations
Yunfan Zhou, Qiming Shi, Yizhou Yang et al. (+2 more)
While recent Large
Language Model (LLM)-based text-to-SQL systems achieve impressive performance on standard
benchmarks, they struggle when user queries implicitly rely on domain-specific knowledge, such as business logic, data conventions, and analytical practices, that is neither captured by the schema nor explicitly stated in the natural
language question. Historical SQL query logs offer a valuable source of such knowledge, yet existing
benchmarks do not adequately support evaluation of history-driven approaches. To address this gap, we introduce BIRD-History, a
benchmark consisting of 1,393 tasks across 11 databases, designed to evaluate text-to-SQL systems' ability to ground underspecified natural
language questions using historical SQL scripts. Each task is annotated with ground-truth labels specifying which historical queries contain relevant knowledge and which SQL clauses encode it, enabling systematic evaluation of both
retrieval effectiveness and knowledge utilization. Alongside the
benchmark, we propose a plug-in retriever that extracts five types of external knowledge from historical SQL scripts, then retrieves and reranks relevant fragments for query generation. The retriever integrates seamlessly into existing few-shot text-to-SQL pipelines without requiring
prompt modifications. Experiments demonstrate consistent improvements across four text-to-SQL systems, highlighting the value of leveraging historical query logs for handling underspecified queries.
Dataset and code are open-sourced on https://github.com/zjuidg/BIRD-History.
Read the full abstract ▾
All authors Yunfan Zhou, Qiming Shi, Yizhou Yang, Di Weng, Yingcai Wu
Computation and Language
When to Adapt: Conditional Memory Adapters for Retention-Preserving Domain Specialization
Jiayu Hou and Lei Wang
Large
language models deployed in specialized domains must improve in-domain performance without sacrificing general capabilities. Existing parameter-efficient fine-tuning methods are typically always on: their learned perturbations are applied to every input, which can degrade out-of-domain (OOD) performance. We propose Engram Adapter, a framework that repurposes pretraining-time conditional
memory as a post-hoc adapter for frozen LLMs. It uses multi-channel matching over local n-gram patterns with explicit occupancy tracking as a lightweight selectivity prior, making residual injection more likely on in-domain inputs while a learned scalar gate suppresses incoherent OOD
retrievals. We evaluate on Qwen3-4B and Qwen3-8B with AG-News and MedMCQA as adaptation tasks and OOD
benchmarks spanning
reasoning, translation, code generation, and legal
reasoning. Engram Adapter improves in-domain accuracy while preserving 99.4%–100.1% of average OOD performance; on LegalBench it slightly exceeds the frozen base model on average, whereas comparable always-on baselines degrade sharply. Mechanistic analyses show that although OOD activations are non-zero, gate and projection attenuation reduce residuals to approximately 0.08% of hidden-state norm, yielding small KL drift and negligible accuracy change. These results suggest conditional activation is a promising route toward modular, retention-preserving domain specialization over frozen backbones.
Read the full abstract ▾
Computation and Language
StageWell: A Process-Aligned Chinese Corpus for Positive-Psychology Support Dialogue
Yuxiong Wang, Ziwei Lin, Bo Wang et al. (+2 more)
Positive psychology dialogue aims to support emotional distress and positive resource building, requiring models to produce not only empathetic replies but also coherent progression through a multi-turn support process. Existing resources often reduce supervision to turn-level strategies or holistic preference labels, leaving process position, support function, and local repair targets implicit. We introduce StageWell, a process-aligned Chinese corpus for positive psychology dialogue, together with HQS, a structured protocol for data construction and evaluation. StageWell organizes support into a six-stage support process and uses a multi-
agent whole-dialogue rewriting workflow to construct 12,445 SFT instances, 1,849 DPO preference pairs, and a GroundTruth subset of 120 expert-revised dialogues and 977 QA pairs. Guided by HQS, DPO pairs are built as process-localized repairs: flawed model outputs are used as rejected responses, and targeted rewrites under the same context and stage constraint are used as chosen responses. Across four 9B-14B open-source LLMs, this supervision yields robust gains in process control, response quality, and safety. Averaged across models, BERTScore improves by 0.037, Q-Overall increases by 1.32 points, S-exact increases by 0.236, and the H-critical rate decreases by 0.167. These results highlight the value of modeling supportive dialogue as a structured multi-turn support process rather than as single-turn response generation.
Read the full abstract ▾
All authors Yuxiong Wang, Ziwei Lin, Bo Wang, Yu Zhang, Shiguang Ni
Computation and Language
Padārtha: Ontology-Grounded Fine-Grained NER Benchmark for Classical Sanskrit
Sujoy Sarkar, Pretam Ray, Paramhans Shah et al. (+4 more)
Annotation schemas are not neutral. When applied to classical literature, tag sets developed for modern journalistic texts impose source-culture definitions on texts they were never designed to describe. We instead ground a schema in the tradition of the text itself introducing Padārtha, the first ontology-grounded fine-grained Named Entity Recognition (NER)
benchmark for Sanskrit, built on the Mahābhārata epic. Our tag set derives from Nyāya-Vaiśesika, a classical Indian ontological system, yielding 18 fine-grained categories organized under 10 ontological nodes and mapped onto five standard coarse tags, ensuring interoperability with existing
benchmarks. Expert annotators label over 12.6K entries from a scholarly index of named entities, linked to corresponding mentions in the Mahānāma corpus, producing fine-grained annotations for 108,335 entity mentions across 73,632 verses, along with a 5,000-verse expert-verified test set sampled to stress rare mentions. We present the first systematic benchmarking of generative NER against traditional architectures for Sanskrit, finding that fine-tuned generative models perform comparably to task-specific systems. However, all systems show a sharp decline from coarse to fine granularity and struggle with out-of-entity mentions unseen during training. The limitation is not due to data scarcity alone, as fine-tuned models recall unseen entities far worse than seen ones and tend to default to the majority sense under lexical ambiguity.
Read the full abstract ▾
All authors Sujoy Sarkar, Pretam Ray, Paramhans Shah, Manoj Balaji Jagadeeshan, Akash Gairola, Arjuna S R, Pawan Goyal
Computation and Language
All You Need Is Non-Commutative Words
Carla M. Quispe Flores, Stanley Salvatierra and Renan Cabrera
We represent lexical
tokens as
unitary matrices and encode each sentence as their ordered product. The noncommutativity of
matrix product captures word order without positional encodings (PEs). The same
algebra yields several capabilities, including antisymmetric self-
attention with no query, key, or value projections, and parallel composition of variable-length text chunks at a reduced
attention cost. Furthermore, it provides a canonical-coset readout layer that encodes all true
unitary degrees of freedom compactly, while supporting continual learning through nested group extensions that enlarge the operator space with each new task preserving prior representations exactly. Across standard text-classification
benchmarks, the method matches or exceeds bag-of-words baselines. Achieving higher accuracy on IMDB and comparable performance on AG News. Notably, this is accomplished by replacing the conventional $\sim$30,000-dimensional vocabulary space with a dense, 64-parameter real-valued encoding, highlighting the expressive efficiency of our parameterization.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.SE
Super Library Agent: Joint Generation and Maintenance of Multiple Applications Beyond the Single Codebase
Daegyu Sung, Yukyeong Lee, Geon Park et al. (+2 more)
Organizations often develop and maintain portfolios of related applications: independently deployable codebases that share substantial domain logic, interface patterns, or operational conventions. As LLM coding
agents are increasingly used to generate and maintain such software, a naive application-by-application workflow duplicates shared logic across codebases and allows prolonged agentic maintenance to accumulate verbosity, dead code, and structural erosion. We introduce the Super Library
Agent problem, where an
agent sequentially generates a portfolio of N related applications while maintaining a shared Super Library of reusable cross-application components. A minimal sequential scaffold can in principle extract shared code and migrate applications to the evolving library, but in practice suffers from low extraction recall and fragile dependency migration. We address these failures with candidate-guided extraction over code chunk summaries, pre-extraction codebase consolidation, and context-aware migration using extraction traces and call-graph information. Across WebGen-Bench and PaperBench, our method preserves application functionality while significantly reducing redundancy and
token footprint (verbosity,
token length) over zero-shot, and avoiding the structural erosion introduced by naive library construction, with additional reductions in LOC and MDL. Our code is available at https://github.com/sbigstar0310/super-library-
agent.
Read the full abstract ▾
All authors Daegyu Sung, Yukyeong Lee, Geon Park, Yumin Choi, Sung Ju Hwang
Computation and Language
Detecting and Repairing Hallucinations in Retrieval-Augmented Generation
Sai Krishna Reddy Mulakkayala, Niki van Stein and Aske Plaat
Language models increasingly answer questions by consulting retrieved documents rather than
memory alone, a design now common in search assistants and enterprise knowledge tools. Grounding a model in retrieved text reduces unsupported statements but does not eliminate them, and a reader cannot tell a grounded sentence from an invented one. Most research on this problem stops at detection, yet flagging a faulty answer changes nothing for the person reading it, and little is known about which action should follow. Using RAGTruth, a
benchmark whose unsupported passages are annotated by hand, we split each flagged answer into individual factual claims, check each against the retrieved source, and compare leaving the answer untouched with three repair strategies of increasing richness: deleting an unsupported claim, replacing it with source text, and rewriting it. Three
language models from different families judge the 916 repaired answers. Every strategy reduces the proportion of answers judged to contain unsupported content, and all three judges agree on the ordering. Deletion achieves the largest reduction while retaining least of the original answer, at 64.3% of the text, whereas rewriting retains 80.1% and reduces least. Repair is not confined to faulty answers: 83.5% of answers annotated clean are edited too. The strategies occupy different points on a grounding preservation trade-off rather than forming a quality ranking, and choosing between them needs evidence about answer usefulness that automatic metrics cannot supply.
Read the full abstract ▾
Computation and Language
Learning Simple Test-Time Environments for LLM Web Agents
Junxuan Li, Zijun Liu, Ziyi Huang et al. (+4 more)
Large
language model (LLM)
agents have demonstrated remarkable proficiency in manually constructed environments, yet their performance frequently collapses when transitioned to complex real-world settings. Existing research largely attribute this degradation to the compositional generalization gaps in LLMs on combinations of multiple simple, well-structured environments. In this work, we propose that LLM web
agents can learn simple environment observations at test time. Specifically, we introduce trial steps for
agents to decompose a complex environment observation into sub-modules, and implement a label-free learning method, Test-Time Environment Decomposition (TTED), to adapt
agent behaviors with experience during inference. Our empirical evaluations demonstrate the framework's efficacy across both synthetic and realistic
benchmarks, showing (1) experience gains acquired within simpler sub-environments can be effectively composed to improve performance in the full one, and (2) test-time training on sub-environments can significantly enhance the compositional generalization of
agents in real-world web automation tasks. We also provide key insights in the design of the label-free learning
algorithm. As more complex environments are accessed by LLM
agents, we believe learning environment decomposition skills at test time will be critical for robust real-world deployment.
Read the full abstract ▾
All authors Junxuan Li, Zijun Liu, Ziyi Huang, Peng Li, Yuzhou Liu, Ming Yan, Yang Liu
Computation and Language
Cloud and On-Premises Deployment of Uzbek Legal RAG via Targeted Retriever Fine-Tuning
Tatul Danielyan, Mariam Avetisyan and Hrant Davtyan
Deploying large
language models for legal question answering raises challenges that general-purpose leaderboards do not capture, particularly for low-resource
languages and under hard operational constraints. We report on building and operating a
retrieval-augmented (RAG) legal assistant for Uzbek that must run in two regimes: a managed cloud service that maximizes answer quality within a per-
token cost ceiling, and an on-premises deployment for clients whose legal data may not leave their infrastructure, restricting us to open-weight models on limited local hardware under latency constraints. Because no evaluation existed for this setting, we build two domain
benchmarks: a
retrieval benchmark of 178 expert-annotated legal queries with gold provision spans, and an end-to-end
benchmark of 504 expert-curated question–answer pairs scored by an LLM judge whose ratings we validate against human judgments and against an independent-family judge. Applying these
benchmarks under each regime, we find the open-versus-proprietary gap is small and cheaply closed by fine-tuning. Therefore, we train UTE-1, which is a state-of-the-art text embedder among open models for Uzbek. We also demonstrate that closing the performance gap via fine-tuning is both impractical due to the intensive hardware demands of long-context legal Q&A and unnecessary, given that legal acts change frequently. We support this by reporting a negative result from a QLoRA experiment. We distill practical guidance for similar deployments, drawn from a system serving real users in production. We release our
benchmarks, evaluation code and the fine-tuned embedder (UTE-1) \href{https://metric-ai-lab.github.io/Uzbek-Legal-RAG/}{at this https URL} to support future work on low-resource legal NLP.
Read the full abstract ▾
Computation and Language
Modality Fault Lines: Structural Corruptions Reveal Fragile Omni-Modal Reasoning
Zhaolu Kang, Meixin Wu, Yu Xue et al. (+6 more)
Omni-modal large
language models are increasingly evaluated on clean text–
vision–audio inputs, where every channel is present, synchronized, and readily interpretable. Such scores are often taken as evidence of robust cross-modal fusion, but clean evaluation cannot tell whether success depends on stable cross-modal structure or on cues sufficient only in intact inputs. To address this gap, we define a modality fault line: a boundary at which model behavior becomes unstable when a modality remains present and human-interpretable, but its internal evidence structure is perturbed. We introduce SCEval (Structure-Corruption Evaluation) a diagnostic evaluation protocol that keeps the question, answer space, and modality channels fixed while applying controlled structural corruptions to text,
vision, and audio individually and jointly. Built from $273$ human-verified tri-modal examples from Social-IQ, OmniBench, and VALOR, SCEval evaluates $15$ proprietary and open-source omni-modal systems. The results show that structural corruption lowers clean accuracy, text–
vision damage forms the most stable shared fault line, and multi-modal degradation is non-additive rather than a simple function of the number of corrupted modalities. Clean omni-modal accuracy therefore does not establish that a model will remain reliable when cross-modal evidence becomes structurally unreliable.
Read the full abstract ▾
All authors Zhaolu Kang, Meixin Wu, Yu Xue, Yingjie He, Qiming Shi, Lei Wei, Yidi Wang, Richeng Xuan, Zhichao Hu
Computation and Language
SHADOWBENCH: Toward Reliable Automatic Evaluation of Semantic Alignment in Autoformalization
Hojae Han, Jongyoon Kim, Sanghyeok Park et al. (+8 more)
Autoformalization translates informal mathematical
theorems into code for proof assistants such as Lean. A central challenge is that current evaluation metrics can accept type-correct but misaligned statements or reject correct statements written in a different formulation. Inspired by Pass@$k$, we propose SA-Pass (*Semantic
Alignment Pass*), which tests formal statements using auxiliary statements called *shadows* that characterize the intended statement. A generated statement receives full credit only when it compiles, implies each shadow (forward check), and is implied by their conjunction (backward check). We instantiate SA-Pass in ShadowBench, a Lean 4 full autoformalization
benchmark of 178 postgraduate- to research-level problems spanning eight mathematical areas. Claude Code (Opus 4.8) with Numina-Lean-
Agent reaches $61.8\%$ compile rate and $11.2\%$ SA-Pass. Across outputs generated by six agentic configurations, SA-Pass achieves $98.8\%$ binary agreement with expert judgments. An early version of ShadowBench served as the
benchmark for Track 4 of the ICML 2026 AI4Math Challenge.
Read the full abstract ▾
All authors Hojae Han, Jongyoon Kim, Sanghyeok Park, Dongwook Cheon, Yeachan Park, Myeong Jae Jeon, Sunjong Choe, Soonho Kong, Wonseok Hur, Seung-won Hwang, Donghoon Hyeon
Computation and Language
Large Language Models Systematically Favor Popular Options: Evidence and Mitigation Across MCQs
Abdelrahman Abdallah, Mohammed Ali, Bhawna Piryani et al. (+2 more)
Multiple-choice questions (MCQs) are a standard format for evaluating large
language models (LLMs), yet the popularity of answer options can confound evaluation. Modern LLMs systematically prefer popular but incorrect options over less popular correct ones, a vulnerability we call popularity bias. This pattern aligns with confidence miscalibration: model confidence remains high even as accuracy collapses for popular options. To systematically isolate this phenomenon, we introduce PopMCQ, a
benchmark with six controlled strategies that vary option popularity while keeping the correct answer fixed. In our most
adversarial setting, where all distractors are more popular than the correct option, models choose popular but wrong answers 66% of the time. To mitigate this bias, we propose PopDebias, a lightweight inference-time correction that estimates and removes a popularity prior from model predictions. It requires no fine-tuning, is label-free at test time (using only a small calibration split for parameter fitting), and adds negligible computational cost. Experiments on 22 open-source LLMs (0.5B to 32B parameters) show consistent improvements, with accuracy gains up to 54.1 percentage points under strong popularity pressure. The code and data are available https://github.com/DataScienceUIBK/PopMCQ
Read the full abstract ▾
All authors Abdelrahman Abdallah, Mohammed Ali, Bhawna Piryani, Mahmoud Abdalla, Adam Jatowt
Computation and Language · Cross-listed from cs.AI
Validating FKG.in: Soundness Assessment in LLM-Augmented Indian Food Knowledge
Saransh Kumar Gupta, Armaan Shah, Lipika Dey et al. (+2 more)
The online culinary ecosystem is increasingly populated by recipe content generated, modified, or summarized by Large
Language Models (LLMs). While often plausible, such outputs may contain hallucinated ingredients, misrepresented quantities, or culturally implausible combinations, limiting their suitability for downstream applications and knowledge graph construction. In this paper, we present a semi-automated soundness assessment workflow for validating structured recipe data extracted and augmented by LLMs from informal culinary sources. Developed as part of FKG(.in), a knowledge graph of Indian food, the pipeline identifies and addresses common failure modes, including structural inconsistencies, semantic and logical incoherence, and deviations from the source text, through a multi-stage process combining formal grammars, vocabulary-based checks, statistical heuristics, Set
Transformer-based coherence modeling, and
retrieval-based verification. Although evaluated on Indian recipes, the proposed methods are applicable to broader
multilingual and multicultural culinary domains. We provide a practical, auditable, and application-agnostic framework for validating LLM-augmented recipe data, thereby strengthening the foundations of machine-readable food knowledge infrastructures in the era of LLM-generated content.
Read the full abstract ▾
All authors Saransh Kumar Gupta, Armaan Shah, Lipika Dey, Partha Pratim Das, Ramesh Jain
Computation and Language
When Patients Cut In: Extending Clinical Conversational AI Safety to Interruptions
Zachary Ellis, Spencer Hazel, Adam Brandt et al. (+3 more)
Clinical voice
agents are now deployed in routine care, where real patients do not wait their turn: they interrupt. These systems typically use a cascaded architecture (
speech-to-text -> LLM -> text-to-
speech), so when a patient cuts the
agent off mid-utterance, clinically required content can be lost even when the model handles cooperative transcripts well. Yet clinical conversational-AI
benchmarks almost universally assume patients wait for the
agent to finish, missing interruption-induced loss of required content. We present a transcript-based evaluation of interruption recovery, adapting conversation-analytic overlap categories into three operational types (recognitional, competitive, transitional sub-unit) and testing four deployment-oriented, non-
reasoning LLM configurations across four
cells spanning history-taking (information gathering) and FAQ (information provision), scored on whether the
agent preserves the clinically required content. In the gathering
cells, target-question failure varied across models; in the provision
cells, where arms are directly comparable, failure rose for every model. Rankings differ across
cells, and competitive FAQ interruption produced 30/30 provision-coverage failures for all four models (Wilson 95% CI: 88.6-100.0%; baseline 0/30 for three, 4/30 for Llama). A brief apology marker ("sorry to interrupt") shifts recovery by tens of percentage points, inconsistently across models, and for one it reduces recovery. Interruption robustness therefore cannot be a single score: evaluation must be content-grounded, reported per
cell, and matched to the deployment's interruption profile.
Read the full abstract ▾
All authors Zachary Ellis, Spencer Hazel, Adam Brandt, Yajie Vera He, Ernest Lim, Jared Joselowitz
Computation and Language
Anchoring Speech with Semantics: A Multimodal Adapter Mechanism for Automatic Speech Recognition in Low-Resource Languages
Kuan-Tang Huang, Cheng-Yeh Yang, Chien-Chun Wang et al. (+3 more)
Low-resource ASR remains difficult because scarce transcripts provide limited supervised evidence for target-side generation. To address this gap, we propose SAMA-ASR, a lightweight adapter mechanism that augments the decoder with semantic anchors from auxiliary translations and an acoustic anchor from
speech; in principle, the mechanism can be applied to similar encoder–decoder multitask
speech models. Through cross-modal adaptation, SAMA-ASR conditions decoder states on translation-derived semantic
embeddings and a
speech embedding, combining utterance-level meaning with
speech-grounded evidence before
token prediction. At evaluation time, these semantic anchors can be generated automatically by an upstream
speech-to-text translator rather than supplied as oracle translations. Experiments on two 30-hour
datasets covering the low-resource Sinitic varieties Taiwanese Hokkien and Hakka show that SAMA-ASR improves over acoustic, prior
prompt-based, and semantic-only translation-guided baselines and remains effective in practical automatic semantic-anchor settings; translator-capacity analyses show that useful semantic anchors can be produced by a compact ST model.
Read the full abstract ▾
All authors Kuan-Tang Huang, Cheng-Yeh Yang, Chien-Chun Wang, Hung-Shin Lee, Hsin-Min Wang, Berlin Chen
Computation and Language
Attribute-Based Activation Steering of LLMs for Group-Specific Explanation Generation
Leandra Fichtel, Janek Prange and Henning Wachsmuth
To effectively enable people to understand new topics, explanations should be tailored to their backgrounds and abilities. Prompting alone has been shown to be insufficient for creating such explanations, and other computational methods are missing so far. Therefore, this paper investigates whether LLMs can be steered to generate explanations that are tailored to a specific group of people. To this end, we propose an approach that first identifies group-specific attributes in terms of explanatory style and knowledge of a specific target group. Building on activation engineering, it then computes attribute-based steering vectors and adds them to the internal activations of an LLM during inference to enable a fine-grained steering. In our experiments, we assess the steering effectiveness in terms of specificity and factuality of the generated explanations. Additionally, we evaluate the explanations in a study with human experts from different target groups. Compared to prompting and state-of-the-art steering baselines, our approach tailors the explanations significantly better to the target group while maintaining the best specificity-factuality balance.
Read the full abstract ▾
Computation and Language
Toward Cultural Alignment: Human-Centered Evaluation of Multimodal AI Stories Across Five African Communities
Millicent Ochieng, Felermino D. M. A. Ali, Elizabeth A. Ankrah et al. (+8 more)
In this paper, we examine how well AI-generated
multimodal stories align with the lived practices, relationships,
language, values, and visual expectations of the communities they represent. We conduct a community-grounded mixed-methods evaluation with 19 culture representatives across five African communities, combining quantitative annotations with qualitative focus group discussions. We find that cultural
alignment depends not simply on recognizable cultural markers, but on how those markers fit social, linguistic, procedural, and visual context. From these evaluations, we develop a taxonomy of cultural
alignment comprising five broader cultural marker categories and eight recurring mechanisms of misalignment. We additionally evaluate five
multimodal LLM judges to examine whether automated evaluation can approximate community-grounded judgments at scale. Judge reliability and score calibration vary substantially across communities, with no single judge performing consistently across all five settings. These findings motivate community-calibrated evaluation pipelines in which automated judges are validated against community judgments to determine where they can be trusted and where human review remains necessary.
Read the full abstract ▾
All authors Millicent Ochieng, Felermino D. M. A. Ali, Elizabeth A. Ankrah, Najeeb Gambo Abdulhamid, Migisha Boyd, Stephanie Nyairo, Mercy Muchai, Samuel Chege Maina, Aditya Vashistha, Anja Thieme, Jacki O'Neill
Computation and Language · Cross-listed from cs.AI
How Identity and Opinion Shape Political Sycophancy in LLMs
Li-Ni Fu, Chang-Chih Meng, Chien-Hua Chen et al. (+2 more)
As Large
Language Models (LLMs) increasingly encourage users to disclose personal profiles for tailored assistance, measuring their political
alignment becomes increasingly important. However, many existing
benchmarks for assessing political behavior rely on closed-ended questions and do not fully capture how a model's stance may adapt to user-provided context during interaction. We introduce a framework that disentangles two distinct triggers of political sycophancy: opinion (aligning with explicit narratives) and identity (stereotyping based on demographic labels). Using 450 manually-checked political dilemmas as controlled probes, we evaluate 13 instruction-tuned LLMs. We uncover a dissociation: a model's susceptibility to explicit opinions does not necessarily predict its susceptibility to identity cues, and vice versa. When both signals are present, their effects are generally sub-additive rather than simply additive. Additionally, system-level personas primarily shift a model's baseline stance while having limited effect on the stance shift caused by user opinion or identity. Ultimately, our results suggest that LLM political stance is interactively and steerably vulnerable rather than being a fixed trait, highlighting how personalization may amplify identity- or opinion-conditioned shifts in the model's behaviors.
Read the full abstract ▾
All authors Li-Ni Fu, Chang-Chih Meng, Chien-Hua Chen, Hen-Hsen Huang, I-Chen Wu
Computation and Language · Cross-listed from q-bio.NC
Rate-Coding Bundle Memory: A Unified Model of Memory and Control for Symbolic Computation in the Brain
Teun van Gils, Rowan P. Sommers, Markus Ostarek et al. (+1 more)
We propose a neurobiologically plausible model of cognition that combines the advantages of connectionist and symbolic systems, and that can explain a wide range of cognitive phenomena. This model, called Rate-Coding Bundle
Memory (RCBM), is based on the Symbolic Subsystem Hypothesis, which posits that the
brain implements a symbolic subsystem within its fundamentally connectionist nature. RCBM is a hybrid model that uses rate coding to represent symbols in a continuous space, and it uses a bundle
memory system to store and retrieve these symbols. The model is capable of solving a wide range of cognitive phenomena, including one-shot learning, pattern separation, and the binding problem. We argue that RCBM provides a promising framework for understanding the nature of cognition, and that it can be used to develop more sophisticated models of cognition in the future.
Read the full abstract ▾
All authors Teun van Gils, Rowan P. Sommers, Markus Ostarek, Peter Hagoort
Computation and Language · Cross-listed from cs.LG
Locked at the Entrance, Open Inside: Where RLVR Narrows the Solution Space
Qiancheng Zhou and Ruizhe Li
Reinforcement learning with verifiable rewards (RLVR) substantially improves single-sample accuracy (pass@1) but causes the policy's solution space to contract, diminishing the returns of test-time scaling. In this work, we investigate where inside a
reasoning trajectory this breadth is lost: does the policy fail to access a valid solution family, or does it fail to execute computation once initiated? To disentangle access from execution, we analyze the Countdown task, whose solution space can be exhaustively enumerated into discrete entrance families defined by the first operand and operator, across PPO on Qwen2.5-3B and GRPO on Qwen2.5-3B-Instruct. Across both training setups, solution coverage falls by up to 67%, halving even on problems solved across all checkpoints. We show that this contraction is heavily concentrated at the entrance: per-
token likelihood shifts are 11x–16x larger prior to the first arithmetic operation than during downstream
reasoning. Supplying only an unselected entrance prefix restores completion rates in low-access families by over an order of magnitude (0.018 -> 0.212 under PPO), demonstrating that alternative solutions remain executable but are no longer initiated. Guided by this localization, we find that while surface prompting fails to recover diversity, entrance-targeted interventions succeed: late-layer parameter interpolation with early checkpoints increases solution coverage by 37% at no loss in pass@1. Finally, we show that early-step
entropy collapse recurs across six math
benchmarks with 7B and 14B models, but is not an inevitable byproduct of
reasoning optimization: an SFT baseline preserves more than double the coverage, and staged SFT–DPO–RLVR pipelines retain early-step
entropy. In summary,
reasoning breadth is lost at the door, not inside the room. Code: https://github.com/ershiyidian/early-branch-locking.
Read the full abstract ▾
Computation and Language
Toward a Cross-Lingual Romanization Ecosystem for Sinitic Languages: A Paired Mandarin-Cantonese Case Study
Zijie Zhang, Tan Lee, Yong Cao et al. (+1 more)
This paper proposes the Sinitic Romanization Ecosystem, a cross-lingual Sinitic romanization design framework with supporting digital infrastructure and a community-driven open-source workflow. The design framework addresses the lack of systematic cross-lingual romanization
alignment among Sinitic
languages through four design principles: phonetic correspondence for representing similar sounds with similar romanized symbols, historical-phonological correspondence for aligning cognate romanization strings, one-phoneme-one-symbol, and basic Latin-letter use, with a balancing consideration recognizing trade-offs among these principles. For the main paired case study, we develop CantRomZJ1 and MandRomZJ1, Cantonese and Mandarin romanization schemes following the design framework, respectively. We also develop schemes for several other Sinitic
languages, including Meixian Hakka, Shanghai Wu, and Nanjing Jianghuai Mandarin, following the same design framework. To bring the romanization schemes into practical use, we develop open-source infrastructure for structured romanization storage, conversion, parsing, dictionary construction, and input-method generation. Finally, we evaluate the design framework through
speech-to-romanization experiments based on Meta's Massively
Multilingual Speech (MMS) fine-tuning. Compared with the Pinyin+Jyutping baseline, our MandRomZJ1+CantRomZJ1 condition reduces Cantonese WER and CER by 7.80% and 10.61%, respectively. These results suggest that cross-lingual romanization
alignment can improve transfer in low-resource Sinitic
speech technology.
Read the full abstract ▾
All authors Zijie Zhang, Tan Lee, Yong Cao, Benyou Wang
Computation and Language
Quantifying Error Tolerance in Synthetic Data: An Atomic-level Operand vs. Operator Perturbation Study
Jiaxiang Liu, Chenhao Yuan, Shuwen Xu et al. (+9 more)
Synthetic data generation has become a cornerstone for advancing large
language models. However, the lack of the quantitative analysis for error tolerance became a critical bottleneck. Consequently, current filtering strategies fluctuate between two extremes: they are either overly aggressive, risking the exclusion of potentially valuable samples, or overly permissive, failing to eliminate erroneous samples effectively. To bridge this gap, this paper introduces Atomic Tree Operation Modeling (ATOM), a framework that decomposes data into functional units ($f(x)\rightarrow y$). ATOM distinguishes benign Operand $x$ perturbations from fatal Operator $f$ perturbations. The former are needlessly discarded by aggressive filtering, while the latter slip through permissive filtering. Our experiments reveal a double dissociation: models are robust to operand perturbations but collapse under operator perturbations. By prioritizing operator over aggressive operand precision, our ATOM-synthesized data outperforms rigorous baselines (e.g., +3.1% gain over LIMA), suggesting that operator diversity matters more than operand precision. Our code is available at https://github.com/Lut-hub/ATOM.
Read the full abstract ▾
All authors Jiaxiang Liu, Chenhao Yuan, Shuwen Xu, Boxuan Xing, Xiusheng Huang, Yinhao Xu, Hao Liu, Wenhao Teng, Xiangwen Liao, Pengfei Cao, Jun Zhao, Kang Liu
Computation and Language
AI Historian: Helping historians organize and verify person-centred temporal clues from dispersed historical narratives
Yifeng Lu, Zijie Yang, Jie Li et al. (+2 more)
History is not preserved in complete, continuous form. Accounts of a person's activities, relationships and historical contexts are scattered across texts, chapters and narrative perspectives; historians must retrieve, identify and compare these materials to reconstruct temporal sequences and verify them against sources. Here we present AI Historian (AIH), an AI
agent system that helps historians organize person-time evidence from dispersed biographical narratives. It takes source sentences as evidence units, identifies people and temporal cues, verifies candidate cross-text associations and infers comparable temporal ranges while preserving traceable source-text evidence. We evaluated AIH on six Shiji cases concerning Liu Bang, Xiang Yu and Xiao He. AIH
Agent achieved a temporal-localization MicroIoU of 86.2%, compared with 81.3% for human-only annotation and 17.1% for direct large-
language-model prompting; it required about 14 min, versus 1 h 32 min for human-only annotation. We further applied AIH to the Twenty-Four Histories and other ancient Chinese histories, ancient Japanese and Korean histories, and modern and contemporary historical materials, and released the results through Westlake Historian. These results indicate that AIH can reduce the cost of organizing historical materials at scale while turning connections obscured by chapter-based narration into traceable, revisable research questions for collaborative testing.
Read the full abstract ▾
All authors Yifeng Lu, Zijie Yang, Jie Li, Qingkai Min, Yue Zhang
Computation and Language
HEAR Who Said What: Unlocking Speaker-Attributed Reasoning via Counterfactual Voice Grounding
Dongwook Lee, Sangkwon Park, Eunwoo Song et al. (+6 more)
Speech Language Models (SLMs) are increasingly deployed in multi-speaker environments, yet their ability to attribute
speech to the correct speaker and reason over speaker identities remains unclear. Hence, we introduce HEAR, a conceptually hierarchical
benchmark diagnosing the foundational capabilities of speaker-attributed
reasoning, comprising 2.4K human-verified samples from 887 diverse multi-party audio clips. Evaluating 20 leading SLMs on HEAR reveals they struggle with these foundational tasks, often relying on semantic priors rather than actual vocal cues. To address this, we present A2R, a 30B model optimized on Counterfactual Audio with Speaker-level Hard negatives (CASH), a
dataset designed to guide the model to prioritize acoustic vocal cues over linguistic signals. A2R achieves strong performance on HEAR and exhibits zero-shot generalization to diverse multi-speaker downstream tasks, demonstrating that learned speaker attribution unlocks the model's latent capacity for speaker-aware
reasoning. All resources are available at https://attributetoreason.github.io/AttributeToReason/
Read the full abstract ▾
All authors Dongwook Lee, Sangkwon Park, Eunwoo Song, Che Hyun Lee, Youngho Cho, Junho Kim, June Young Yi, Heeseung Kim, Sungroh Yoon
Computation and Language · Cross-listed from cs.AI
Emergent Misalignment Is Not Magical
Mingxuan Li, Qirun Dai, Heran Wang et al. (+1 more)
Fine-tuning large
language models (LLMs) on narrowly harmful
datasets can lead to misalignment broadly, a phenomenon known as emergent misalignment (EM). EM poses a challenge for AI safety and our understanding of LLMs. Prior work often frames EM as an unexpected behavior, and explains it by appealing to general misalignment directions or anthropomorphizing it as acquiring an evil persona. However, the mechanisms behind these framings remain obscure. In this work, we show that EM is a predictable and data-dependent generalization phenomenon. By examining the base model's representation of EM training data and evaluation
prompts, we find that evilness after EM training is highly predictable from representational distance: the closer an evaluation
prompt is to training data centroid, the more evilness it elicits from EM models after training (with an average Spearman correlation of -0.73 across 12 model-
dataset settings). Building upon this analysis, we further demystify EM by showing that (1) its effectiveness changes significantly based on training data format; (2) there is not a general misalignment direction that transfers across different EM models; (3) the effect of EM is fundamentally different from persona changes. Furthermore, we extend the EM generalization metric from a scalar distance to a
dataset-specific generalization direction, which robustly predicts EM models' evilness under semantics-preserving
prompt perturbations including appending random
tokens and paraphrasing, where other methods do not reliably generalize.
Read the full abstract ▾
All authors Mingxuan Li, Qirun Dai, Heran Wang, Chenhao Tan
Computation and Language
Recognition-Refusal Misalignment in LLMs: Why Models Answer Structurally Unanswerable Questions
Yucheng Du and Xiyang Hu
Large
language models often answer structurally unanswerable questions, such as computing cot(-540{\deg}) or evaluating (1).startswith("1"), instead of abstaining. We ask whether this failure reflects missing recognition or failed routing from recognition to abstention. Across instruction-tuned models from 1.7B to 70B parameters, a single linear direction in the hidden state separates answerable from structurally impossible math and code
prompts, showing that models represent impossibility before generation. Yet this recognition direction is nearly orthogonal to the canonical safety-refusal direction that mediates trained harmful-content refusal. An in-domain behavior-defined invalidity-aware direction is closer to recognition, but only partially aligned with it, and remains near-orthogonal to safety refusal. Generation-time steering along the recognition direction changes invalidity-aware behavior bidirectionally and dose-responsively on structural math and code
cells, while random directions do not. Base/instruct comparisons further show that the low-cosine geometry is already present at the pretraining endpoint. The confident-on-impossible failure is therefore better explained as a routing failure than as an encoding failure: the model has a usable "no admissible answer" signal, but the safety-refusal pathway is not aligned to use it.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.CR
A Comprehensive Survey on Linguistic Steganography: Methods, Countermeasures, Evaluation, and Challenges
Ruiyi Yan, Chenhui Chu, Zhongliang Yang et al. (+1 more)
Linguistic steganography hides secret messages in natural
language text. Large
language models (LLMs) have reshaped the field, but a systematic account of how these scattered advances collectively reshape the field in this new era is still missing. We provide one along four axes: 148 steganographic methods, 60 linguistic steganalysis countermeasures, 23 evaluation metrics, and 9 open challenges, each with taxonomies, reviews, and adoption analyses. Cutting across these axes, we identify five specific paradigm shifts in the LLM era: (1) from covertext modification to
prompt-only generation, (2) from heuristic to provable security, (3) from white-box symmetric LMs to black-box or asymmetric access, (4) from security-centric designs to joint optimization, and (5) from text-quality concerns to engineering issues. The survey aims to serve as both a reference and a roadmap for practical and responsible linguistic steganography in the LLM era.
Read the full abstract ▾
All authors Ruiyi Yan, Chenhui Chu, Zhongliang Yang, Yugo Murawaki