Computation and Language · Cross-listed from cs.IR
Evaluating Factual Density in Multi-Source RAG: A Study in Medical AI Accuracy
Michael R. DeMarco
Retrieval-Augmented Generation (RAG) is the current industry standard for grounding AI in real-world facts. Traditional
retrieval methods rely on keyword matching and topic proximity, ranking content based on how closely it sounds like the user's query. What they do not measure is how many verified facts the content actually contains. This structural gap, termed the Expert Blindness Effect, causes standard RAG pipelines to consistently bury high-density factual evidence in favor of lexically dominant text on the same topic. To address this gap, this paper introduces Factual Density (FD*), a novel
retrieval optimization signal that measures the proportion of verified atomic claims relative to total
token count. Using the NexusAgentics Ghost Audit preprocessing pipeline, raw text is scored for factual specificity using probabilistic factuality analysis to filter content before corpus ingestion. An initial formulation introduced a severe document-length confound (Pearson R = -0.8636, p = 2.27e-07). Implementing Z-score normalization within length bins resolved this bias, validating FD* as a length-independent density signal (p = 0.0749). Evaluated against the HealthFC
benchmark (750 health claims labeled Supported, Refuted, or No Evidence by medical experts), FD*-optimized
retrieval was the only condition to achieve 100% systematic review saturation in top-5 results, surfacing Cochrane evidence that standard cosine similarity ranked outside the top ten. Ground truth verification confirmed 25 mappings across seven HealthFC-supported claims. While full statistical validation across n=50 queries remains future work due to constraints on corpus-
benchmark alignment, these findings establish factual density reranking as a low-cost, high-impact intervention for improving factual precision in health RAG architectures.
Read the full abstract ▾
Computation and Language
Consolidating Rewarded Perturbations for LLM Post-Training
Zheyu Zhang, Shuo Yang and Gjergji Kasneci
Post-training of
language models is commonly framed as a sample-score-update loop implemented by gradient descent. A recent line of work, exemplified by RandOpt, relocates this loop to weight space, sampling Gaussian perturbations around a pretrained model and ensembling the top-K rewarded specialists at inference. While competitive with PPO and GRPO under matched training compute, this prediction-level ensemble incurs K forward passes per test example and does not extend cleanly to free-form generation. We ask whether the rewarded population can instead be folded into a single deployable model, replacing the inference-time ensemble with one consolidated update. A split-half analysis over 25 model-task pairs reveals reproducible low-rank structure in every case. We turn this geometry into CoRP (Consolidating Rewarded Perturbations), a gradient-free operator that combines reward-weighted aggregation, compatibility-aware reweighting, and a held-out validation gate, with no gradient flowing through the
language model. Across five
language models from 0.5B to 8B and five tasks covering math, code, and creative writing, CoRP improves the base model by 8.1 points on average. Using one tenth of RandOpt's perturbation budget, CoRP exceeds single-inference RandOpt by 6.5 points and recovers more than half of the gain of the 50-pass majority-vote ensemble, at one forward pass per test example.
Read the full abstract ▾
Computation and Language
Are Full Rollouts Necessary for On-Policy Distillation?
Yaocheng Zhang, Jiajun Chai, Yuqian Fu et al. (+7 more)
On-policy
distillation (OPD) provides dense teacher feedback along student-generated rollouts rather than fixed teacher traces and has emerged as a promising post-training paradigm. However, standard OPD typically generates full rollouts during training, which is computationally expensive and may expose the student to unreliable teacher feedback at late rollout positions, especially during early training. We identify the rollout horizon as a key bottleneck in OPD that substantially impacts training efficiency. Unlike Reinforcement Learning with Verifiable Rewards (RLVR), OPD does not require a final answer reward to provide learning signals. Therefore, full rollouts may not always be necessary for OPD. Motivated by this insight, we propose two simple horizon-control strategies: Progressive OPD (POPD), which gradually expands the rollout horizon during training, and Truncated OPD (TOPD), which permanently performs
distillation on reliable truncated rollouts. Experiments on mathematical
reasoning show that POPD improves the training efficiency of OPD by up to 3$\times$, while TOPD matches OPD performance using only 10% of the rollout horizon, leading to substantial wall-clock and
memory reductions. These results demonstrate that controlling the rollout horizon offers a simple and practical path to more efficient OPD.
Read the full abstract ▾
All authors Yaocheng Zhang, Jiajun Chai, Yuqian Fu, Songjun Tu, Xiaohan Wang, Wei Lin, Guojun Yin, Qichao Zhang, Yuanheng Zhu, Dongbin Zhao
Computation and Language
BenHalluEval: A Multi-Task Hallucination Evaluation Framework for Large Language Models on Bengali
Shefayat E Shams Adib, Ahmed Alfey Sani, Ekramul Alam Esham et al. (+3 more)
Despite Bengali being the sixth most spoken
language in the world, no prior work has systematically evaluated
hallucination in large
language models (LLMs) for Bengali. We introduce BenHalluEval, a fine-grained
hallucination evaluation framework for Bengali covering four tasks: Generative Question Answering (GQA), Bangla-English Code-Mixed QA, Summarization, and
Reasoning. We construct 12,000 hallucinated candidates using GPT-5.4 across twelve task-specific
hallucination types, drawn from three existing Bengali
datasets, and evaluate seven LLMs spanning
reasoning-oriented,
multilingual, and Bengali-centric categories under a dual-track protocol that independently measures false-positive rate on ground-truth instances (Track A) and
hallucination detection rate on hallucinated candidates (Track B). To jointly penalise both failure modes and prevent inflated scores from uniform response bias, we propose BenHalluScore, a dual-track calibration metric that ranges from 7.72% to 55.42% across models and tasks, revealing substantial variation in
hallucination calibration. Chain-of-thought prompting, applied as a mitigation strategy, shifts response distributions without consistently improving
hallucination discrimination. BenHalluEval establishes the first dedicated
hallucination benchmark for Bengali and highlights the inadequacy of single-track and prompting-only evaluation approaches for low-resource
language settings. The
dataset and code are available at https://anonymous.4open.science/r/BanglaHalluEval-EB77.
Read the full abstract ▾
All authors Shefayat E Shams Adib, Ahmed Alfey Sani, Ekramul Alam Esham, Ajwad Abrar, Ishmam Tashdeed, Md Taukir Azam Chowdhury
Computation and Language
Language Models Can Resolve Reference Compositionally, But It's Not Their Native Strength: The Case of the Personal Relation Task
Bart Evelo, Meaghan Fowlie and Denis Paperno
Do neural models, such as Large
Language Models, genuinely acquire compositional abilities for interpretation of natural
language? When we talk about semantic interpretation, we can distinguish two complementary aspects: establishing what an expression refers to in the world (which we call the Extensional task) and representing its sense in a structured way (which we call the Intensional task). We evaluate LLMs and humans on both tasks in the setting of the Personal Relation Task (Paperno 2022) in which, given a universe of people and their relationships with each other, one is asked to interpret a noun phrase such as "Amber's parent's friend". Here, for the Intensional task, the answer is the formula "friend(parent(amber))", and for the Extensional task, the person. We find that humans and LLMs show opposite strengths: humans perform better on Extensional than Intensional tasks, and LLMs vice versa. Our methodology brings greater nuance to the understanding of compositional abilities in modern machine learning models. Our results support the notion that the lack of referential grounding in LLM training is a crucial missing component in mimicking human-like
language understanding.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.SE
Knowledge boundary probing and demand-guided intervention for LLM-based power system code generation
Hui Wu, Xiaoyang Wang and Zhong Fan
Large
language models (LLMs) can turn grid-analysis requests into executable programs for power-system simulation, but utilities and research laboratories often require on-premise deployment. In this setting, first-pass failures frequently arise at an API-knowledge boundary, through hallucinated functions, misused parameters, and mishandled result tables. We present PowerCodeBench, a parameterised
benchmark generator released as a frozen 2,000-task suite for pandapower, and a deployment-time workflow that requires no weight updates. Documentation-driven L0-L3 probes produce per-model API profiles for diagnosis, model comparison, documentation allocation, and backend calibration. A query-side demand
estimator selects layered API evidence before generation, while execution feedback routes targeted repair. Across ten open-weight LLMs (1.5B-480B) and four mid-tier APIs, the validation-enabled workflow raises scalar-match accuracy by 32-56 percentage points after up to three repair rounds relative to an unassisted first pass, for every model of at least 7B and every API. Open-weight models in the 70B-120B range reach the four-vendor mid-tier accuracy range under matched no-tool conditions. Selective injection approaches the full-layer reference using 41% of its
prompt tokens. Among model-item pairs passing numerical checks under both workflows, engineering review confirms the requested analysis in 88% of full-workflow outputs versus 66% under plain repair. Round-0 pilots on OpenDSS and PyPSA motivate staged onboarding from broad
retrieval at cold start to calibrated selective injection. Measured throughput, latency, energy, and allocated GPU
memory establish a practical on-premise serving envelope.
Read the full abstract ▾
Computation and Language
Scaling Conversational Hungarian ASR: The BEA-Dialogue+ Corpus
Máté Gedeon, Piroska Zsófia Barta, Péter Mihajlik et al. (+1 more)
Conversational automatic
speech recognition in Hungarian is constrained by the limited amount of publicly available dialogue-style training data. The BEA-Dialogue corpus addresses this need, but its strictly speaker-disjoint train/dev/eval split reduces the usable material to only 85 hours. In this paper, we introduce BEA-Dialogue+, an expanded version of the corpus that relaxes the split criterion for experimenters and dialogue partners while preserving complete separation of the primary speakers. This results in 200 hours of transcribed natural conversations and enables a controlled study of the trade-off between additional training data and speaker overlap across the splits. We evaluate several Whisper- and FastConformer-based models on both corpus versions, including Serialized Output Training (SOT)-based fine-tuning for dialogue transcription. Our results show that the larger corpus is more challenging for models without fine-tuning, whereas SOT-based adaptation yields consistent improvements in WER, CER, cpWER, and cpCER. Overall, BEA-Dialogue+ provides a substantially larger yet still demanding
benchmark for Hungarian dialogue ASR, and a practical resource for training and evaluating dialogue transcription systems.
Read the full abstract ▾
All authors Máté Gedeon, Piroska Zsófia Barta, Péter Mihajlik, Katalin Mády
Computation and Language · Cross-listed from cs.LG
PithTrain: A Compact and Agent-Native MoE Training System
Ruihang Lai, Hao Kang, Haozhan Tang et al. (+6 more)
Mixture-of-Experts (MoE) has become the dominant architecture for frontier
language models. To meet this demand, production frameworks have built optimized MoE training stacks over years of engineering effort. Yet evolving these stacks for new architectures and system optimizations remains expensive. With the rise of AI coding
agents, they could automate parts of training-framework development and accelerate this
evolution. But applying them to these existing frameworks carries hidden costs, invisible to today's throughput-only evaluations. We name this missing dimension
agent-task efficiency (ATE): the cost of using coding
agents to understand, operate, and extend a framework. Grounded in four
agent-native design principles, we build PithTrain, a compact,
agent-native MoE training framework. We further introduce ATE-Bench, covering real-world training-framework tasks. Our evaluation shows PithTrain matches the throughput of production frameworks, and on ATE-Bench, PithTrain enables higher
agent-task efficiency, with up to 62% fewer
Agent Turns and 64% less Active GPU Time.
Read the full abstract ▾
All authors Ruihang Lai, Hao Kang, Haozhan Tang, Akaash R. Parthasarathy, Zichun Yu, Junru Shao, Todd C. Mowry, Chenyan Xiong, Tianqi Chen
Computation and Language · Cross-listed from cs.LG
DRIFT: Decoupled Rollouts and Importance-Weighted Fine-Tuning for Efficient Multi-Turn Optimization
Jian Mu, Tianyi Lin, Chengwei Qin et al. (+2 more)
Large
language models are increasingly deployed in multi-turn interactive settings where users or environments can iteratively provide lightweight feedback. Unfortunately, optimizing such behavior presents a sharp dilemma in practice: online reinforcement learning is able to effectively address multi-turn dynamics but is prohibitively expensive due to the cost of generating full correction trajectories at every update, whereas offline supervised fine-tuning (SFT) is efficient but suffers from distribution shift and behavioral collapse. To this end, we novelly propose DRIFT (Decoupled Rollouts and Importance-Weighted Fine-Tuning), a framework that operationalizes the theoretical insight that the KL-regularized RL objective is equivalent to importance-weighted supervised learning. DRIFT decouples rollout from optimization by sampling offline interaction trajectories from a fixed reference policy, deriving return-based importance weights, and optimizing the policy via weighted SFT on the resulting
dataset. Empirically, we demonstrate that DRIFT matches or exceeds the performance of multi-turn reinforcement learning baselines while maintaining the training efficiency and simplicity of standard supervised fine-tuning. Code is available at https://github.com/2020-qqtcg/DRIFT.
Read the full abstract ▾
All authors Jian Mu, Tianyi Lin, Chengwei Qin, Zhongxiang Dai, Yao Shu
Computation and Language
Translation Analytics for Freelancers II: Benchmarking Local LLMs for Confidential Translation Workflows
Yuri Balashov, Rex VanHorn, Mingxi Xu et al. (+1 more)
Building on our previous work, this paper develops practical, low-barrier methods for freelance translators and smaller
language service providers to evaluate translation technologies using rigorous yet accessible analytic methods. Here we address a high-stakes, specialized need: offline translation for confidentiality-sensitive domains in which
privacy constraints preclude the use of cloud-based engines and commercial LLMs. We expand the Reeve Foundation Trilingual Corpus (RFTC) used in our previous work into a
multilingual corpus (RFMC) by adding sentence-aligned German and Simplified Chinese reference translations. We then
benchmark several locally runnable
language models (via Ollama) across four
language directions on 1000+ sentences selected from this corpus. We use consistent single-
prompt calls without fine-tuning or domain adaptation, comparing local LLM outputs against commercial NMTs (DeepL, Baidu), a frontier LLM (GPT-5.2), and professional-grade local NMT systems (OPUS-CAT, NeuralDesktop, Promt). Automatic evaluation is conducted with MATEO. Results reveal substantial variation in local LLM performance across
language directions and model sizes. The best local LLMs match or surpass local NMT systems and a frontier LLM, though they remain behind top commercial NMTs. These findings underscore the viability of carefully selected local LLM translation for
privacy-constrained professionals and inform future research on model scaling and
multilingual capability.
Read the full abstract ▾
All authors Yuri Balashov, Rex VanHorn, Mingxi Xu, Austin Downes
Computation and Language
Fine-grained Verification via Diagnostic Reasoning Supervision for Aspect Sentiment Triplet Extraction
Wenna Lai, Haoran Xie, Guandong Xu et al. (+2 more)
Aspect Sentiment Triplet Extraction (ASTE) aims to identify aspect terms, opinion terms, and sentiment polarities as structured triplets, providing essential inputs for downstream information system applications such as opinion mining, explainable recommendations, and review summarization. Prior work mainly focuses on end-to-end extraction, while post hoc verification of extracted triplets remains comparatively underexplored. This gap limits the reliability of ASTE systems, since predicted triplets may be locally plausible while being globally invalid. Moreover, candidate invalidity is multi-faceted and candidate usability is inherently graded, motivating a fine-grained verification mechanism that can filter or re-rank outputs from diverse extractors. In this paper, we propose FiVeD, a framework for Fine-grained Verification with Diagnostic
reasoning supervision. Specifically, the verifier is trained with multiple complementary objectives, including validity classification and quality score estimation as primary tasks, with error type classification and rationale generation as auxiliary tasks. We define hierarchical error categories and construct plausible incorrect triplets under semantic and syntactic constraints, and leverage an off-the-shelf LLM with task-specific rubrics to produce quality scores and diagnostic rationales. During inference, the resulting quality scores are used to filter candidate outputs, supporting adjustable precision-recall tradeoffs. Experiments across multiple ASTE baselines demonstrate that FiVeD consistently improves extraction performance by up to 3.53 F1 points as a plug-and-play verification module.
Read the full abstract ▾
All authors Wenna Lai, Haoran Xie, Guandong Xu, Qing Li, S. Joe Qin
Computation and Language · Cross-listed from cs.GT
Used Car Salesbots? Honesty and Credulity of LLMs as Bargaining Agents under Partial Information
Antonio Valerio Miceli-Barone, Vaishak Belle and Shay B. Cohen
In this work we study
agents in simulated bargaining scenarios, where a buyer and a seller communicate through a text channel and attempt to negotiate mutually beneficial trades, under different information regimes (complete information, information asymmetry or mutual uncertainty). We evaluate their performance w.r.t. game-theoretical solutions and further investigate their honesty (their tendency to disclose or withhold information or to mislead and deceive) as well as their credulity (their tendency to trust or distrust information provided by the other
agent). We study zero-shot LLM
agents with simple prompting scaffolding as well as fine-tuned
agents, in order to investigate whether optimising the
agents to maximise financial profits makes them stronger negotiators but also more dishonest and less trusting. We find that off-the-shelf LLMs all substantially deviate from game-theoretical
equilibria, they attempt to lie about their private information but cannot efficiently exploit information asymmetries. Fine-tuning on financial utility makes the
agents stronger at achieving better deals but also more dishonest, highlighting the risks that optimising
agents for a task can have on their safety. We release our code and a
dataset of bargaining scenarios.
Read the full abstract ▾
Computation and Language
SCOPE: Self-Play via Co-Evolving Policies for Open-Ended Tasks
Wai-Chung Kwan, Aryo Pradipta Gema, Joshua Ong Jun Leang et al. (+1 more)
Self-play can train
language models without external supervision. However, existing methods require rule-checkable answers, leaving open-ended tasks dependent on curated
prompts or frontier-model judges. We introduce SCOPE, a data-free self-play framework for open-ended tasks that co-evolves two policies: a Challenger that generates document-grounded tasks, and a Solver that answers them through multi-turn
retrieval. A frozen copy of the initial model serves as the self-judge, which writes task-specific rubrics from the source document and grades Solver responses against them. Across three 7-8B instruction-tuned models (Qwen2.5, Qwen3, OLMo-3), SCOPE improves open-ended performance by up to +10.4 points on eight
benchmarks and matches or exceeds GRPO_data trained on ~9K curated
prompts. Although trained only on open-ended tasks, SCOPE also improves held-out short-form QA by up to +13.8 points on seven held-out
benchmarks, surpassing GRPO_data on all three models. Ablations show that co-evolving the Challenger is necessary to keep tasks near the Solver's frontier, that gains arise from improvements in both
retrieval and synthesis with the relative contribution varying by task, and that rubric generation quality is the bottleneck for self-judging.
Read the full abstract ▾
All authors Wai-Chung Kwan, Aryo Pradipta Gema, Joshua Ong Jun Leang, Pasquale Minervini
Computation and Language
DOA: Training-Free Decoder-Only Attention Policy for Long-Form Simultaneous Translation with SpeechLLMs
Sara Papi and Luisa Bentivogli
Simultaneous
speech-to-text translation (SimulST) generates translations while
speech is still unfolding, requiring a streaming policy that decides when to read and when to write. State-of-the-art approaches rely on
attention-based encoder-decoder models where cross-
attention provides explicit
alignment signals. In contrast,
Speech Large
Language Models (SpeechLLMs) are decoder-only architectures relying solely on self-
attention. This raises a central question: whether decoder self-
attention contains sufficiently stable
alignment signals to guide the streaming policy. Moreover, existing approaches typically rely on training-based adaptations or heuristic wait-$k$ policies and have not been validated in long-form settings. To fill these gaps, we propose Decoder-Only
Attention (DOA), a training-free policy that enables long-form simultaneous translation with off-the-shelf SpeechLLMs by deriving a proxy
alignment from self-
attention. Experiments on Phi4-
Multimodal and Qwen3-Omni show that DOA provides an effective
alignment signal for supporting streaming decisions, enabling low-latency long-form SimulST with quality close to offline decoding without retraining.
Read the full abstract ▾
Computation and Language
Neuro-symbolic Syntactic Parsing: Shaping a Neural Network with the CYK Algorithm
Fabio Massimo Zanzotto, Federico Ranaldi and Giorgio Satta
In this paper, we show the possibility of a direct injection of
algorithms into neural network architecture. We focus on a complex
algorithm, that is, Cocke-Youger-Kasami (CYK) for parsing context-free grammars in Chomsky Normal Form and we propose CYKNN, a simple recurrent neural network architecture for encoding the CYK
algorithm in trainable
matrix-vector multiplications.We experimented with a very simple grammar with 4 variations showing that our approach outperforms existing LLMs with more than 20B parameters with an in-context learning setting and smaller LLMs of the Qwen family fine-tuned with LoRA. Our attempt paves the way to a different approach to neuro-symbolic methodologies.
Read the full abstract ▾
Computation and Language
Skill Availability and Presentation Granularity in Large-Language-Model Agents: A Controlled SkillsBench Study
Xiaonan Xu and Wenjing Wu
Skill documents provide procedural knowledge to large-
language-model
agents at inference time. This article studies whether the presentation granularity of controlled skill knowledge changes downstream task success. The experiment uses a pinned SkillsBench version, a 30-task domain-balanced subset validated by official oracle runs, two
reasoning-enabled model configurations, six skill conditions, and five trials per task-condition-model
cell. Skill availability is the clearest empirical signal. Relative to no skill, skill conditions increase task-mean pass rate by 26.7 to 36.0 percentage points for GPT-5.5 and by 18.0 to 26.0 percentage points for DeepSeek V4-Flash. The final data contain 1,800 rows, with 900 rows for each model. The task is the inference unit. Five trials are aggregated within each task-condition-model
cell before paired contrasts are estimated over 30 tasks. The primary presentation contrasts are smaller and uncertain. Low-abstraction guidance differs from high-abstraction guidance by +0.7 percentage points for GPT-5.5 and -6.7 percentage points for DeepSeek V4-Flash, with both 95% bootstrap confidence intervals crossing zero. Adding one worked example to medium-abstraction guidance differs from the no-example variant by +0.7 and +1.3 percentage points. Mean-reward robustness checks preserve the same substantive conclusion. In this controlled subset, skill availability is associated with higher success than no skill, while the tested presentation-granularity changes yield small, uncertain, and model-dependent effects.
Read the full abstract ▾
Computation and Language
The Sword, Shield, and Achilles' Heel: Characterizing the Linguistic Inductive Bias of Large Language Models for Spatial Reasoning in Navigation Planning
Xudong Zhang, Jian Yang, Shengkai Wang et al. (+5 more)
Large
Language Model (LLM)-based
navigation systems commonly construct explicit spatial representations (e.g., topological graphs, semantic raster maps) and translate them into textual descriptions as LLMs' inputs. However, the linguistic structures of such text-based spatial representations and the choices of contextual features (e.g.,
topology, geometry) they contain are often treated as neutral engineering decisions rather than key factors that shape LLMs' behavior. To fill the gap, we propose a dual-interventional framework that disentangles linguistic structures from different contextual cues to evaluate the linguistic inductive bias of LLMs for
navigation planning. In the framework, representation intervention varies the linguistic format and the degree of linguistic compression, clarifying when linguistic representations support or inhibit
navigation planning. Context intervention, combined with contextual feature combination and conflict probing, explicitly clarifies the preferences and weaknesses of LLMs when processing different contextual cues. Experiments across diverse spatial
reasoning tasks and multiple model scales reveal a consistent pattern: topological information is a sturdy shield and the backbone of robust planning; linguistic format is a double-edged sword whose effect depends on model size, task demands, and the compression level; and semantic information is a fatal Achilles' heel – incorrect semantic cues can systematically derail the planning process. Overall, our study shows that effective text-based spatial representations in LLM-based
navigation should preserve topological integrity, calibrate representational compression to model capacity, and ensure semantic correctness, rather than simply adopting a single representation. Our code is publicly available at https://github.com/jonesdong150/LLM-
Navigation-Inductive-Bias.
Read the full abstract ▾
All authors Xudong Zhang, Jian Yang, Shengkai Wang, Jiangpeng Tian, Shaowen Chen, Xian Wei, Ke Li, Xiong You
Computation and Language
"Înţelegi Româneşte?” A Recipe for Romanian Vision-Language Models
Mihai Masala, Marius Leordeanu, Mihai Dascalu et al. (+1 more)
Vision-
Language Models (VLMs) largely follow the text-only LLM trajectory, excelling on English
benchmarks but sharply degrading on low-resource
languages, where neither large-scale image-text corpora nor culturally grounded evaluations exist. We present a systematic study of building a
language-specific VLM for Romanian, covering the full pipeline from data construction to architectural choices. We translate established English VLM training and evaluation corpora into Romanian, applying machine translation to textual annotations and to in-image text, preserving visual grounding while adapting the textual content. Using this data, we train and ablate a series of VLMs to isolate the contribution of (i)
vision backbones of varying scale and pretraining, (ii)
language backbones from
multilingual to Romanian-adapted LLMs, and (iii) OCR-style image-text data. We further curate HoraVQA, a culturally native evaluation set grounded in Romanian everyday scenes. Romanian-adapted VLMs consistently outperform their same-sized counterparts and, across all evaluated
benchmarks, even surpass models from the next larger size category.
Read the full abstract ▾
All authors Mihai Masala, Marius Leordeanu, Mihai Dascalu, Traian Rebedea
Computation and Language
Target-Side Paraphrase Augmentation for Sign Language Translation with Large Language Models
Pedro Dal Bianco, Jean Paul Nunes Reinhold, Oscar Stanchi et al. (+3 more)
Sign
language translation (SLT) remains constrained by the limited availability of paired sign-
video/text corpora and by the heavy-tailed vocabularies typical of real-world
datasets. We study a target-side augmentation strategy in which a large
language model (LLM) generates controlled paraphrase variants of the reference spoken-
language sentence while the sign input remains unchanged. Concretely, we use GPT-4o to produce semantically faithful variants of the training targets and train a Signformer-style pose-based
Transformer under a two-stage schedule: pre-training on the augmented corpus followed by fine-tuning on the original references. We evaluate this strategy on three
datasets that span complementary challenges: PHOENIX14T (German Sign
Language), a real-world corpus with moderate lexical diversity; the Greek Sign
Language Dataset with highly controlled, repetitive recordings; and LSA-T (Argentinian Sign
Language), a naturalistic corpus with a large vocabulary and severe long-tail sparsity. This range allows us to characterize precisely when and why target-side augmentation is beneficial. On PHOENIX14T, augmentation improves BLEU-4 from 9.56 to 10.33, demonstrating that paraphrastic exposure helps the decoder generalize beyond memorized reference phrasing. The near-saturated GSL baseline and the extremely sparse LSA-T setting reveal the limits of the approach: in both cases, single-reference lexical overlap metrics are insufficient to capture the full picture, motivating a complementary semantic evaluation. To our knowledge, this is the first study to examine LLM-generated target-side paraphrases as an augmentation mechanism for SLT, and the first to apply an LLM-as-a-Judge evaluation protocol to SLT. This complementary evaluation reveals gains in semantic fidelity that lexical overlap metrics understate.
Read the full abstract ▾
All authors Pedro Dal Bianco, Jean Paul Nunes Reinhold, Oscar Stanchi, Facundo Quiroga, Franco Ronchetti, Ulisses Brisolara Corrêa
Computation and Language
Multi-Turn Multi-Agent Dialogue for Collaborative Reconstruction Improves VLM Performance on Spatial Reasoning, But Only Barely
Chalamalasetti Kranti, Sherzod Hakimov and David Schlangen
Robots operating in diverse environments rely on visual input to interpret objects and spatial layouts. In human-collaborative tasks, they are expected to communicate this understanding through
language.
Vision-
language models (VLMs) support robotic tasks involving visual interpretation, question answering, and instruction following, but their capabilities in collaborative dialogue tasks requiring spatial
reasoning remain underexplored. We study this gap through a collaborative structure-building task that combines visual interpretation, grounding,
language-guided interaction, and action generation. We develop a framework in which VLMs use dialogue to reconstruct a target structure from visual and textual inputs. We evaluate open-weight and closed VLMs across interaction settings, input modalities, and image representations. Results show that spatial
reasoning over visual representations remains difficult for the evaluated VLMs. Detailed text representations of the target yield higher reconstruction success across modality conditions, while decomposed image representations improve performance. These findings reveal limits in visual spatial grounding and grounded instruction generation for collaborative VLM
agents.
Read the full abstract ▾
Computation and Language
LLM Judges Inconsistently Disagree Across Safety Criteria and Harm Categories
Krishnapriya Vishnubhotla, Sowmya Vajjala, Akriti Vij et al. (+1 more)
We evaluate the consistency of automated judges in conducting a multi-dimensional safety evaluation in a reference-free setup. Our results indicate that Large
Language Models are unreliable judges in identifying safety issues related to machine-generated advice in regulated domains such as finance, although they are more reliable at identifying more overt forms of unsafe/harmful content such as violence. The degree of inconsistency in a model's judgments can vary significantly by the chosen safety criteria and can be impacted by the
language of the content and its linguistic style as well. Finally, there is high disagreement among different judges for the same output, across domains, safety criteria, and
languages. These findings provide new insights on the practice of using LLMs as evaluators and offer several recommendations for practitioners on how to use automated judges in practical scenarios.
Read the full abstract ▾
All authors Krishnapriya Vishnubhotla, Sowmya Vajjala, Akriti Vij, Isar Nejadgholi
Computation and Language
Unlocking Fine-Grained Translation Quality Estimation in LRMs through Mutually Boosting Implicit and Explicit Reasoning
Renfei Dang, Xinye Wang, Zhejian Lai et al. (+5 more)
Large
Reasoning Models (LRMs) still struggle with fine-grained translation quality estimation (QE), even with long
reasoning chains. We argue that LRMs already possess strong
multilingual capabilities, while the core challenge stems from the intrinsic difficulty of learning the fine-grained QE task. In this paper, we propose $\textbf{RIEQE}$ ($\textbf{R}$easoning both $\textbf{I}$mplicitly and $\textbf{E}$xplicitly for $\textbf{QE}$), a simple two-stage training framework that enables the mutually boosting of implicit (layer-wise) and explicit (
token-wise)
reasoning capabilities. To make implicit
reasoning feasible, we first decompose the complex QE task into straightforward subtasks. Based on this, our two-stage approach applies: (1) $\textit{NonThinking-SFT}$, Supervised Fine-Tuning (SFT) without
reasoning chains to directly boost the model's implicit
reasoning tendency and capability; and (2) $\textit{Thinking-RLVR}$, standard Reinforcement Learning with Verifiable Reward (RLVR) to subsequently strengthen explicit
reasoning. On the WMT test sets, RIEQE based on Qwen3-4B-Thinking-2507 surpasses all baselines in explicit
reasoning performance, while its implicit
reasoning capability is also comparable to the best current encoder-based models. We further provide evidence for the mutually boosting between implicit and explicit
reasoning, showing how they benefit each other in a bidirectional manner. Our code is available at https://github.com/NJUNLP/RIEQE.
Read the full abstract ▾
All authors Renfei Dang, Xinye Wang, Zhejian Lai, Weilu Xu, Shimin Tao, Daimeng Wei, Min Zhang, Shujian Huang
Computation and Language · Cross-listed from cs.LG
Trading Complexity for Expressivity Through Structured Generalized Linear Token Mixing
Erwan Fagnou, Paul Caillon, Blaise Delattre et al. (+1 more)
Token mixing layers play a key role in how
language models can learn and generate long-range dependencies. Their efficiency relies on the necessary trade-off between decoding speed and the
memory requirements, along with the cache size. Considering causal generation, this paper explores new trade-offs thanks to a unified framework which separates two crucial features: (i) the direct influence of inputs on outputs in one generation step; (ii) the recurrent propagation of information through past outputs. This framework encompasses major architectures such as
attention and state-space models, but also generalizes the recurrence equations by allowing each state to depend on multiple past states rather than only the immediate predecessor. By introducing structure, we design new recurrence patterns that provably achieve the desired complexity, while providing theoretical insights on their expressivity – trading runtime for expressivity in a principled way. Empirical validation is performed on synthetic tasks, along with
language modeling. Together, these results provide a unified toolkit for the understanding and design of efficient and expressive
token mixers across model families.
Read the full abstract ▾
All authors Erwan Fagnou, Paul Caillon, Blaise Delattre, Alexandre Allauzen
Computation and Language
The Latin Substrate: How Language Models Represent and Mediate Script Choice
Daniil Gurgurov, Alan Saji, Katharina Trinley et al. (+2 more)
Many
languages are written in multiple scripts, requiring large
language models (LLMs) to generate equivalent linguistic content in distinct orthographic forms. While prior work suggests that LLMs route information through shared latent representations, how they internally mediate script variation remains poorly understood. We study this question by first examining per-layer output distributions with the logit lens, which reveals consistent latent romanization during transliteration, and then through representational and mechanistic analyses of script generation. At the representational level, we show that scripts of the same
language become increasingly separable across layers and that a simple linear steering direction can flip a model's output script while largely preserving semantic content. The vector generalizes to writing systems unseen during construction: it flips non-Latin output to Latin, but maps Latin output into varied non-Latin scripts. At the mechanistic level, we localize a small set of late-layer
attention heads that causally mediate script choice. These heads transfer across unrelated
languages and writing systems, suggesting that script routing is implemented by
language-agnostic components. Across both analyses, we observe a consistent directional asymmetry: non-Latin output is produced by a compact, identifiable gate, while Latin-script output emerges from diffuse contributions across the model. Overall, our findings hint that LLMs organize script variation around shared latent representations while exhibiting a privileged substrate toward Latin script.
Read the full abstract ▾
All authors Daniil Gurgurov, Alan Saji, Katharina Trinley, Josef van Genabith, Simon Ostermann
Computation and Language
VIABLE: A Visually Impaired Assistance Benchmark for VLM-as-a-Judge Evaluation
Yi Zhao, Siqi Wang, Zhe Hu et al. (+2 more)
AI-based Visually Impaired Assistance (VIA) remains challenging, largely due to the high cost of human evaluation. The VLM-as-a-Judge paradigm may offer a promising alternative, although it has mostly been studied in general domains. We therefore ask whether such judges can be trusted for VIA tasks. To investigate this question, we introduce VIABLE (Visually Impaired Assistance
Benchmark for VLM-as-a-Judge Evaluation), the first
benchmark for VLM-as-a-Judge evaluation in VIA. VIABLE contains over 300K judgment samples across three scenarios and introduces an Effectiveness–Impartiality–Stability framework with a 12-mode failure taxonomy. Based on VIABLE, our systematic study of seven judges across different model scales shows that existing models are largely unreliable across all evaluation axes. The strongest judge, GPT-5.4, achieves only 52.6% single-failure diagnostic accuracy, yet exhibits the highest self-preference rate at 94.2%; while open-source judges are strongly biased and adversarially fragile. To address these issues, we propose VIA-Judge-
Agent, a model-agnostic inference-time harness that augments judges with visual evidence extraction and a taxonomy-guided workflow. It enables positive improvements in diagnostic accuracy and downstream VIA responses more preferred by BLV users. Data and code are available at: https://github.com/YiyiyiZhao/VIABLE
Read the full abstract ▾
All authors Yi Zhao, Siqi Wang, Zhe Hu, Yushi Li, Jing Li
Computation and Language
FBHM: Functional Benchmarking and Steering of VLMs for Hateful Meme Detection
Paramananda Bhaskar, Naquee Rizwan, Daksh Jogchand et al. (+2 more)
Hateful meme detection remains a formidable challenge for
vision-
language models, as existing
benchmarks are structurally observational - confounding rhetorical hate mechanisms with target community features and preventing causal evaluation of model vulnerabilities. To address this, we introduce FBHM, a systematically curated
benchmark of Functionality Based Hateful Memes constructed along two orthogonal axes: 25 distinct rhetorical functionalities and 10 target communities (5,000 memes total). Benchmarking state-of-the-art VLMs reveals a severe generalization gap: models highly accurate on standard
datasets catastrophically drop to near-random performance on FBHM, proving they exploit
dataset-specific heuristics rather than robust
multimodal reasoning. To efficiently close this gap, we propose LSV (learnable steering vectors), an ultra-low data regime strategy that applies a causal intervention objective on as few as 500 steering samples (50 unique base memes), boosting FBHM performance by ~30 Macro-F1 points while outperforming in-context learning and PEFT without degrading source-domain performance.
Read the full abstract ▾
All authors Paramananda Bhaskar, Naquee Rizwan, Daksh Jogchand, Saurabh Kumar Pandey, Animesh Mukherjee
Computation and Language
Bundesrecht: An Open Library and Corpus for German Statutory Reference Processing
Harshil Darji, Martin Heckelmann, Christina Kratsch et al. (+1 more)
Statutory references are central to legal
language understanding, but are difficult to process automatically, as they appear in compact and variable surface forms, may combine multiple targets, use special abbreviations, and often point to lower-level units. Existing tools for German focus either on parsing references from legal documents or accessing statutory text once citations are explicit. This paper introduces bundesrecht, an open resource for German statutory reference processing, consisting of a software library and a structured corpus of German federal law. The library parses, normalizes, and resolves German statutory references, mapping raw citation strings to structured objects, expanding compact references into canonical forms, and linking them to statutory provisions. The accompanying
dataset preserves the internal hierarchy of statutes from laws to fine-granular subclauses. We evaluate the parser and normalizer on 2,944 annotated German legal references using strict exact-match and micro information extraction metrics. We further evaluate canonical reference deduplication and show that normalized references group real citation surface variants far more reliably than string matching. bundesrecht is the first open resource that covers German statutory reference processing as an end-to-end pipeline, from raw citation string to resolved statutory provision, and is available on PyPI.
Read the full abstract ▾
All authors Harshil Darji, Martin Heckelmann, Christina Kratsch, Gerard de Melo
Computation and Language
Reinforcement Learning Can Amplify Emergent Misalignment from Harmless Rewards
Magnus Jørgenvåg, David Kaczér, Lasse Ruttert et al. (+3 more)
Emergent misalignment (EM) is the surprising tendency of
language models to become broadly misaligned after fine-tuning on narrowly misaligned examples. While EM has been extensively studied in the supervised fine-tuning (SFT) setting, evidence that it also arises from reinforcement learning (RL) is limited to large, closed-source models, leaving the phenomenon expensive to study and difficult to reproduce. We characterize EM from RL in small, off-the-shelf open-weight models along three axes. First, we show that rewarding narrow, overtly misaligned behavior produces substantially higher general-domain misalignment than sample-matched SFT. Second, we show that EM from RL can be induced by reward signals that could plausibly arise naturally, such as unpopular aesthetic preferences or poor rhetorical appeals. Third, we evaluate in-training mitigations developed for SFT-induced EM and find that they broadly transfer, with preventive steering with persona vectors, interleaving safety data and inoculation prompting all performing well.
Read the full abstract ▾
All authors Magnus Jørgenvåg, David Kaczér, Lasse Ruttert, Marvin Gülhan, Lucie Flek, Florian Mai
Computation and Language · Cross-listed from cs.CV
Learning from Fine-Grained Visual Discrepancies: Mitigating Multimodal Hallucinations via In-Context Visual Contrastive Optimization
Haolin Deng, Xin Zou, Zhiwei Jin et al. (+3 more)
Multimodal hallucination remains a persistent challenge for
Vision-
Language Models (VLMs). Standard textual Direct Preference Optimization (DPO) often fails to mitigate it due to a lack of explicit visual supervision. While existing works introduce visual preference DPO by contrasting original images against negative ones, they suffer from a theoretically inconsistent objective caused by partition function mismatches and rely on coarse-grained negatives that could enable shortcut learning. In this work, we propose In-Context Visual Contrastive Optimization (IC-VCO). By placing contrastive images within a shared multi-image context, IC-VCO ensures a mathematically rigorous objective. We further introduce Visual Contrast
Distillation (VCDist), an auxiliary reliability-gated regularizer that encourages consistency between multi-image contrastive training and single-image inference. Finally, we propose a contrastive sample editing strategy that generates hard negatives via precise semantic perturbations. Experiments on five
benchmarks demonstrate IC-VCO's best overall performance and the effectiveness of our sample editing strategy. Code and data are available at https://github.com/OPPO-Mente-Lab/IC-VCO.
Read the full abstract ▾
All authors Haolin Deng, Xin Zou, Zhiwei Jin, Chen Chen, Haonan Lu, Xuming Hu
Computation and Language
Divergence Decoding: Inference-Time Unlearning via Auxiliary Models
Humzah Merchant and Bradford Levy
Large
Language Models (LLMs) frequently memorize sensitive training data thereby creating significant
privacy and copyright risks. Addressing these risks, i.e., removing such knowledge from an existing model checkpoint, has proven challenging as many unlearning methods lead to catastrophic utility loss or are ineffective for complex queries. We introduce Divergence Decoding (DD), a mechanism that uses small auxiliary models to steer the logits of the LLM away from specific data during inference. Training these models is straight forward, i.e., we use standard pre-training and fine-tuning setups. We find the method decisively outperforms state-of-the-art (SOTA) baselines on unlearning
benchmarks across a variety of model and training
dataset scales consistent with DD being an effective and inexpensive solution to unlearning. We then demonstrate that this steered distribution can be trivially distilled back into the base model. Since the method is generally applicable to any probabilistic model, we explore its efficacy outside of text generation and find evidence of generalization to the domain of images.
Read the full abstract ▾