Computation and Language
HELEA: Hard-Negative Benchmark and LLM-based Reranking for Robust Entity Alignment
Yoonjin Jang, Junwoo Kim and Youngjoong Ko
Entity
Alignment (EA) is essential for knowledge graph (KG) fusion, but existing
benchmarks often allow models to exploit name overlap rather than relational structure. This makes it difficult to evaluate whether models can reject same-name entities that refer to different real-world objects. Our primary contribution is a same-name hard-negative augmentation strategy that simultaneously yields quality-controlled evaluation
benchmarks (DW-HN29K, DY-HN27K) and augmented training corpora (DW-Train, DY-Train), by mining same-name but distinct entity pairs from KG name-collision groups. We further introduce HELEA, a two-stage framework integrating (i) entity encoder
retrieval trained on hard-negative-augmented training corpora with 1-hop KG context, and (ii) LLM-based reranking without additional training. Experiments show that name-dependent baselines collapse to near-random performance on our hard-negative
benchmarks, while HELEA achieves F1 0.967 on DW-HN29K while maintaining Hit@1 0.993 on standard DW-15K.
Read the full abstract ▾
Computation and Language
RA-MoE: Routing-Aligned Fine-Tuning for Multilingual Adaptation of Mixture-of-Experts Models
Guanzhi Deng, Kuan Wu, Haibo Wang et al. (+6 more)
Mixture-of-Experts (MoE) models enable efficient LLM scaling, yet adapting them to non-English downstream tasks remains challenging. Standard
multilingual fine-tuning largely ignores their heterogeneous routing structure. Across multiple MoE models and tasks, we find strong cross-lingual routing
alignment in middle layers, with routing divergence associated with target-
language performance gaps. Motivated by this observation, we propose RA-MoE (Routing-Aligned MoE Fine-Tuning), a three-stage framework for
multilingual MoE adaptation. RA-MoE categorizes parallel examples into four correctness groups (cc/ci/ic/ii) and identifies task-relevant experts in middle layers. It then selectively aligns target-
language routing on ci examples toward successful English routing patterns, jointly matching the total routing mass assigned to task experts and its relative allocation among them. Experiments across three MoE models, three downstream tasks, and six target
languages show that RA-MoE consistently outperforms standard SFT and strong routing-aware baselines. Further analyses confirm the intended routing changes and reveal that middle-layer task routing is largely shared and transferable across
languages, providing mechanistic evidence for the cross-
language transferability of task-specific routing.
Read the full abstract ▾
All authors Guanzhi Deng, Kuan Wu, Haibo Wang, Shing Yin Wong, Sichun Luo, Zengyan Liu, Fang Xu, Zhuoran Wang, Linqi Song
Computation and Language
Revisiting Anthropomorphic Reflection Markers in Large Language Model Reasoning
Yahan Yu, Noa Nakanishi and Fei Cheng
Large
Language Models (LLMs) often produce explicit reflective traces during complex
reasoning, accompanied by anthropomorphic markers such as wait, hmm, and alternatively. Although these markers are commonly used as visible indicators of reflection, their mechanisms remain unclear, which leaves the risk of overthinking associated with redundant and repetitive reflection markers. In this work, we revisit anthropomorphic reflection markers, examining their necessity for
reasoning and role in the reflection. We suppress these markers through
prompt-level and
token-level interventions, and analyze their effects on task performance across four
benchmarks and two model scales. Our results show that anthropomorphic markers are not uniformly necessary for
reasoning performance: suppressing them can preserve or improve performance in several settings, especially under larger sampling budgets. Meanwhile, marker suppression does not necessarily remove reflection behavior, as models can still perform marker-free verification. These suggest that anthropomorphic markers tend to be surface cues rather than reliable proxies for reflection itself, and motivate future research on
reasoning mechanisms beyond explicit marker patterns.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.AI
Diversifying RLVR Rollouts via First-Token Exploration
Soeun Kim and Albert No
Reinforcement learning with verifiable rewards (RLVR) trains
reasoning models without labeled trajectories, using groups of verifier-scored rollouts to explore alternative
reasoning paths. Limited rollout diversity is a central bottleneck, typically addressed through adjustments to temperature, prefixes, or rollout selection. We identify the first
token of the response as a structurally distinct target for diversification, largely overlooked in prior work. We find that the first-
token distribution is sharply concentrated and only weakly related to downstream correctness, as lower-probability candidates can yield similarly accurate responses. Diversifying the first
token can therefore broaden the
reasoning paths explored within each rollout group with little loss in response quality. Motivated by this observation, we introduce REFT (Rollout Exploration with First-
Token Diversification), a lightweight modification to RLVR. REFT samples first
tokens uniformly from the policy's top-$N$ candidates and allocates rollouts evenly across the sampled
tokens, leaving the rest of the pipeline unchanged. We evaluate REFT on eight models spanning multiple architectures and sizes (0.5B-14B), with mathematical
reasoning and code-generation tasks under GRPO and DAPO. Across these settings, REFT consistently improves Pass@1, Pass@8, and Pass@64. It also outperforms competing diversification methods at every evaluated budget, incurring the lowest rollout cost.
Read the full abstract ▾
Computation and Language
CIRF: Tokenizing Chain-of-Thoughts into Reusable Functional Units for Efficient Latent Reasoning in Large Language Models
Yukyung Lee, Yumeng Shen, Jinhyeong Park et al. (+2 more)
Implicit Chain-of-Thought (CoT) reduces the inference cost of large
language models by internalizing the explicit rationales. However, existing approaches typically lack
alignment with explicit rationales and adaptivity to example complexity. In this work, we propose CIRF (\textit{\underline{C}hain-of-thoughts \underline{I}nto \underline{R}eusable \underline{F}unctional units}), an implicit CoT framework that performs
reasoning as a dynamic sequence of discrete functional
tokens. CIRF assigns a functional
token to each semantically coherent
reasoning unit in explicit CoT traces. The model is then fine-tuned to autoregressively generate functional
tokens and their optional results, followed by the final answer. This design aligns latent
reasoning with a sequence of functional units, facilitating parallel training, explicit rationale
alignment, and adaptive
reasoning. Extensive experiments on mathematical, symbolic, and commonsense
reasoning benchmarks show that CIRF provides a favorable accuracy-latency trade-off compared with state-of-the-art implicit CoT methods. Further analyses demonstrate that CIRF constructs distinct, interpretable functional
tokens, leading to consistent performance improvements.
Read the full abstract ▾
All authors Yukyung Lee, Yumeng Shen, Jinhyeong Park, Hyein Yang, Jun-Hyung Park
Computation and Language
PrunePath: Towards Highly Structured Sparse Language Models
Zhexuan Gu, Zixun Fu and Yancheng Yuan
Feed-forward networks (FFNs) dominate the parameter count and computation of modern
language models, yet existing
pruning methods often struggle to convert sparsity into hardware-friendly inference efficiency gains. We introduce PrunePath, a budget-adaptive structured sparsification framework for FFN layers. Built on MoEfication, PrunePath replaces independent expert-wise thresholding with a softmax-normalized routing distribution and activates important experts under a cumulative-mass threshold. This formulation imposes a
token-level probability budget, enabling adaptive expert counts and a direct inference-time sparsity knob from a single checkpoint. Across NLU, NLG, and instruction-tuning evaluations, PrunePath achieves a favorable sparsity–performance trade-off compared with existing static
pruning and MoEfication-based methods. We further implement Triton kernels for KV-cache decoding to translate the resulting structured sparsity into practical
memory savings and measurable decoding-speed improvements. These results demonstrate the superior performance of PrunePath for building highly sparse, deployment-friendly large
language models.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.AI
AI, Take the Wheel: What Drives Delegation and Trust in Human-Computer Cooperative Question Answering?
Maharshi Gor, Yoo Yeon Sung, Yu Hou et al. (+4 more)
AI systems are fallible, and humans can make mistakes in deciding whether to trust AI over their own judgment. Thus, improving human-AI collaboration requires understanding when, why, and how humans decide to rely on AI. We study two distinct reliance decisions: the delegation choice – deciding when to let AI act autonomously without knowing its output, and the adoption choice – evaluating AI suggestions and deciding how to use them. Both of these decoupled reliance patterns shape collaboration, but prior work rarely studies them together in realistic settings with the same users. We address this gap by studying collaborative human–AI teams competing in a question-answering game in which humans can choose when and how to work with AI
agents to win. Our 24 matches pair 23 expert humans with 16 AI
agents, capturing 387 delegation and 1440 adoption decisions. While human–AI collaboration performs better than either AI or humans alone, humans make suboptimal collaboration decisions, both under-relying on correct AI suggestions (3.9% of opportunities missed) and over-relying when AI misleads them (1.7%). Both parties contribute wrong answers: reported model confidence is near chance when humans and AI disagree, while confirmation bias drives higher under-reliance (64.5%) when an AI suggestion agrees with humans' initial incorrect answer. To close this gap, we recommend calibrated confidence, evidence-grounded explanations, and mechanisms that help users refine trust.
Read the full abstract ▾
All authors Maharshi Gor, Yoo Yeon Sung, Yu Hou, Eve Fleisig, Irene Ying, Tianyi Zhou, Jordan Boyd-Graber
Computation and Language
Building Community-Centred NLP Resources for Puno Quechua
Elwin Huaman, Adrian Gamarra Lafuente, Johanna Cordova et al. (+1 more)
The preservation of under-resourced
languages requires digital tools and resources shaped by and for their speakers. We present the first dedicated ASR resources for Puno Quechua (ISO 639-3: qxp): (1) the largest
speech corpus for any single Quechua variety, consisting in 66 hours of recordings for scripted and spontaneous
speech (including 36 hours of manually transcribed and validated data), collected via a participatory design campaign; (2) the first systematic ASR
benchmark for Puno Quechua, evaluating state-of-the-art models and fine-tuning Whisper-base, wav2vec2-base, and XLS-R-300M, with and without continued pre-training (CPT); (3) an open release of all
datasets and fine-tuned models.
Read the full abstract ▾
All authors Elwin Huaman, Adrian Gamarra Lafuente, Johanna Cordova, Anna Korhonen
Computation and Language
When Seekers Are Hard to Help: Evaluating Emotional Support Dialogue Systems in Worst-Case Interactions
Jiajie Yang, Yangchun Li, Guanyi Chen et al. (+3 more)
Emotional Support Dialogue Systems (ESDSes) are increasingly evaluated and trained with LLM-simulated seekers. However, such simulated seekers often behave as cooperative, average-case users who disclose clearly, respond constructively, and accept support within a few turns. This can lead to overly optimistic evaluation and obscure whether ESDSes can handle difficult help-seeking interactions. In this work, we study ESDS evaluation under worst-case interactions, where seekers are hard to help due to low engagement, resistance, limited self-disclosure, emotional
volatility, or rigid negative interpretations. We first conduct an expert simulation study with eight experienced counselling professionals, who simulate difficult seekers, interact with existing Chinese ESDSes, provide scale ratings, and participate in semi-structured interviews. Based on this study, we derive worst-case seeker behaviours and identify key limitations of current systems. We then propose a worst-case evaluation framework consisting of an LLM-based worst-case seeker simulator and four worst-case-oriented metrics: Deep Emotional Understanding, Guided Exploration, Balanced Emotional Support, and Authentic and Grounded Support. Evaluating 17 systems, we find that nearly all models suffer substantial performance drops under worst-case interactions. Large general-purpose LLMs are generally more robust than specialised ESDSes, but even the strongest models struggle to sustain engagement and improve seekers' emotional states. Finally, we show that worst-case simulation can also generate useful training data, improving the robustness of smaller models.
Read the full abstract ▾
All authors Jiajie Yang, Yangchun Li, Guanyi Chen, Rui Fan, Xin Bai, Tingting He
Computation and Language
Why We Need Speech to Evaluate Speech Translation
Maike Züfle, Danni Liu, Vilém Zouhar et al. (+1 more)
Speech translation models are increasingly capable of preserving
speech-specific information (e.g., speaker gender, prosody, and emphasis), yet evaluation metrics remain blind to such phenomena. We meta-evaluate both text- and
speech-based quality estimation metrics on two contrastive
datasets targeting gender agreement and prosody, and find that both fall short, even when given direct access to the
speech signal. We then train SpeechCOMET, a family of quality estimation models with
speech encoders, and evaluate a state-of-the-art SpeechLLM as a judge. Both match or exceed text-based COMET on standard quality estimation, but neither consistently assesses
speech-specific phenomena. We identify three causes: (1)
speech-specific features are not reliably preserved in current encoders, (2) models tend to ignore the
speech source signal, and (3) quality estimation training data contains too few relevant examples. We release all models and code, and argue that progress requires dedicated
speech-specific training data and models that genuinely condition on
speech.
Read the full abstract ▾
All authors Maike Züfle, Danni Liu, Vilém Zouhar, Jan Niehues
Computation and Language
Supervised Semantic Differential for Cross-Cultural Concept Analysis: A Case Study of Human Affect
Jan Sikora, Paweł Lenartowicz and Hubert Plisiecki
Cross-cultural comparison of psychological meaning requires methods that go beyond word-level translation and examine how semantic dimensions are organized across
languages. We introduce a cross-lingual extension of the Supervised Semantic Differential (SSD), which estimates supervised semantic gradients in
embedding space and compares them across aligned
multilingual word
embeddings. The method tests gradient
alignment and difference using permutation procedures and bootstrap intervals, and interprets residual differences through clustering around the difference gradient. We demonstrate the approach on Polish, English, and French affective norm lexicons, modeling Valence, Arousal, and Dominance where available. Affective dimensions were significantly recoverable across
languages and model settings. Cross-lingual comparisons showed broad
alignment together with structured residual differences: Valence appeared mostly shared, whereas Arousal and Dominance produced more interpretable contrasts involving bodily threat, aesthetic stimulation, internal emotionality, macro-level authority, and everyday control. Several clusters also reflected corpus-specific artifacts, underscoring the need for cautious interpretation. Cross-lingual SSD offers an explainable framework for testing semantic
alignment, identifying divergence, and generating hypotheses about cross-cultural differences in psychological meaning.
Read the full abstract ▾
Computation and Language
Analyzing Quality-Latency-Resource Trade-offs in a Technical Documentation RAG Assistant Using LoRA Adaptation
Evgenii Palnikov and Elizaveta Gavrilova
We study quality-latency-resource trade-offs in a documentation-grounded
retrieval-augmented generation (RAG) system that uses Low-Rank Adaptation (LoRA) of the generator. We build a manually verified
benchmark of 5,144 question-answer pairs over the official Kubernetes documentation and combine it with a fixed hybrid-
retrieval pipeline (BGE-M3 dense, BGE-M3 native sparse, Reciprocal Rank Fusion, cross-encoder reranking). Over this
benchmark we ablate 20 LoRA configurations on Llama-3.2-3B-Instruct and Llama-3.1-8B-Instruct across rank and target-module choices, and evaluate each on
token-level F1, LLM-judged groundedness and correctness (pass@4), inference latency, inference
memory, and training cost, all reported with bootstrap 95% confidence intervals. Pareto analysis shows that LoRA adapters acting only on the q and v
attention projections consistently dominate the front, while the 3B/8B choice mainly defines operating regime. A param-matched control comparison further indicates that the q/v advantage is structural rather than purely parametric. The
benchmark, selected adapters, and code are available at https://github.com/EugPal/rag-lora-tradeoffs.
Read the full abstract ▾
Computation and Language
IFMTBench: A Comprehensive Benchmark for Multilingual Translation Instruction Following
Mingrui Sun, Mao Zheng, Zheng Li et al. (+1 more)
Modern translation workflows demand more than semantic equivalence. Users routinely require models to preserve JSON or HTML schemas, honor curated glossaries, disambiguate with provided context, and match prescribed registers, often several at once. Conventional metrics such as BLEU and xCOMET capture semantic fidelity but provide little signal on constraint adherence, while general instruction following
benchmarks ignore the cross-lingual nature of translation. We introduce \bench, a
benchmark for
multilingual translation instruction following covering seven
languages, with 4,506 single-constraint and 2,838 multi-constraint items spanning six constraint dimensions and five compositional patterns with instructions issued in all seven
languages. Constraints are split into a gating subset verified by deterministic checkers and a continuous subset scored by a rubric-based LLM judge, combined under a multiplicative rule that resists reward hacking. Evaluating 15 models reveals systematic gaps that prior protocols miss: Instruction following scales with size more sharply than translation quality, glossary and structured-format constraints dominate the difficulty gradient, and general instruction following rankings correlate only weakly with translation behavior. Our
benchmark are available at https://github.com/Tencent-Hunyuan/Hy-MT2/tree/main/IFMTBench.
Read the full abstract ▾
All authors Mingrui Sun, Mao Zheng, Zheng Li, Mingyang Song
Computation and Language · Cross-listed from cs.AI
Explaining is Harder Than Predicting Alone: Evaluating Concept-based Explanations of MLLMs as ICL Visual Classifiers
Carmen Quiles-Ramírez, Leticia L. Rodríguez, Nicolás Martorell et al. (+1 more)
In-context learning (ICL) enables
multimodal large
language models (MLLMs) to classify images from a few labelled examples. Yet, how these models use the provided context remains opaque. While Chain-of-Thought prompting is widely used, recent work argues that it may not reflect true internal computation. In this paper, we systematically evaluate the concept-based explainability of frozen MLLMs under few-shot ICL using five conditions of increasing formal rigour, ranging from baseline classification to Description Logics (DL) axiom generation. Evaluating four state-of-the-art MLLMs via an independent LLM-as-a-judge pipeline, we demonstrate that explaining is genuinely harder than predicting alone. Surprisingly, forcing models to generate formally structured, concept-based explanations degrades predictive accuracy monotonically (from 93.8% to 90.1%), contradicting the assumption that explicit
reasoning universally aids performance. However, when models successfully articulate class-discriminative visual features, explanation quality strongly correlates with correct predictions. Our findings suggest that while MLLMs excel at visual classification, they lack the specific instruction-tuning required for formal, machine-verifiable explainability.
Read the full abstract ▾
All authors Carmen Quiles-Ramírez, Leticia L. Rodríguez, Nicolás Martorell, Natalia Díaz-Rodríguez
Computation and Language
When Helpful Context Leaks: Privacy Risks in Domain-Adapted ASR
Maike Züfle and Jan Niehues
SpeechLLMs are increasingly deployed in professional settings where domain customisation is standard practice: users supply context in
prompts with sensitive information, fine-tune on proprietary recordings, or both. We identify and systematically investigate an overlooked
privacy risk of such customisation: a model adapted to recognise domain-specific terminology can be nudged into transcribing a phonetically similar word from its context or training data, even when a different word is spoken, thereby leaking private information. To evaluate this risk, we propose a technique to automatically construct
benchmarks of such attacks and apply it to measure leakage rates across two customisation mechanisms, prompting and fine-tuning. Both mechanisms cause measurable leakage, compounding when combined. We evaluate a
prompt-level mitigation strategy and analyse the accuracy-leakage trade-off across customisation approaches, finding that fine-tuning without context
prompts offers the best balance.
Read the full abstract ▾
Computation and Language
Pruning and Distilling Mixture-of-Experts into Dense Language Models
Junhyuck Kim, Jihun Yun, Haechan Kim et al. (+3 more)
Mixture-of-Experts (MoE) is now the dominant architecture for frontier
language models, yet it requires all expert parameters to be loaded in
memory, making it less preferable for
memory-constrained deployment. Existing compression methods reduce the number of experts but the output remains an MoE model with the same fundamental limitation. We present the first systematic framework for converting a trained MoE into a standard fully dense architecture: experts are scored, selected, and grouped, then concatenated into a dense FFN and refined by knowledge
distillation from the MoE teacher. We evaluate 7 scoring, 5 grouping, and 2 magnitude scaling methods across a range of selected expert counts on Qwen3-30B-A3B, yielding 350 configurations. We find that the choice of scoring method is the most impactful, with our novel diversity-aware scoring consistently outperforming prior methods on Qwen3-30B-A3B, DeepSeek-V2-Lite, and GPT-OSS-20B. Under a controlled comparison at matched parameter count, MoE-to-dense outperforms dense-to-dense
pruning by +6.3 pp in average downstream accuracy after ~4B-
token distillation at 1.6x faster training wall-clock speed.
Read the full abstract ▾
All authors Junhyuck Kim, Jihun Yun, Haechan Kim, Gyeongman Kim, Joonghyun Bae, Jaewoong Cho
Computation and Language
The Harder Text Embedding Benchmark (HTEB): Beyond One-dimensional Static Robustness
Manuel Frank and Haithem Afli
Embedding benchmarks like MTEB report a single score per model, implicitly treating robustness as a static, scalar property. We argue that
embedding robustness is multidimensional, since models respond differently to different types of variation, and requires dynamic evaluation to expose failures hidden by static
benchmarks. We introduce the Harder Text
Embedding Benchmark (HTEB), a dynamic evaluation framework that challenges model robustness along three practically interpretable axes (Lexical/Stylistic, Length and
Language) by stochastically transforming inputs at evaluation time with an LLM. Evaluating 16 open-weight
embedding models on 32
datasets covering 42
languages under transformations validated by 4,800 individual human ratings on an English subsample, supplemented by a Spanish-source evaluation and an exploratory STS-B study of pair-level label preservation, we find three patterns: (1) Models exhibit specific, partly decoupled robustness profiles across axes. (2) Across three model families, scale increases absolute scores but does not close the gap between original and transformed evaluations. Here, scaling tends to improve specifically the
Language axis. (3) English
datasets are more sensitive to HTEB transformations than
multilingual datasets. This demonstrates that HTEB identifies strengths and weaknesses of models along deployment-relevant axes, challenging current
embedding benchmarks and arguing for multidimensional, dynamic robustness evaluation. We make the code to run HTEB publicly available.
Read the full abstract ▾
Computation and Language
Framing Matters: Addressing Framing Sensitivity in Decision-Making through Behaviorally-Grounded Value Alignment
Seojin Hwang, Minju Kim, Junhyuk Choi et al. (+2 more)
Large
Language Models (LLMs) are increasingly deployed in high-stakes decision-making settings such as legal
reasoning, where consistency under factually equivalent inputs is critical. However, we find that fact-preserved but differently framed inputs can significantly destabilize LLM decisions. To systematically investigate this problem, we introduce Fragile, a large-scale
benchmark that isolates fact-preserving semantic framing across three controlled dimensions: value-tinted narration, temporal slice, and narrative vividness. Our experiments reveal a high susceptibility of LLMs to framing, with an average decision flip rate of 28.6%. We find that simple prior
prompt-level and activation-level interventions not only fail to suppress framing sensitivity but actively amplify it. We therefore propose Valign, a representation-level method that explicitly targets these framing dimensions by anchoring decisions to a stable value prior, steering hidden states toward the model's value-consistent direction, and projecting out temporal-vividness-sensitive directions from the model's hidden states. Valign consistently reduces framing-induced decision flips, demonstrating that robust mitigation requires directly targeting the internal pathways in which framing operates.
Read the full abstract ▾
All authors Seojin Hwang, Minju Kim, Junhyuk Choi, JeongHyun Park, Hwanhee Lee
Computation and Language
BenGER: Benchmarking LLM Systems on Subsumption-Based Legal Reasoning in German Law
Sebastian Nagl, Ann-Kristin Mayrhofer, Martin Heidebach et al. (+6 more)
We introduce BenGER (
Benchmark for German Law), a
benchmark and
dataset for evaluating LLM systems on subsumption-based legal
reasoning in German law. The
dataset combines 596 exam-style free-text legal case tasks across multiple levels of legal education and 531 short doctrinal
reasoning tasks. It includes a controlled validation subset of timed human-written solutions under both unaided and human-AI co-creation conditions. We evaluate 12 contemporary LLM systems - closed flagship, efficiency-oriented, and open-weight - with a rubric-aligned LLM-as-a-Judge cross-validated against a multi-rater human-grading layer (three blind reviews per solution, six judge families benchmarked against the human pool). Closed-flagship systems lead the leaderboard across all three corpora, human-AI co-creation measurably improves on unaided human work, and the LLM judge tracks human grading at Pearson r=0.76 and Cohen's k=0.60. System rankings are stable across judge families and two judges from independent providers clear the Calderon single-reviewer replacement bar on human-authored solutions.
Read the full abstract ▾
All authors Sebastian Nagl, Ann-Kristin Mayrhofer, Martin Heidebach, Aleyna Koçak, Anne Zettelmeier, Elly Breu, Angelina Greiner, Sofija Milijas, Matthias Grabmair
Computation and Language
When Confidence Misleads: Suffix Anchoring and Anchor-Proximity Confidence Modulation for Diffusion Language Models
Jungwon Park, Jimyeong Kim, Jungmin Ko et al. (+2 more)
Diffusion language models generate text by iteratively selecting and denoising masked positions, making position selection a central inference-time decision. Most training-free methods rely on model confidence, assuming that high-confidence positions are ready to be decoded. However, this assumption can fail in fully non-autoregressive (fully non-AR) decoding, where selection spans the entire masked response sequence. This can cause incomplete generation, partly due to EOT overconfidence. Existing remedies directly suppress EOT
tokens, but provide limited gains or require additional fine-tuning. We show that placing even a semantically minimal suffix anchor near the end of the response region discourages premature EOT prediction and promotes complete generation. However, despite providing little supporting context, the anchor makes nearby positions highly confident and causes them to be decoded too early. We therefore propose Suffix-Anchored Confidence Modulation, a simple training-free method that uses suffix anchoring with progress-dependent confidence modulation near the anchor. Across text-only
reasoning,
vision-
language reasoning, and code-generation
benchmarks, our method outperforms diverse fully non-AR and semi-AR baselines. It also achieves particularly large gains under highly parallel decoding, demonstrating the benefit of improving the fully non-AR regime over block-wise semi-AR regime.
Read the full abstract ▾
All authors Jungwon Park, Jimyeong Kim, Jungmin Ko, Nojun Kwak, Wonjong Rhee
Computation and Language
SuperValid: Capability-Aligned OOD Validation for Generalizable Downstream Scaling
Quanen Sun, Changxin Tian, Ke Shi et al. (+6 more)
Scaling laws guide large
language model training by relating compute to cross-
entropy loss, and recent work further extends them to predict downstream
benchmark performance. However, prior approaches face generalization limitations from two aspects: focusing on
benchmark-level performance introduces scenario-specific artifacts, while relying on IID validation loss fails to track capability improvements when training distributions vary. In this work, we argue that downstream scaling should be studied at the capability level, which captures shared skill factors across related tasks while abstracting away
benchmark-specific noise. We propose SuperValid, a framework that synthesizes OOD (out-of-distribution), capability-aligned validation data by distilling core concepts from
benchmarks within a capability domain and expanding them into diverse, knowledge-rich texts. Extensive experiments spanning 16
benchmarks grouped into 6 capability domains show that SuperValid loss exhibits strong and stable correlation with downstream performance across models of different architectures, scales, and training data distributions. As a training-free metric computable during training without
benchmark evaluation, SuperValid enables effective model selection, early stopping, and scaling decisions.
Read the full abstract ▾
All authors Quanen Sun, Changxin Tian, Ke Shi, Cai Chen, Cunyin Peng, Jia Liu, Kunlong Chen, Zhiqiang Zhang, Jun Zhou
Computation and Language
DEPART: DEcomposing PARiTy across Multilingual LLMs
Manan Uppadhyay, Prashant Kodali, Pranjal Chitale et al. (+3 more)
Multilingual Large
Language Models (mLLMs) leaderboards report per-
language accuracy but rarely explain why disparities emerge, leaving systemic biases unattributed and offering practitioners no actionable levers. We first establish that these gaps are systematic rather than artifacts of sampling noise via distribution-free Friedman and Kruskal–Wallis tests, then introduce a two-step
Bayesian hierarchical framework that decomposes
multilingual performance variance into interpretable components. First, isolating the variance attributable to
language identity, we show that observable
language features (script, family, typological distance) explain $R^2_{\text{ling}} = 79\%$ of this variance on understanding tasks and $92\%$ on
reasoning, with a model's internal representational similarity to English emerging as the dominant predictor across both task buckets. Second, decomposing the full (model$\times$
benchmark$\times$
language) cube, we find that NLU and
reasoning have fundamentally divergent variance profiles: model identity dominates understanding ($66.7\%$ of variance), whereas the
benchmark$\times$model interaction dominates
reasoning ($46.3\%$). Together these results recast
multilingual evaluation from passive performance mapping into an explainable, diagnostic framework with concrete levers for targeting the root drivers of
language disparity.
Read the full abstract ▾
All authors Manan Uppadhyay, Prashant Kodali, Pranjal Chitale, Reshma Ramaprasad, Himanshu Beniwal, Sunayana Sitaram
Computation and Language · Cross-listed from cs.LG
Self-Consistency via Marginal Sharpening
Aleksei Arzhantsev, Otmane Sakhi and Nicolas Chopin
Inference-time sampling can elicit strong
reasoning abilities from
language models without additional training. Existing power-sampling methods do so by sharpening the distribution over full generated outputs, favoring completions that are individually likely under the model. We argue that this is the wrong object to target for
reasoning: a completion entangles a
reasoning trace with a final answer, whereas what matters is whether an answer is supported by many plausible
reasoning paths. We therefore shift the target from the full-output distribution to the sharpened answer marginal, making self-consistency an inference-time objective rather than a post-hoc voting criterion. Surprisingly, this marginal target admits an efficient approximation: we propose a simple, purely autoregressive parallel sampling
algorithm that approximately samples from the sharpened answer marginal, eliciting stronger performance than standard power sampling on mathematics and coding
benchmarks while being orders of magnitude faster.
Read the full abstract ▾
Computation and Language
Better heads do not guarantee better binarized constituency parsing
Zeyao Qi, Yige Chen, Eitan Klinger et al. (+2 more)
We revisit punctuation-aware tree binarization for constituency parsing and ask whether dependency-induced headedness improves binary parser supervision. Although learned heads substantially outperform rule-based heads in intrinsic head prediction, they do not yield consistent parsing gains after debinarization. In particular, punctuation-conditioned evaluation shows that learned headedness underperforms rule-based binarization in macro-average punctuation-sensitive $F_1$, despite a small overall gain on CTB. Similar instability appears under cross-treebank transfer. These results suggest that \ycc{linguistically grounded} headedness is not necessarily parser-optimal when used as a binarization control signal. The paper presents a negative result: better head prediction does not imply better punctuation-sensitive constituency parsing.
Read the full abstract ▾
All authors Zeyao Qi, Yige Chen, Eitan Klinger, Vivaan Wadhwa, Jungyeul Park
Computation and Language
Chinese Word Boundary Recovery through Character Alignment Projection
Lusha Wang, Yuchen Li, Su Yuan et al. (+1 more)
Chinese word
segmentation is especially fragile in non-standard text, where
language learner errors and other character-level divergences disrupt the word boundaries assumed by downstream annotation and evaluation. This paper formulates Chinese word boundary recovery as an
alignment-based projection task. Given a noisy source sentence and a cleaner target counterpart, we first align the two strings at the character level and then project target-side word boundaries back onto the source. Beyond the recovery method itself, we introduce two evaluation resources: a manually checked learner Chinese
benchmark based on MuCGEC and a controlled synthetic
benchmark derived from the Chinese Penn Treebank. Experiments show that direct
segmentation remains vulnerable to compound fragmentation in learner input, whereas the proposed two step projection method corrects many over-
segmentation errors by using the corrected target to recover source-side word spans. The results show that word boundary recovery is distinct from ordinary
segmentation and that
alignment projection provides a principled mechanism for stabilizing Chinese annotation and evaluation under noisy input.
Read the full abstract ▾
All authors Lusha Wang, Yuchen Li, Su Yuan, Jungyeul Park
Computation and Language
Risk-aware Selective Prompting for Hallucination Mitigation in Large Vision-Language Models
Yuang Huang, Yafeng Zhang and Yu Zilan
Prompt-based verification is widely used to mitigate
hallucinations in large
vision-
language models (LVLMs), yet when it helps remains poorly understood. We systematically study verification prompting across two representative LVLM architectures and
hallucination benchmarks, and find that it is a risk-bearing intervention: its corrections increase with input difficulty, while newly introduced errors persist across difficulty levels. As a result, always-on prompting helps on hard inputs but offers little benefit – and can harm – easier ones. Our analysis further shows that this behavior is associated with a conservative output shift. Verification
prompts redistribute
attention from visual
tokens toward instruction
tokens and induce a distinct middle-layer
entropy pattern absent in a neutral-
prompt control, suggesting instruction-conditioned
attention redistribution rather than uniformly improved visual grounding. Motivated by this input-dependent risk, we propose Risk-aware Selective Prompting (RSP), a training-free approach that uses pre-generation uncertainty signals to trigger verification selectively. RSP mitigates the degradation of always-on prompting while preserving baseline performance, and reveals that effective selection signals vary across architectures.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.CR
SNARE: Adaptive Scenario Synthesis for Eliciting Overeager Behavior in Coding Agents
Yubin Qu, Yi Liu, Gelei Deng et al. (+4 more)
A coding
agent executes a benign task as a sequence of shell, file, and network actions, any of which can quietly exceed the authorized scope while the task still completes. We call this overeager behavior: the
prompt is not
adversarial and the run succeeds, yet an out-of-scope step can leak credentials or delete files. Existing
benchmarks miss it: task-completion suites credit any finished run, jailbreak suites probe
adversarial prompts, and the one prior overeager
benchmark applies a single fixed
prompt set to every
agent-model pair, leaving its easiest and most resistant pairs under-measured. We present SNARE (Synthesizing Non-
adversarial scenarios for Adaptive Reward-guided Elicitation), a pipeline that composes benign scenarios from reusable scope and trap fragments, scores each run with a judge-free oracle flagging trap-pattern matches and unsolicited file additions or deletions, and uses Thompson sampling to steer each pair's run budget toward the scenarios that most often trigger it. Instantiating it over 24 overeager archetypes yields OverEager, which we run across a 4x5
matrix of four coding
agents and five base models. Across 10,000 benign runs, 19.51% trigger overeager behavior, with per-pair rates spanning 11.9x. This variation is driven by the
agent framework, not the model: the framework accounts for 56% of it against the model's 21%, so any single-framework or single-model evaluation undercounts the
matrix by about a fifth.
Read the full abstract ▾
All authors Yubin Qu, Yi Liu, Gelei Deng, Yanjun Zhang, Yuekang Li, Ying Zhang, Leo Yu Zhang
Computation and Language
LegalGraphRAG: Multi-Agent Graph Retrieval-Augmented Generation for Reliable Legal Reasoning
Zerui Chen, Qinggang Zhang, Zhishang Xiang et al. (+5 more)
Graph-based
Retrieval-Augmented Generation (GraphRAG) advances flat document
retrieval by structuring knowledge as relational graphs, enabling more coherent and effective
reasoning. However, applying it to specific domains like legal
reasoning faces critical challenges. (i) Legal corpora are heterogeneous, containing multi-granular knowledge from cases, articles and interpretations. A flat knowledge graph cannot adequately differentiate between factual details, applied rules, and abstract principles, limiting accurate
retrieval. (ii) Reliable legal judgment demands transparent, evidence-based
reasoning. Traditional RAG passes retrieved context directly to an LLM without verification, resulting in opaque, error-prone
reasoning. To this end, we propose LegalGraphRAG, a framework designed for reliable legal
reasoning. Our approach introduces two core components: a hierarchical legal graph that hierarchically organizes legal sources to enable
retrieval at appropriate abstraction levels, and a multi-
agent system for reliable legal
reasoning, where a Researcher retrieves candidate evidence, an Auditor rigorously verifies its validity against source documents, and an Adjudicator synthesizes the set of verified evidence to render a final judgment. Extensive experiments show that LegalGraphRAG achieves the state-of-the-art performance, outperforming existing GraphRAG baselines in accurate and trustworthy legal analysis. Our code,
datasets and implementation details are available at https://github.com/XMUDeepLIT/LegalGraphRAG.
Read the full abstract ▾
All authors Zerui Chen, Qinggang Zhang, Zhishang Xiang, Zhimin Wei, Linfeng Gao, Xiao Huang, Zhihong Zhang, Jinsong Su
Computation and Language · Cross-listed from cs.CR
MIRAGE: Context-Aware Prompt Injection against Mobile GUI Agents via User-Generated Content
Ruoqi Guo, Yi Liu, Gelei Deng et al. (+7 more)
Mobile graphical user interface (GUI)
agents driven by
vision-
language models (VLMs) perceive the screen as rendered pixels and choose actions from what they see, so they cannot reliably separate trusted interface elements from user-generated content. We present MIRAGE (Mobile Injection of Realistic
Adversarial GUI Examples), a pipeline that turns benign mobile screenshots into
prompt-injection samples by placing attacker-controlled text into ordinary user-generated content regions, without modifying the
agent, the application, or the operating system. MIRAGE operates in three stages: a Localizer identifies user-controllable regions on the screenshot, a Generator synthesises context-aware payloads and renders them in the application's native style, and a Curator moderates realism and balances the samples across applications, region types, and attack intents. A key challenge is that an injected screenshot must stay visually indistinguishable from genuine user content while still diverting the
agent; we address this by separating the stages that control reach, realism, and distributional balance. On a 1,111-sample
benchmark spanning ten applications and eleven attack intents, all five evaluated VLM
agents are vulnerable, with attack success rates of 23%-30%, and MIRAGE scores higher on human realism ratings than the strongest prior attack (3.02 versus 2.52 out of 5). We further find that per-sample realism and attack success are uncorrelated, so visual-quality filtering alone cannot reliably defend against this threat.
Read the full abstract ▾
All authors Ruoqi Guo, Yi Liu, Gelei Deng, Yiheng Xiong, Yuekang Li, Ying Zhang, Leo Yu Zhang, Lida Zhao, Ji Jie, Yuxiao Lu
Computation and Language · Cross-listed from cs.CR
A Wolf in Sheep's Clothing: Targeted Routing Hijacking in Federated RAG
Junjie Mu and Qiongxiu Li
Federated Retrieval-Augmented Generation (FedRAG) is attractive for
privacy-sensitive applications because full local corpora remain on clients. As a result, routing must rely on client-provided semantic profiles, creating a new opportunity for manipulation. We introduce Routing Hijacking, a routing-stage attack in which a malicious client forges its profile to attract target queries despite having irrelevant underlying data. We show that this vulnerability is severe. Across three representative FedRAG routing architectures, Routing Hijacking consistently misroutes target queries and leads to downstream disruptions and failures, including missing evidence, poisoning, incorrect answers, and
hallucinations. In a controlled MedQA-USMLE stress test, we further show that poisoned retrieved evidence can mislead models across scales, leading to incorrect answers,
hallucinations, and sycophantic failures. Existing defenses do not close this gap: encrypted routing preserves the exploited ranking, and Byzantine-robust
Federated Learning (FL) rules transfer poorly to heterogeneous routing profiles. To address this gap, we propose a trust-aware post-routing framework that reweights clients using returned-evidence feedback, including
retrieval relevance, profile consistency, and cross-client agreement; online experiments show that it suppresses persistent hijacking over recurring queries and transfers to a learned neural router. Our findings establish routing integrity as a security challenge in FedRAG and highlight the need for stronger defenses for secure
federated retrieval.
Read the full abstract ▾