Computation and Language
Weave of Formal Thought
Alexandre Bouayad
Large
language models attain remarkable surface fluency on code, yet they do not formally guarantee the syntactic validity of their output, nor do they typically leverage the hierarchical structure that defines the target
language. While existing constrained-decoding frameworks offer a solution to the former, they predominantly operate under rigid assumptions that preclude critical lexical mechanisms relied upon by modern parsers (e.g., Pythonic indentation). In this work, we present a formal engine and constrained decoder that is sound and complete with respect to the full Tree-sitter specification by augmenting generalized LR (GLR) parsing with a novel speculative-lexing construction that maintains concurrent lexer-state hypotheses synchronized with the GLR graph-structured stack. We also introduce Weave of Formal Thought (WoFT), a latent-variable fine-tuning method that trains the
language model to interleave non-terminal grammar symbols directly into the generation process. Utilizing the reweighted wake-sleep (RWS)
algorithm to optimize the importance-weighted evidence lower bound (IW-ELBO) of the surface text, the model learns to selectively retain formal derivations as an adaptive structural scratchpad. Across ten widely used programming
languages spanning multiple paradigms, fine-tuning StarCoder2-3B with WoFT consistently outperforms text-only SFT baselines, achieving up to a 14.9% relative reduction in surface-
token cross-
entropy, and demonstrating that discretionary latent syntax recovers critical structural information that flat autoregressive training discards. Our code and implementation are publicly available at https://github.com/alexbouayad/formal.
Read the full abstract ▾
Computation and Language
Overview of HIPE-2026: Person-Place Relation Extraction from Multilingual Historical Texts
Juri Opitz, Maud Ehrmann, Corina Raclé et al. (+3 more)
Was this person ever at that place, and if so, when? Answering such questions from noisy,
multilingual historical documents is the central challenge of HIPE-2026, the third edition of the HIPE evaluation series. Moving from named entity recognition and linking (HIPE-2020, HIPE-2022) to
reasoning about relationships between entities, HIPE-2026 targets two temporally grounded relation types: $at$, indicating that a person was present at a location at some point prior to a document's publication date, and $isAt$, indicating presence contemporaneous with that date. This paper presents the results of the evaluation campaign, which confronted 17 participating teams with the challenges of historical
language variation, OCR noise, and indirect contextual cues across three
languages: French, German, and English. The
datasets include historical newspaper text from the nineteenth and twentieth centuries, as well as a surprise-domain generalization set drawn from early modern French literary texts. A distinctive feature of HIPE-2026 is its three-fold evaluation framework, which assesses predictive accuracy, computational efficiency, and cross-domain generalization, reflecting the practical demands of large-scale historical document processing in the cultural heritage domain. Across more than 40 submitted runs, results reveal a wide range of strategies, from state-of-the-art large
language models to lightweight task-specific classifiers, and highlight the trade-offs between accuracy, efficiency, and robustness inherent to historical relation extraction at corpus scale. System descriptions,
datasets, and findings are presented and discussed, offering a detailed picture of the current state of temporally grounded relation extraction for historical documents.
Read the full abstract ▾
All authors Juri Opitz, Maud Ehrmann, Corina Raclé, Andrianos Michail, Matteo Romanello, Simon Clematide
Computation and Language
SARA: Unlocking Multilingual Knowledge in Mixture-of-Experts via Semantically Anchored Routing Alignment
Tianyu Dong, Yangyang Liu, Jiang Zhou et al. (+9 more)
Sparse Mixture-of-Experts (MoE) architectures have emerged as an increasingly influential paradigm as they offer a strategic balance between parameter scalability and computational efficiency. However, low-resource
languages, which suffer from a scarcity of high-quality training data, often have their
tokens routed to different experts than those predominantly activated by high-resource inputs, which limits cross-lingual expert sharing. This cross-lingual routing divergence consequently hinders their efficacy in
multilingual contexts. To address this issue, we propose SARA (Semantically Anchored Routing
Alignment), a framework designed to transfer specialized capabilities from high-resource
languages as anchors to low-resource
languages. SARA explicitly aligns the routing distribution of
multilingual inputs with high-resource semantic anchors using a symmetric Jensen-Shannon (JS) divergence constraint. Unlike traditional
distillation methods that operate on output logits, SARA directly aligns the internal routing distributions of MoE layers, encouraging mechanistic consistency in expert selection across
languages. We conduct experiments on 2 LLMs across 5 low-resource
languages and 3
benchmarks. Experiment results demonstrate that SARA outperforms standard instruction tuning, e.g., +0.8% on Qwen3-30B-A3B and +1.2% on Phi-3.5-MoE-instruct on Global-MMLU. Further analyses show that SARA effectively addresses performance bottlenecks in low-resource
languages, providing a scalable pathway to enhance
multilingual capabilities in sparse architectures.
Read the full abstract ▾
All authors Tianyu Dong, Yangyang Liu, Jiang Zhou, Xinwei Wu, Xiaohu Zhao, Hao Wang, Heng Liu, Linlong Xu, Longyue Wang, Weihua Luo, Shaolin Zhu, Deyi Xiong
Computation and Language
Beyond Function Calling: Benchmarking Tool-Using Agents under Tool-Environment Unreliability
Yang Tian, Zhengpeng Shi, Yu Zhou et al. (+1 more)
Large
language models are increasingly deployed as
agents that solve tasks by interacting with external tool environments. Although recent tool-use
benchmarks increasingly cover complex task settings, they still largely assume clean, stable, and trustworthy tool environments, leaving tool-environment unreliability insufficiently examined. We introduce ToolBench-X, a
benchmark for evaluating
agents under recoverable reliability hazards. ToolBench-X contains executable multi-step tasks across diverse domains and sequential, parallel, and mixed workflows, each paired with deterministic tools and a canonical final answer for automatic evaluation. Starting from clean tool environments, ToolBench-X injects five structured hazard types: Specification Drift, Invocation Error, Execution Failure, Output Drift, and Cross-source Conflict. Crucially, each injected instance remains solvable through at least one valid recovery path, such as retrying, fallback, verification, or cross-checking. Experiments reveal a substantial reliability gap:
agents that perform well with reliable tools often fail under recoverable hazards. Further analysis shows that failures are driven less by tool-use volume or inference budget than by limited hazard diagnosis and ineffective recovery. Targeted recovery hints recover many failed tasks, while test-time scaling yields more limited gains. These results suggest that tool-use evaluation should move beyond function-call accuracy toward task completion under unreliable tool environments. The code and data is available at https://github.com/Foreverskyou/ToolBench-X.
Read the full abstract ▾
All authors Yang Tian, Zhengpeng Shi, Yu Zhou, Bo Zhao
Computation and Language · Cross-listed from cs.IR
How Large Language Models Source Brand Reputation Across Languages and Markets
Dmitrij Zatuchin
When a large
language model (LLM) answers a question about a company, it grounds the answer in retrieved web sources, and those sources decide what the model says. Most analysis of AI brand visibility looks at the answer text. This study looks one step earlier, at the citations. We merge three Rankfor.AI
datasets covering 128 brands across 12 home
markets and 13
languages, and analyse 167,551 URL-grounded citations (189,974 total attribution rows). We classify each citation by domain and source type and measure where AI gets its brand information, by
language and by
market. Four patterns hold. First, AI grounds brand answers overwhelmingly in third-party sources: 85.7% of citations point to sites the brand does not own, against 14.3% owned. Second, the source base is concentrated and long-tailed: 80% of citations come from about 18% of domains, fitting a Zipf law (alpha = 0.86, R^2 = 0.983). Third, one reference site dominates almost everywhere: Wikipedia is the most-cited domain in 11 of 12
languages, the exception being Lithuanian, where the business daily vz.lt edges it (4.38%). Fourth, the source mix is
market-specific at the margin: for 46 Polish national brands the most-cited domain is YouTube, and four HR and careers portals supply 637 citations against 297 for Polish Wikipedia, about twice as many.
Read the full abstract ▾
Computation and Language
Do Encoders Suffice? A Systematic Comparison of Encoder and Decoder Safety Judges for LLM Adversarial Evaluation
Han Jeon, Shiv Medler, Joseph Voyles et al. (+1 more)
With the widespread adoption of large
language models (LLMs) in chatbots and everyday applications, companies increasingly need guardrails that are effective while remaining low-cost and low-latency. Safety evaluation of LLM outputs has generally relied on LLM-based judges, which can be effective but are often slow and expensive to deploy at scale. In this paper, we evaluate whether fine-tuned modern encoder classifiers from the ModernBERT family, including ModernBERT and Ettin, can reliably identify harmful LLM outputs in user-model conversations without substantial performance loss relative to LLM-based judges. We
benchmark these encoder classifiers against rule-based prefix matching, fine-tuned LLM classifiers, and LLM judges using a range of judge-prompting strategies across open-source
adversarial datasets. The LLM judges include evaluation methodologies from StrongReject, ShieldGemma, JailbreakBench, AILuminate, SorryBench, and a Claude-as-a-judge setup, as well as fine-tuned safety classifiers such as LlamaGuard 3 and LlamaGuard 4. The encoder classifiers are fine-tuned on judge-labeled data using a majority-voting label strategy and are then evaluated on a gold-standard holdout
dataset to assess their performance relative to LLM judges. We report absolute performance using F1 score, false negative rate, and precision-recall metrics. We also break down results by attack technique, including single-turn prompting, decomposition, escalation, and context manipulation, to identify where encoder classifiers align with or diverge from LLM-based judges. Our findings provide guidance on when encoder classifiers can serve as cost- and latency-efficient alternatives to LLM-based safety evaluation.
Read the full abstract ▾
All authors Han Jeon, Shiv Medler, Joseph Voyles, Matt Wood
Computation and Language · Cross-listed from cs.DS
Space-Efficient Language Generation in the Limit
Nicolas Flammarion, Chirag Pabbaraju, Hristo Papazov et al. (+2 more)
We initiate a resource-aware theory of
language generation in the limit under the minimal constraint of space efficiency. In our framework, a learner observes an
adversarial positive stream from a target
language $K$ and must eventually output a
hallucination-free hypothesis
language $L \subseteq K$ while omitting at most $\Delta$ strings of $K$. We focus on $\mathcal{C}_{s,k}$, the collection of
languages recognized by DFAs with at most $s$ states over an alphabet of size $k$, as the natural hypothesis class for
memory-bounded learners. In the exponential-space regime, we prove that a learner can exactly identify the target $K$. Under a stricter
memory budget, we characterize the strongest possible generation guarantees. In particular, we present a streaming
algorithm using $\mathrm{poly}(s,k)$ space that converges to a hypothesis with generation gap $\Delta = O(k^{2s-2})$. Moreover, the learned hypothesis captures every string in $K$ of length at least $2s-1$. We complement this result with a near-matching lower bound through a reduction from a standard communication complexity problem. Specifically, achieving generation gap $\Delta \le k^{(1-\varepsilon)s}$ requires $k^{\Omega(\varepsilon s)}$
memory. Together, these results reveal a sharp transition between
polynomial-space generation and exponential-space exact identification.
Read the full abstract ▾
All authors Nicolas Flammarion, Chirag Pabbaraju, Hristo Papazov, Miltiadis Stouras, Ola Svensson
Computation and Language · Cross-listed from cs.LG
Uncertainty Quantification for Computer-Use Agents: A Benchmark across Vision-Language Models and GUI Grounding Datasets
Divake Kumar, Sina Tayebati, Devashri Naik et al. (+5 more)
Computer-use
agents turn
vision-
language model (VLM) predictions into executable GUI clicks, so reliable uncertainty estimates are essential for rejection, calibration, miss-severity ranking, and spatial safety regions. Yet evidence on post-hoc uncertainty quantification (UQ) for these
agents is fragmented across isolated model and
dataset pairs, leaving it unclear whether UQ rankings stay stable when the
agent,
benchmark, or observable interface changes. We present Argus, a cross-regime
benchmark for post-hoc UQ in single-step executable GUI grounding: a 27-method open-weight
matrix over 4 VLM
agents and 4
datasets, plus an 8-method closed-source
matrix across 3 frontier vendors where logits, hidden states, and
attention maps are unavailable. Evaluated methods span logit-based scores, sampling and consistency measures, hidden-state and density
estimators (Mahalanobis, SAPLMA),
attention-based scores, P(True) and verbalised-confidence prompting, and split-conformal prediction. The main finding is selective transfer: UQ rankings are stable across
datasets for a fixed model, but degrade across model classes and observable interfaces. Hidden-state and density methods are the most stable open-weight family, while CoCoA-1MCA, Focus, sampling-based scores, and verbalised self-assessment win in specific regimes. Within-model ranking transfer is strong (Spearman rho up to 0.969), but cross-tier transfer to closed-source vendors averages only +0.08, so closed-source UQ should be reranked on the target rather than extrapolated. Conformal click regions show score-level discrimination is not enough for deployment: locally weighted disks shrink radii by 40-60% when the plug-in UQ is calibrated, but coverage degrades under calibration-test or interface mismatch. We release per-item records, calibration/test splits, UQ scores, and analysis scripts for regime-aware UQ selection in GUI
agents.
Read the full abstract ▾
All authors Divake Kumar, Sina Tayebati, Devashri Naik, Amanda Sofie Rios, Nilesh Ahuja, Omesh Tickoo, Ranganath Krishnan, Amit Ranjan Trivedi
Computation and Language
OPERA: Aligning Open-Ended Reasoning via Objective Perplexity-based Reinforcement Learning
Wenxuan Jiang, Zining Fan, Zijian Zhang et al. (+6 more)
Reinforcement Learning (RL) has enabled LLMs to excel in objective
reasoning tasks such as mathematics and code generation. However, applying RL to open-ended tasks, such as creative writing, remains challenging because LLM-as-a-judge reward models often exhibit stylistic biases and positional inconsistencies, leading to unstable supervision. To address this, we propose OPERA (Objective Perplexity-based Reflective
Alignment), which replaces unreliable external judges with intrinsic rewards derived from perplexity dynamics. Specifically, we derive an intrinsic reward signal from perplexity dynamics, quantifying uncertainty reduction at critical reflective states. During the cold-start phase, we introduce a data synthesis method that leverages carefully designed guiding words to generate diverse
reasoning traces, along with perplexity-prioritized rollouts that utilize internal log-probabilities to identify logically consistent
reasoning branches. This pipeline yields a large-scale
dataset comprising 20,000 high-quality
reasoning trajectories. Empirical evaluations consistently demonstrate the scalability and efficacy of our approach in
alignment for open-ended tasks. Implementing OPERA on Qwen3-8B establishes a new state-of-the-art among open-source models, achieving parity with or surpassing proprietary models like Gemini2.5 and MiniMax-M2.5 in some open-ended tasks. The code is available at https://github.com/pangpang-xuan/OPERA.
Read the full abstract ▾
All authors Wenxuan Jiang, Zining Fan, Zijian Zhang, Xuecheng Wu, Hongming Tan, Haoyang Dai, Xiaoyu Li, Xuezhi Cao, Ninghao Liu
Computation and Language · Cross-listed from cs.CR
RAS: Measuring LLM Safety Through Refusal Alignment
Chang-Chieh Huang, Yan-Lun Chen, Chia-Mu Yu et al. (+1 more)
Safety evaluation of large
language models (LLMs) is commonly performed by querying models with unsafe or jailbreak
prompts and judging whether their outputs violate a safety policy. Although useful, output-level evaluation is expensive, sensitive to judge choice, and easily tied to fixed question banks. We propose **SafeVec**, a white-box evaluation procedure that measures safety from internal representations rather than generated answers. **SafeVec** first extracts layer-wise refusal directions from a safety-aligned reference model, then selects stable layer windows where safe and unsafe behaviors are separable, and finally scores a target model by measuring whether its hidden states align with these refusal directions under unsafe and jailbreak
prompts. The resulting metric, **RAS** (**R**efusal **A**lignment **S**core), maps representation-level refusal
alignment to a calibrated 0-100 safety score. Across `Llama`, `Gemma`, and `Qwen` model families, RAS separates aligned models from uncensored and abliterated variants, tracks output-level attack success rate, and is substantially faster than judge-based evaluation. These results suggest that refusal
alignment provides a compact and efficient signal for white-box LLM safety evaluation.
Read the full abstract ▾
All authors Chang-Chieh Huang, Yan-Lun Chen, Chia-Mu Yu, Wei-Bin Lee
Computation and Language · Cross-listed from cs.CR
Tracing Target Answers in Poisoned Retrieval Corpora via Token Influence Attribution
Yan-Lun Chen, Pin-Yu Chen, Chia-Mu Yu et al. (+3 more)
Retrieval-Augmented Generation (RAG) systems are vulnerable to corpus poisoning attacks that manipulate model outputs through malicious retrieved documents. Existing detection methods typically rely on auxiliary classifiers or additional LLM-based verification, introducing substantial computational overhead. We present TRACE, a lightweight detection framework that identifies poisoning attacks by tracing answer-related
tokens through
token influence attribution. TRACE first discovers recurrent high-influence keywords across retrieved documents and then performs a secondary verification to confirm their influence on model predictions. Experiments on three QA
benchmarks and six LLMs demonstrate strong detection performance while simultaneously uncovering attacker-specified target answers.
Read the full abstract ▾
All authors Yan-Lun Chen, Pin-Yu Chen, Chia-Mu Yu, Ying-Dar Lin, Yu-Sung Wu, Wei-Bin Lee
Computation and Language
Zhen Li, Xin Huang, Liang Wang et al. (+8 more)
LLM-based text embedders have substantially improved
retrieval and semantic representation quality, but their deployment remains costly: large backbone models slow down
embedding inference, while high-dimensional full-precision
embeddings impose substantial storage and bandwidth overhead on large-scale indexes. In this paper, we present BITEMBED, an extreme low-bit framework for LLM-based text
embedding that jointly targets encoding efficiency and vector storage. BITEMBED converts pretrained LLM backbones into BitNet-style
embedding encoders with ternary weights, quantized activations, and lightweight normalization refinement. The converted model is adapted to representation learning through continual contrastive pre-training, followed by supervised contrastive fine-tuning with both similarity-distribution
distillation and
attention-relation
distillation from a full-precision teacher. Beyond quantizing the backbone, BITEMBED further trains output
embeddings to support multiple storage precisions meeting different storage needs in various scenarios. Experiments on MMTEB (eng, v2) with Qwen3-0.6B and Gemma3-270M show that BITEMBED is largely comparable to full precision teacher embedders. Moreover, BITEMBED flexibly obtains text
embeddings of various precisions, achieving a trade-off between performance and storage cost.
Read the full abstract ▾
All authors Zhen Li, Xin Huang, Liang Wang, Nan Yang, Ting Song, Yan Xia, Xun Wu, Shaohan Huang, Huishuai Zhang, Furu Wei, Dongyan Zhao
Computation and Language
Is GraphRAG Needed? From Basic RAG to Graph-/Agentic Solutions with Context Optimization
Long Chen, Ryan Razkenari, Yuxuan Zhou et al. (+5 more)
As advanced RAG variants like GraphRAG and Agentic RAG emerge, one leading question is when and how to use them. Here, we introduce a framework for different RAG scenarios evaluation and comparison on semi-structured knowledge bases, including regular RAG, GraphRAG, Modular RAG and Agentic RAG. We provide implementation for 9 standardized RAG scenarios, and conduct experiments for a comprehensive comparison. These scenarios are designed for real use cases regarding data and domain restrictions, spanning from simple document-based
retrieval to advanced features such as hybrid text-graph
retrieval, integration with computed or pre-defined domain knowledge graphs, agentic multi-step planning, and
agent-graph integration. Besides, we present a novel context engineering method for GraphRAG and Agentic RAG, addressing the context/
memory overflow issues, efficiently managing text and graph
retrievals with new representations and agentic loop design, leading to 19%-53% reduction on
token usage. Moreover, further analysis identifies a
retrieval-generation gap where expanded
retrieval does not proportionally improve generation quality, suggesting
retrieval-oriented metrics overstate advanced
retrieval benefits. This work provides data-driven insights on when and how to use them for building production-ready intelligent RAG systems.
Read the full abstract ▾
All authors Long Chen, Ryan Razkenari, Yuxuan Zhou, Yuan Tian, Rahul Ghosh, Venkatesh Pappakrishnan, Disha Ahuja, Vidya Sagar Ravipati
Computation and Language
MedGuards: Multi-Agent System for Reliable Medical Error Detection and Correction
Congbo Ma, Hu Wang, Yichun Zhang et al. (+1 more)
As Large
Language Models (LLMs) are increasingly deployed in healthcare settings, accurate error detection and correction in generated or existing text becomes critical, as even minor mistakes can pose risks to patient safety. Existing methods for error detection and correction, including automated checks and heuristic-based approaches, do not generalize well across unseen
datasets. In this paper, we propose MedGuards as a medical safety guardrail, which is a new framework that treats medical error detection and correction as a multi-
agent in-context learning task. Specialized
agents separately detect, localize, and correct errors, while a confidence-guided arbitration mechanism resolves disagreements using
reasoning traces and confidence scores. This design enhances interpretability, robustness, and adaptability, without requiring additional training of the base LLMs. Additionally, we introduce the Keyword-Prioritized Correction Score (KPCS), a new evaluation metric that considers whether critical keywords within the reference text are generated correctly, providing a more comprehensive assessment than conventional metrics. Experiments across four
multilingual medical
datasets consisting of clinical notes demonstrate significant improvements by the proposed framework across several metrics and models. Our aim is to enable safer deployment of LLMs in real-world healthcare applications. For reproducibility, we make our code publicly available at https://github.com/congboma/MedGuards.
Read the full abstract ▾
All authors Congbo Ma, Hu Wang, Yichun Zhang, Farah E. Shamout
Computation and Language
Staying In Character: Perspective-Bounded Memory For Book-Based Role-Playing Agents
Xushuo Tang, Junhe Zhang, Zihan Yang et al. (+4 more)
Recent LLM role-playing systems build character
agents from novels by extracting characters, scenes, and relations. Yet long-narrative role-playing suffers from two failures: Factual Overreach, where shared
retrieval or parametric
memory lets a character use facts outside its perspective, and Stylistic Monotony, where profile descriptions flatten a character into a fixed voice. To address these failures, we propose REVERIEMEM, a three-layer
memory architecture for book-based character
agents. The episodic layer stores first-person scene
memories; the semantic layer stores visibility-tagged facts; and the personality layer stores situation-dependent
speech and behaviour patterns. For evaluation, we construct KBF-QA, a 4,386-question
benchmark over eight novels for testing knowledge boundaries. REVERIEMEM improves Knowledge Boundary Fidelity by 34.6 percentage points over the strongest prior method. On BOOKWORLD's five-dimension pairwise narrative protocol, REVERIEMEM achieves a ~ 79% win rate, suggesting that perspective-bounded
memory improves both boundary fidelity and character-grounded narrative generation.
Read the full abstract ▾
All authors Xushuo Tang, Junhe Zhang, Zihan Yang, Yifu Tang, Sichao Li, Longbin Lai, Zhengyi Yang
Computation and Language
Constraint Tax in Open-Weight LLMs: An Empirical Study of Tool Calling Suppression Under Structured Output Constraints
Fangzheng Li, Aimin Zhang and Chen Lv
Tool Calling and Structured Output are two core capabilities of modern
Agent systems, yet their interaction under joint deployment conditions remains insufficiently understood. This paper reports a reproducible phenomenon observed in a production
Agent system: when Tool Calling and JSON Schema constraints are simultaneously enabled, multiple open-weight models cease invoking tools despite maintaining high schema compliance. We refer to this behavior as Tool Suppression. Through controlled experiments across multiple model families and deployment settings, we consistently reproduce Tool Suppression under joint constraints, while tool execution and schema compliance remain functional when evaluated independently. Further analysis reveals that JSON Schema constraints are compiled into grammar-based
token masks, causing tool-call
tokens to become unreachable during decoding. This provides an implementation-level explanation for the observed behavior. To interpret the phenomenon, we formulate the Constraint Priority Inversion (CPI) hypothesis, which suggests that schema satisfaction may dominate action-selection behavior under multiple simultaneous constraints. We present CPI as a behavioral hypothesis consistent with the observed evidence rather than a verified internal mechanism. To mitigate the problem, we propose Transparent Two-Pass Execution, an inference-time strategy that decouples tool execution from schema-constrained response generation. Experimental results show that this approach restores tool invocation while preserving structured output guarantees without requiring model retraining. These findings suggest that evaluating tool use and structured output separately may overlook important reliability issues in production
Agent systems. Code, data, and docs will be released at https://github.com/Fzsama/Constrain-Tax-26-06.git.
Read the full abstract ▾
Computation and Language
Riazi-8B: An Urdu Large Language Model for Mathematical Reasoning
Azher Ali, Ibtsam Haider, Raja Khurram Shahzad et al. (+2 more)
Recent LLMs demonstrate strong mathematical
reasoning capabilities, but existing gains rely heavily on English-centric training resources and
benchmarks. As a result,
reasoning performance degrades substantially in low-resource
languages such as Urdu, where
reasoning-oriented
datasets and adapted models remain scarce. Urdu lacks both
reasoning-oriented resources and models adapted for multi-step mathematical problem solving, limiting the applicability of recent progress to Urdu-speaking users. We address this gap through Riazi-8B, an Urdu mathematical
reasoning model developed through a two-step adaptation process comprising continued pre-training on Urdu Wikipedia and supervised fine-tuning on Urdu Chain-of-Thought data derived from GSM8K. We evaluate Riazi-8B on MGSM-Urdu against existing Urdu instruction-tuned models. Our results show consistent improvements in answer correctness,
reasoning quality, response completeness, and Urdu generation. Our findings demonstrate that combining Urdu
language adaptation with
reasoning-focused fine-tuning is an effective strategy for extending mathematical
reasoning capabilities to low-resource
languages.
Read the full abstract ▾
All authors Azher Ali, Ibtsam Haider, Raja Khurram Shahzad, Seemab Latif, Mehwish Fatima
Computation and Language
BiPACE: Bisimulation-Guided Policy Optimization with Action Counterfactual Estimation for LLM Agents
Hanyang Wang, Weijieying Ren, Yuxiang Zhang et al. (+4 more)
Stepwise group-based RL is an attractive way to train long-horizon LLM
agents without a learned critic: it reuses multiple sampled rollouts to estimate local advantages. Its weakness is less visible but more fundamental: every group-relative
estimator assumes that the steps it compares are equivalent for credit assignment. We show that current agentic variants violate this assumption through a state-action credit mismatch. The observation-hash partition is overly fine on the state side, creating singleton groups with zero step-level signal, while a single within-group mean is too coarse on the action side, mixing state-value estimation with action-specific credit. We introduce BiPACE (Bisimulation-Guided Policy Optimization with Action Counterfactual Estimation), a drop-in advantage
estimator that fixes both sides without adding a critic, auxiliary loss, or extra rollouts. BiGPO clusters steps by cosine distance in the actor's own hidden-state geometry, an empirical policy-induced proxy for bisimulation that substantially lowers the singleton rate left by observation hashing. PACE then recenters returns within each behavioral cluster using action-conditioned peer baselines; its Q-style instance estimates a local Q(s,a)-V(s) nonparametrically. On ALFWorld/Qwen2.5-7B, BiPACE_Q raises overall validation success from GiGPO's 90.8 to $97.1\pm0.9$ over three seeds, and crosses the 95% threshold on every seed, which GiGPO never does within the same budget. On Qwen2.5-1.5B it reaches $93.5\pm1.2$ versus GiGPO's 86.7, and on WebShop and TextCraft it improves over GRPO and GiGPO at both model scales. The measured BiPACE-specific overhead is 11.3% of a single training-step wall time. Yet it changes the
estimator's comparison unit from surface identity to approximate behavioral equivalence plus action-side counterfactuals. The code is available at https://github.com/TianxiangZhao/BiPACE.
Read the full abstract ▾
All authors Hanyang Wang, Weijieying Ren, Yuxiang Zhang, Ding Cao, Zhizhao Zeng, Ke Zeng, Tianxiang Zhao
Computation and Language
SFL-MTSC: Leveraging Semantic Frame-Level Multi-Task Self-Consistency for Robust Multi-Intent Spoken Language Understanding
Po-Yen Chen and Berlin Chen
Prompt-based spoken
language understanding (SLU) with large
language models (LLMs) often suffers from inconsistent intent–slot structures due to decoding stochasticity, particularly in multi-intent scenarios. In view of this, we propose Semantic Frame-Level Multi-Task Self-Consistency (SFL-MTSC), a novel structured aggregation framework operating at the semantic frame level. Instead of output-level majority voting, SFL-MTSC decomposes predictions into intent-specific frames, applies domain–intent grouping and slot-level clustering, and evaluates cluster reliability using path support scoring. Reliable frames are retained and re-integrated to form the final prediction. Zero-shot experiments on the MAC-SLU
benchmark dataset show improved slot F1 and overall accuracy over single-path inference, while intent accuracy remains largely stable across most settings.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.CR
Security and Privacy in Retrieval-Augmented Generation: Architectures, Threats, Defenses, and Future Directions for Building Trustworthy Systems
Balamurugan Palanisamy, G S S Chalapathi, Vikas Hassija et al. (+1 more)
Retrieval-Augmented Generation (RAG) has emerged as a dominant paradigm for enhancing large
language models with external knowledge. By coupling
retrieval mechanisms with generative models, RAG systems improve factual grounding and adaptability across domains. However, integrating
retrieval pipelines introduces new security and
privacy risks that extend beyond conventional
language modeling threats. Sensitive information may be exposed through
retrieval indices, query logs, context construction, or
federated updates, while
adversarial manipulation of knowledge bases can undermine trust in generated outputs. This survey provides a comprehensive examination of
privacy and security challenges across RAG systems deployed in centralized, on-device (Micro-RAG),
federated, and hybrid paradigms. We present a unified taxonomy of threat surfaces spanning the
retrieval, context construction, and generation stages and systematically analyze attack classes, including membership inference, index inference, poisoning, gradient leakage, and collusion. We further review architectural, algorithmic, and cryptographic defenses, highlighting
privacy-utility trade-offs and deployment considerations. Finally, we outline open research challenges toward building trustworthy, secure, and resilient RAG systems for real-world applications.
Read the full abstract ▾
All authors Balamurugan Palanisamy, G S S Chalapathi, Vikas Hassija, Rajkumar Buyya
Computation and Language · Cross-listed from cs.SE
Evaluating LLMs on Real-World Software Performance Optimization
Ezgi Sar{\i}kayak, Wenchao Gu, Hesham Ghonim et al. (+1 more)
Software performance optimization is a notoriously complex and manual task. Despite the growing use of Large
Language Models (LLMs) for code refinement, we still lack
benchmarks that capture how optimization actually happens in real-world codebases. Existing frameworks often oversimplify the problem by focusing on isolated functions or a single performance metric, missing the critical trade-offs between execution time and
memory footprint, the inherent noise of the measurement environment, and the variability introduced by different input data and execution conditions. We address this by introducing SWE-Pro, a repository-level
benchmark derived from 102 expert-written optimizations from open-source projects. Unlike previous
benchmarks, SWE-Pro pairs each task with parameterized tests to evaluate runtime, peak
memory, and Time-Weighted
Memory Usage (TWMU) across varying input data and execution conditions under noise-aware measurement conditions. Our evaluation shows that current LLMs struggle significantly: runtime gains are negligible, and
memory optimizations are nearly non-existent. This stands in sharp contrast to expert implementations, which achieve an aggregate speedup of 15.5x and peak
memory reduction of 171.3x over
benchmark tasks. Expert-written improvements are observed in 91.2% of tasks for runtime and 65.7% for peak
memory. Our findings expose a substantial gap between current LLM capabilities and the demands of expert-level engineering.
Read the full abstract ▾
All authors Ezgi Sar{\i}kayak, Wenchao Gu, Hesham Ghonim, Chunyang Chen
Computation and Language · Cross-listed from cs.AI
Cliff Tokens: Analyzing Failure Trigger Tokens in LLM Mathematical Reasoning
Jaeyong Ko, Jinu Lee, Pilsung Kang et al. (+1 more)
Large
language models reach high accuracy in mathematical
reasoning, but individual traces on the same problem diverge; some arrive at the correct answer while others fail. Prior work localizes such failures at the step, chunk, or sentence level, or identifies
tokens where failure has already occurred. These approaches leave open which
token triggers failure. We introduce the cliff
token, a
token at which the estimated probability of reaching the correct answer (success probability) drops beyond an adaptive threshold. Across seven models and three mathematical
reasoning benchmarks (GSM1K, MATH500, AIME 2025), cliff
tokens act as failure triggers. For incorrect traces containing cliff
tokens, we compare resampling immediately before and after the first cliff
token. Resampling before it shows higher pass@$k$ at the same sample count. We further introduce a cliff taxonomy of deterministic, uncertain, and sampled-off cliffs, defined by greedy choice and
token entropy. Additionally, we show that the three types differ as training signals. Using single-
token preference optimization at cliff positions (Cliff-DPO), we find that uncertain and sampled-off cliffs show larger accuracy gains than deterministic cliffs on three evaluation
benchmarks. We release
token-level rollout data and source code to enable further analysis without regenerating costly rollouts: https://github.com/beaver-22/Cliff-
token
Read the full abstract ▾
All authors Jaeyong Ko, Jinu Lee, Pilsung Kang, Yukyung Lee
Computation and Language
Fault of Our Stars: Behavioral Drivers of Rating-Sentiment Incongruence
Ramanaish Abaiyan, Ruththiragayan Sutharsan, Kusal Amantha et al. (+7 more)
When people share experiences online, they often express thoughts in two ways: a
star rating and a written review. In sentiment analysis, ratings are widely used as convenient weak labels for textual sentiment, yet whether the two actually agree is rarely questioned. This study investigates sentiment-rating incongruence, where the sentiment expressed in review text differs from the sentiment implied by the assigned
star rating, in Sri Lankan tourism attraction reviews. A
dataset of 16,156 reviews from 2010 to 2023 is analyzed using a
transformer-based sentiment pipeline that derives textual sentiment independently of assigned ratings. Incongruence occurs in 18.6% of reviews and falls into six directional patterns, with Conservative Rater and Obligatory 5-
Star behaviors accounting for the majority of mismatches. Prevalence also varies across venue types, with museums showing the highest rates. Statistical tests, logistic regression, Random Forest, and SHAP analysis identify venue type, reviewer expertise, review length, and temporal factors as contributors to rating-text divergence. Overall, this study demonstrates that
star ratings are not interchangeable with textual sentiment and should be validated before being treated as ground-truth labels in NLP.
Read the full abstract ▾
All authors Ramanaish Abaiyan, Ruththiragayan Sutharsan, Kusal Amantha, Anusan Krishnathas, Asma Rauff, Kovindarajah Sriyathurshan, Patalee Narasinghe, Nirasha Munasinghe, Nisansa de Silva, Sandareka Wickramanayake
Computation and Language
Spam and Sentiment Detection in Arabic Tweets Using MARBERT Model
Abrar Alotaibi, Atta-ur Rahman, Raheel Alhaza et al. (+6 more)
Saudi Telecom Company (STC) is among the most popular companies in Saudi Arabia, with many customers. Yet, there is still a big room for improvement in users' satisfaction. Social media is the most robust platform to gauge users' satisfaction and determine their sentiments and critics. Twitter is among the most popular social media platform in this regard. STC customers prefer to use Twitter to write their feedback because it's a fast way to get responses due to the STC customer services account. One way to achieve customer demands and improve customer service is using the Sentiment Analysis tool. Sentiment Analysis on Twitter is highly used because of the significant number of tweets and the different opinions. Likewise, Deep learning is the best existing Sentiment Analysis method, and it has diverse models. Bidirectional Encoder Representations from
Transformers (BERT) model is one of the deep learning models which have achieved excellent results in Sentiment Analysis for Natural
Language Processing (NLP). NLP is mainly investigated in the English
language. However, for Arabic, there is a significant gap to be filled. This study trained the proposed model using MARBERT and measured the performance using f1-score, precision, and recall metrics. We trained the model with an Arabic
dataset of 24,513 tweets, including 1,437 positive, 13,828 negative, 5,694 neutral, 1,221 sarcasm, and 2,297 indeterminate tweets. The main goal is to analyze the tweets and get the sentiment to improve STC customer service. The proposed scheme is promising in terms of accuracy in contrast to existing techniques in the literature.
Read the full abstract ▾
All authors Abrar Alotaibi, Atta-ur Rahman, Raheel Alhaza, Wala Alkhalifa, Narjes Alhajjaj, Atheer Alharthi, Dhai Abushoumi, Maryam Alqahtani, Dania Alkhulaifi
Computation and Language
How Reliable Is Your Jailbreak Judge? Calibration and Adversarial Robustness of Automated ASR Scoring
Yang Gao
Almost every paper on LLM jailbreaks and
prompt injection reports an attack-success rate (ASR), and that number is assigned not by people but by an automated judge: either a safety classifier trained for the task, or a general chat model prompted to grade. The judge is rarely checked. We check it. Using 596 human-labeled completions from the HarmBench classifier validation set, we compare the two judge families against human majority votes and then attack them. The two families fail in opposite ways. The dedicated classifier over-flags (precision 0.835, recall 0.974); three different LLM-as-judges keep high precision (0.81 to 0.94) but show erratic recall (0.06 to 0.65), so the same responses produce very different ASR depending on which judge scores them. The two families also differ sharply in robustness. Wrappers that leave the harmful text untouched and only add benign framing flip every LLM-judge between 57% and 100% of the time, and a single prepended refusal sentence accounts for much of this (39% to 88%). The dedicated classifier resists these surface attacks (at most 6.7%), but a white-box GCG attack on its open weights flips 70% of confident true positives (21 of 30; 95% CI 54 to 86%) even at a small optimization budget. A two-annotator audit confirms the attacks leave the harm intact: every one of 80 sampled flips still contained the harmful content. Because a large and growing share of reported ASR comes from LLM-judges, many such numbers are unreliable both on average and under deliberate pressure. We recommend that papers report judge precision and recall on a human-labeled slice, report ASR corrected for judge precision, and include an
adversarial check of the judge. Our code is released.
Read the full abstract ▾
Computation and Language
A Red Teaming Framework for Large Language Models: A Case Study on Faithfulness Evaluation
Abrar Alotaibi, Raed Mughus and Moataz Ahmed
Large
language models (LLMs) have demonstrated remarkable performance across natural
language processing tasks, yet their deployment in high-stakes applications raises critical concerns regarding reliability, safety, and trustworthiness. In this paper, we present a red teaming framework that systematically uncovers vulnerabilities in LLM outputs. Our approach employs a novel multi-role architecture comprising target, attacker, and jury models. The attackers generate increasingly effective
adversarial prompts while the jury rigorously evaluates response accuracy and consistency across tasks. In a case study, our strategy proved particularly effective at exposing unfaithfulness in LLM responses. Exploitative
adversarial prompts increased the attack success rate by up to 7.9% in question-answering tasks, revealing weaknesses in reliability. The approach identifies how structural constraints in summarization can shape vulnerability patterns, with format limitations yielding measurable gains in faithfulness, and shows that architectural design choices typically outweigh parameter scaling in determining model safety. The framework's key strength is its adaptability across evaluation tasks, from English question-answering to Arabic summarization, enabling comprehensive comparison of model vulnerabilities. While it excels at comparing cross-model and cross-linguistic vulnerabilities, it faces challenges in fully automating
adversarial prompt generation across
languages. Our experiments also reveal limitations in detecting subtle forms of unfaithfulness that do not manifest as explicit factual contradictions, particularly across linguistic contexts. Overall, this architecture provides both actionable insights into current LLM vulnerabilities and a scalable methodology for ongoing safety evaluation as models evolve.
Read the full abstract ▾
Computation and Language
Optimizing Abstractive Summarization With Fine-Tuned PEGASUS
Sadiul Arefin Rafi, Naimur Rahman, Kazi Nazibul Islam et al. (+2 more)
Abstractive text summarization is the technique of generating a short and concise summary comprising the salient ideas of a source text without making a subset of the salient sentences from the source text. The introduction of
transformer models such as BART, T5, and PEGASUS has made this sort of summarization process more efficient and accurate. The objective of this paper is to fine-tune PEGASUS on the XL-Sum English corpus to achieve a better performance compared to the baseline mT5 model. The performance of the generated summaries from the fine-tuned model is evaluated using the ROUGE metric, which basically compares the auto-generated summaries with human-created summaries. To the best of our knowledge, the results from our fine-tuned PEGASUS model give a state-of-the-art performance on the XL-Sum English Corpus. To quantify the improvement, there is a 4.04% improvement in the ROUGE-1 score, a 15.25% increase in the ROUGE-2 score, and a 3.39% improvement in the ROUGE-L score from the baseline model.
Read the full abstract ▾
All authors Sadiul Arefin Rafi, Naimur Rahman, Kazi Nazibul Islam, Ha-mim Ahmad, Farig Yousuf Sadeque
Computation and Language · Cross-listed from eess.AS
Fully Differentiable Neural Forced Alignment via Soft Dynamic Programming
Rotem Rousso, Eyal Cohen and Joseph Keshet
Recent advances in sequence modeling have significantly improved ASR systems, bringing them close to human-level recognition accuracy and enhancing robustness across diverse acoustic conditions and
languages. In contrast, Forced
Alignment has not experienced comparable progress, and traditional HMM-GMM frameworks remain widely adopted and highly competitive. To address this gap, we propose an end-to-end, fully differentiable neural architecture specifically designed for phoneme
alignment. The model consists of an encoder that processes the input signal and a decoder that produces
alignment decisions. The encoder is structured into two complementary branches: one dedicated to phoneme identity verification and the other to phoneme boundary detection. The decoder is implemented as a trainable module based on differentiable soft dynamic programming. The entire system is optimized end-to-end using a novel contrastive loss that encourages clear separation between steady-state phoneme regions and transition boundaries. The proposed approach outperforms the current state of the art in phoneme
alignment on hand-annotated English
benchmarks, achieves strong word-level generalization results, and demonstrates generalization on unseen
languages.
Read the full abstract ▾
Computation and Language
Probing in the Wild: A Case Study of Self-Supervised Speech Representations on Mandarin Sub-dialects with Unsupervised Articulatory Analysis
Shu Shang, Fuliang Weng, Zeqian Hu et al. (+1 more)
While self-supervised
speech models have achieved strong performance across
speech tasks, relatively little is known about how their internal phonetic representations behave under fine-grained dialect variation. Existing probing studies typically rely on curated corpora with manual phonetic annotations, limiting their applicability to naturally occurring dialect
speech. We present a case study of articulatory feature representations in a Mandarin self-supervised
speech model using an entirely unlabeled probing pipeline. Phone sequences are generated using a
language-agnostic universal phone recognizer and mapped to articulatory feature vectors, enabling frame-level probing without manual annotation. Our results reveal a structured pattern in articulatory feature decodability across Mandarin sub-dialects. Acoustically salient features such as labiality and stridency remain comparatively stable, whereas features associated with finer spectral distinctions exhibit larger dialect-dependent variation. This variation is driven primarily by elevated decodability for Beijing
speech relative to other Mandarin sub-dialects. Layer-wise analyses further show distinct representational dynamics for these feature groups. These findings suggest that
language-agnostic articulatory probing can be applied to real-world dialect corpora and that dialect sensitivity in self-supervised
speech representations is unevenly distributed across articulatory dimensions.
Read the full abstract ▾
All authors Shu Shang, Fuliang Weng, Zeqian Hu, Yaqian Zhou
Computation and Language · Cross-listed from cs.LG
The Generalization Spectrum: A Chromatographic Approach to Evaluating Learning Algorithms
Jinghan Zhang, Zerui Cheng, Shiqi Chen et al. (+5 more)
Traditional evaluations measure a learning
algorithm's final performance on an i.i.d. test set, reducing learning to a single aggregate score. This approach obscures a fundamental question: to what extent does learning from a specific example generalize to others? Such per-sample generalization, akin to learning by analogy in human cognition, captures how far the knowledge extracted from one example can transfer, yet remains invisible to standard
benchmarks. We introduce the Generalization Spectrum, an evaluation framework designed to expose this hidden dimension. For each training example, we construct a controlled suite of test variants arranged by increasing transfer distance, from exact recall to implementation transfer across
languages, context transfer under complete narrative re-framing, category-matched in-domain problems, and an unpaired baseline. By tracking performance across these distances, we reveal not just whether an
algorithm learns, but how far that learning extends. We instantiate this framework on competitive programming, using a selection-and-synthesis pipeline seeded with recent problems to mitigate contamination. We first compare three canonical learning paradigms under matched memorization. RL converts memorization into near-transfer more efficiently than SFT-family baselines, while ICL exhibits strong but correspondence-dependent transfer. We then use the Spectrum to diagnose within-family variants. The resulting profiles show that local gains need not expand the generalization radius: abstractions and hints mainly lift local transfer, RFT preserves a stronger far-transfer tail than reference SFT, and self-
distillation or hint-assisted RL can reduce far transfer even when local transfer or optimization improves.
Read the full abstract ▾
All authors Jinghan Zhang, Zerui Cheng, Shiqi Chen, Ge Zhang, Wenhao Huang, Jiashuo Liu, Junxian He, Tianle Cai