Computation and Language · Cross-listed from cs.NI
Multi-Modal Data-Enhanced Foundation Models for Prediction and Control in Wireless Networks: A Survey
Han Zhang, Mohammad Farzanullah, Mohammad Ghassemi et al. (+3 more)
Foundation models (FMs) are recognized as a transformative breakthrough that has started to reshape the future of artificial intelligence (AI) across both academia and industry. The integration of FMs into wireless networks is expected to enable the development of general-purpose AI
agents capable of handling diverse network management requests and highly complex wireless-related tasks involving multi-modal data. Inspired by these ideas, this work discusses the utilization of FMs, especially multi-modal FMs in wireless networks. We focus on two important types of tasks in wireless network management: prediction tasks and control tasks. In particular, we first discuss FMs-enabled multi-modal contextual information understanding in wireless networks. Then, we explain how FMs can be applied to prediction and control tasks, respectively. Following this, we introduce the development of wireless-specific FMs from two perspectives: available
datasets for development and the methodologies used. Finally, we conclude with a discussion of the challenges and future directions for FM-enhanced wireless networks.
Read the full abstract ▾
All authors Han Zhang, Mohammad Farzanullah, Mohammad Ghassemi, Akram Bin Sediq, Ali Afana, Melike Erol-Kantarci
Computation and Language
Can Embedding Similarity Predict Cross-Lingual Transfer? A Systematic Study on African Languages
Tewodros Kederalah Idris, Prasenjit Mitra and Roald Eiselen
Cross-lingual transfer is essential for building NLP systems for low-resource African
languages, but practitioners lack reliable methods for selecting source
languages. We systematically evaluate five
embedding similarity metrics across 816 transfer experiments spanning three NLP tasks, three African-centric
multilingual models, and 12
languages from four
language families. We find that cosine gap and
retrieval-based metrics (P@1, CSLS) reliably predict transfer success ($\rho = 0.4-0.6$), while CKA shows negligible predictive power ($\rho \approx 0.1$). Critically, correlation signs reverse when pooling across models (Simpson's Paradox), so practitioners must validate per-model.
Embedding metrics achieve comparable predictive power to URIEL linguistic typology. Our results provide concrete guidance for source
language selection and highlight the importance of model-specific analysis.
Read the full abstract ▾
Computation and Language
WebAnchor: Anchoring Agent Planning to Stabilize Long-Horizon Web Reasoning
Xinmiao Yu, Liwen Zhang, Xiaocheng Feng et al. (+4 more)
Large
Language Model(LLM)-based
agents have shown strong capabilities in web information seeking, with reinforcement learning (RL) becoming a key optimization paradigm. However, planning remains a bottleneck, as existing methods struggle with long-horizon strategies. Our analysis reveals a critical phenomenon, plan anchor, where the first
reasoning step disproportionately impacts downstream behavior in long-horizon web
reasoning tasks. Current RL
algorithms, fail to account for this by uniformly distributing rewards across the trajectory. To address this, we propose Anchor-GRPO, a two-stage RL framework that decouples planning and execution. In Stage 1, the
agent optimizes its first-step planning using fine-grained rubrics derived from self-play experiences and human calibration. In Stage 2, execution is aligned with the initial plan through sparse rewards, ensuring stable and efficient tool usage. We evaluate Anchor-GRPO on four
benchmarks: BrowseComp, BrowseComp-Zh,
GAIA, and XBench-DeepSearch. Across models from 3B to 30B, Anchor-GRPO outperforms baseline GRPO and First-step GRPO, improving task success and tool efficiency. Notably, WebAnchor-30B achieves 46.0% pass@1 on BrowseComp and 76.4% on
GAIA. Anchor-GRPO also demonstrates strong scalability, getting higher accuracy as model size and context length increase.
Read the full abstract ▾
All authors Xinmiao Yu, Liwen Zhang, Xiaocheng Feng, Yong Jiang, Bing Qin, Pengjun Xie, Jingren Zhou
Computation and Language · Cross-listed from cs.LG
Prompt-Counterfactual Explanations for Generative AI System Behavior
Sofie Goethals, Foster Provost and João Sedoc
As generative AI systems become integrated into real-world applications, organizations increasingly need to be able to understand and interpret their behavior. In particular, decision-makers need to understand what causes generative AI systems to exhibit specific output characteristics. Within this general topic, this paper examines a key question: what is it about the input – the
prompt – that causes an LLM-based generative AI system to produce output that exhibits specific characteristics, such as toxicity, negative sentiment, or political bias. To examine this question, we adapt a common technique from the Explainable AI literature: counterfactual explanations. We explain why traditional counterfactual explanations cannot be applied directly to generative AI systems, due to several differences in how generative AI systems function. We then propose a flexible framework that adapts counterfactual explanations to non-deterministic, generative AI systems in scenarios where downstream classifiers can reveal key characteristics of their outputs. Based on this framework, we introduce an
algorithm for generating
prompt-counterfactual explanations (PCEs). Finally, we demonstrate the production of counterfactual explanations for generative AI systems with three case studies, examining different output characteristics (viz., political leaning, toxicity, and sentiment). The case studies further show that PCEs can streamline
prompt engineering to suppress undesirable output characteristics and can enhance red-teaming efforts to uncover additional
prompts that elicit undesirable outputs. Ultimately, this work lays a foundation for
prompt-focused interpretability in generative AI: a capability that will become indispensable as these models are entrusted with higher-stakes tasks and subject to emerging regulatory requirements for transparency and accountability.
Read the full abstract ▾
Computation and Language
Decoupling the Effect of Chain-of-Thought Reasoning: A Human Label Variation Perspective
Beiduo Chen, Tiancheng Hu, Caiqi Zhang et al. (+3 more)
Reasoning-tuned LLMs utilizing long Chain-of-Thought (CoT) excel at single-answer tasks, yet their ability to model Human Label Variation–which requires capturing probabilistic ambiguity rather than resolving it–remains underexplored. We investigate this through systematic disentanglement experiments on distribution-based tasks, employing Cross-CoT experiments to isolate the effect of
reasoning text from intrinsic model priors. We observe a distinct "decoupled mechanism": while CoT improves distributional
alignment, final accuracy is dictated by CoT content (99% variance contribution), whereas distributional ranking is governed by model priors (over 80%). Step-wise analysis further shows that while CoT's influence on accuracy grows monotonically during the
reasoning process, distributional structure is largely determined by LLM's intrinsic priors. These findings suggest that long CoT serves as a decisive LLM decision-maker for the top option but fails to function as a granular distribution calibrator for ambiguous tasks.
Read the full abstract ▾
All authors Beiduo Chen, Tiancheng Hu, Caiqi Zhang, Robert Litschko, Anna Korhonen, Barbara Plank
Computation and Language
Self-Verification is All You Need To Pass The Japanese Bar Examination
Andrew Shin
Despite rapid advances in large
language models (LLMs), achieving reliable performance on highly professional and structured examinations remains a significant challenge. The Japanese bar examination is a particularly demanding
benchmark, requiring not only advanced legal
reasoning but also strict adherence to complex answer formats that involve joint evaluation of multiple propositions. While recent studies have reported improvements by decomposing such questions into simpler true–false judgments, these approaches have not been systematically evaluated under the original exam format and scoring scheme, leaving open the question of whether they truly capture exam-level competence. In this paper, we present a self-verification model trained on a newly constructed
dataset that faithfully replicates the authentic format and evaluation scale of the exam. Our model is able to exceed the official passing score when evaluated on the actual exam scale, marking the first demonstration, to our knowledge, of an LLM passing the Japanese bar examination without altering its original question structure or scoring rules. We further conduct extensive comparisons with alternative strategies, including multi-
agent inference and decomposition-based supervision, and find that these methods fail to achieve comparable performance. Our results highlight the importance of format-faithful supervision and consistency verification, and suggest that carefully designed single-model approaches can outperform more complex systems in high-stakes professional
reasoning tasks. Our
dataset and codes are publicly available.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.DB
Accurate Table Question Answering with Accessible LLMs
Yangfan Jiang, Fei Wei, Ergute Bao et al. (+4 more)
Given a table T in a database and a question Q in natural
language, the table question answering (TQA) task aims to return an accurate answer to Q based on the content of T. Recent state-of-the-art solutions leverage large
language models (LLMs) to obtain high-quality answers. However, most rely on proprietary, large-scale LLMs with costly API access, posing a significant financial barrier. This paper instead focuses on TQA with smaller, open-weight LLMs that can run on a desktop or laptop. This setting is challenging, as such LLMs typically have weaker capabilities than large proprietary models, leading to substantial performance degradation with existing methods. We observe that a key reason for this degradation is that prior approaches often require the LLM to solve a highly sophisticated task using long, complex
prompts, which exceed the capabilities of small open-weight LLMs. Motivated by this observation, we present Orchestra, a multi-
agent approach that unlocks the potential of accessible LLMs for high-quality, cost-effective TQA. Orchestra coordinates a group of LLM
agents, each responsible for a relatively simple task, through a structured, layered workflow to solve complex TQA problems – akin to an orchestra. By reducing the
prompt complexity faced by each
agent, Orchestra significantly improves output reliability. We implement Orchestra on top of AgentScope, an open-source multi-
agent framework, and evaluate it on multiple TQA
benchmarks using a wide range of open-weight LLMs. Experimental results show that Orchestra achieves strong performance even with small- to medium-sized models. For example, with Qwen2.5-14B, Orchestra reaches 72.1% accuracy on WikiTQ, approaching the best prior result of 75.3% achieved with GPT-4; with larger Qwen, Llama, or DeepSeek models, Orchestra outperforms all prior methods and establishes new state-of-the-art results across all
benchmarks.
Read the full abstract ▾
All authors Yangfan Jiang, Fei Wei, Ergute Bao, Yaliang Li, Bolin Ding, Yin Yang, Xiaokui Xiao
Computation and Language
Limited Linguistic Diversity in Embodied AI Datasets
Selma Wanna, Agnes Luhtaru, Jonathan Salfity et al. (+4 more)
Language plays a critical role in
Vision-
Language-Action (VLA) models, yet the linguistic characteristics of the
datasets used to train and evaluate these systems remain poorly documented. In this work, we present a systematic
dataset audit of several widely used VLA corpora, aiming to characterize what kinds of instructions these
datasets actually contain and how much linguistic variety they provide. We quantify instruction
language along complementary dimensions–including lexical variety, duplication and overlap, semantic similarity, and syntactic complexity. Our analysis shows that many
datasets rely on highly repetitive, template-like commands with limited structural variation, yielding a narrow distribution of instruction forms. We position these findings as descriptive documentation of the
language signal available in current VLA training and evaluation data, intended to support more detailed
dataset reporting, more principled
dataset selection, and targeted curation or augmentation strategies that broaden
language coverage.
Read the full abstract ▾
All authors Selma Wanna, Agnes Luhtaru, Jonathan Salfity, Ryan Barron, Juston Moore, Cynthia Matuszek, Mitch Pryor
Computation and Language
Improving Indigenous Language Machine Translation with Synthetic Data and Language-Specific Preprocessing
Aashish Dhawan, Christopher Driggers-Ellis, Christan Grant et al. (+1 more)
Low-resource indigenous
languages often lack the parallel corpora required for effective neural machine translation (NMT). Synthetic data generation offers a practical strategy for mitigating this limitation in data-scarce settings. In this work, we augment curated parallel
datasets for indigenous
languages of the Americas with synthetic sentence pairs generated using a high-capacity
multilingual translation model. We fine-tune a
multilingual mBART model on curated-only and synthetically augmented data and evaluate translation quality using chrF++, the primary metric used in recent AmericasNLP shared tasks for agglutinative
languages. We further apply
language-specific preprocessing, including orthographic normalization and noise-aware filtering, to reduce corpus artifacts. Experiments on Guarani-Spanish and Quechua-Spanish translation show consistent chrF++ improvements from synthetic data augmentation, while diagnostic experiments on Aymara highlight the limitations of generic preprocessing for highly agglutinative
languages.
Read the full abstract ▾
All authors Aashish Dhawan, Christopher Driggers-Ellis, Christan Grant, Daisy Zhe Wang
Computation and Language
The Anatomy of Conversational Scams: A Topic-Based Red Teaming Analysis of Multi-Turn Interactions in LLMs
Xiangzhe Yuan, Zhenhao Zhang, Haoming Tang et al. (+1 more)
As LLMs gain persuasive capabilities through extended dialogues, they create new opportunities for studying
adversarial conversational behavior in extended interaction settings that traditional single-turn safety evaluations fail to capture. We systematically study these interactional dynamics using a controlled LLM-to-LLM simulation framework for automated red-teaming across bilingual social engineering scenarios. Evaluating eight state-of-the-art models in English and Chinese, we analyze dialogue-level outcomes, annotate attacker and defender strategy families, and model interaction dynamics between them. Results show that multi-turn
adversarial dialogues follow recurrent escalation patterns, while defensive responses frequently rely on verification, delay, and channel control. We further find statistically significant cross-model and cross-lingual differences in outcome distributions, and transition analysis reveals systematic structural variation in how defender strategies respond to attacker tactics across
languages. These findings highlight the importance of studying interactional structure in multi-turn
adversarial dialogue settings and demonstrate how controlled LLM-to-LLM simulations can support mechanistic analysis of
adversarial conversational dynamics.
Read the full abstract ▾
All authors Xiangzhe Yuan, Zhenhao Zhang, Haoming Tang, Siying Hu
Computation and Language · Cross-listed from cs.AI
Automatic Prompt Engineering with No Task Cues and No Tuning
Faisal Chowdhury, Nandana Mihindukulasooriya, Niharika S D'Souza et al. (+4 more)
This paper presents a system for automatic
prompt engineering that is much simpler in both design and application and yet as effective as the existing approaches. It requires no tuning and no explicit clues about the task. We evaluated our approach on cryptic column name expansion (CNE) in database tables, a task which is critical for tabular data search, access, and understanding and yet there has been very little existing work. We evaluated on
datasets in two
languages, English and German. This is the first work to report on the application of automatic
prompt engineering for the CNE task. To the best of our knowledge, this is also the first work on the application of automatic
prompt engineering for a
language other than English.
Read the full abstract ▾
All authors Faisal Chowdhury, Nandana Mihindukulasooriya, Niharika S D'Souza, Horst Samulowitz, Neeru Gupta, Tomasz Hanusiak, Michal Kapitonow
Computation and Language
ToxiGAN: Toxic Data Augmentation via LLM-Guided Directional Adversarial Generation
Peiran Li, Jan Fillies and Adrian Paschke
Augmenting toxic
language data in a controllable and class-specific manner is crucial for improving robustness in toxicity classification, yet remains challenging due to limited supervision and distributional skew. We propose ToxiGAN, a class-aware text augmentation framework that combines
adversarial generation with semantic guidance from large
language models (LLMs). To address common issues in GAN-based augmentation such as mode collapse and semantic drift, ToxiGAN introduces a two-step directional training strategy and leverages LLM-generated neutral texts as semantic ballast. Unlike prior work that treats LLMs as static generators, our approach dynamically selects neutral exemplars to provide balanced guidance. Toxic samples are explicitly optimized to diverge from these exemplars, reinforcing class-specific contrastive signals. Experiments on four hate
speech benchmarks show that ToxiGAN achieves the strongest average performance in both macro-F1 and hate-F1, consistently outperforming traditional and LLM-based augmentation methods. Ablation and sensitivity analyses further confirm the benefits of semantic ballast and directional training in enhancing classifier robustness.
Read the full abstract ▾
Computation and Language
Discovering and Causally Validating Emotion-Sensitive Neurons in Large Audio-Language Models
Xiutian Zhao, Björn Schuller and Berrak Sisman
Emotion is a central dimension of spoken communication, yet, we still lack a mechanistic account of how modern large audio-
language models (LALMs) encode it internally. We present the first
neuron-level interpretability study of
emotion-sensitive
neurons (ESNs) in LALMs and provide causal evidence supporting the existence of such units in Qwen2.5-Omni, Kimi-Audio, and Audio Flamingo 3. Across these three widely used open-source models, we compare frequency-,
entropy-, mean-deviation-, and contrast-based
neuron selectors on multiple
emotion recognition
benchmarks. Using inference-time interventions, we reveal a consistent
emotion-specific signature: deactivating
neurons selected for a given
emotion disproportionately degrades recognition of that
emotion while largely preserving other classes, whereas targeted steering amplifies these units to bias predictions toward the target
emotion. These effects arise with modest amounts of identification data and scale systematically with intervention strength. We further observe that ESNs exhibit non-uniform layer-wise clustering with partial cross-
dataset transfer. Taken together, our results offer a causal,
neuron-level account of
emotion decisions in LALMs and highlight targeted
neuron interventions as an actionable handle for controllable affective behaviors.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.LG
One Sample to Rule Them All: Extreme Data Efficiency in Multidiscipline Reasoning with Reinforcement Learning
Yiyuan Li, Zhen Huang, Yanan Wu et al. (+6 more)
The
reasoning ability of large
language models (LLMs) can be unleashed with reinforcement learning (RL) (OpenAI, 2024; DeepSeek-AI et al., 2025a; Zeng et al., 2025). The success of existing RL attempts in LLMs usually rely on high-quality samples of large volumes. In this paper, we challenge conventional assumptions about data requirements in RL for LLMs by demonstrating the effectiveness of one-shot reinforcement learning. Specifically, we introduce polymath learning, a framework for designing one training sample that elicits multidisciplinary
reasoning improvement. We present three key findings: (1) A single, strategically selected math
reasoning sample can produce significant performance improvements across multiple domains, including physics, chemistry, and biology; (2) Analysis of salient mathematical skills provides insight into the characteristics associated with effective polymath samples; and (3) An engineered synthetic sample that integrates multidisciplinary elements and broader skill coverage achieves stronger performance than naturally occurring individual samples. Across various
reasoning benchmarks, polymath learning achieves stronger performance than larger
datasets, demonstrating that
reasoning structure and skills in samples, rather than quantity, may be the key to unlock enhanced
reasoning capabilities in
language models. Our results suggest a shift, dubbed as sample engineering, toward precision engineering of samples that complements simply increasing data volume.
Read the full abstract ▾
All authors Yiyuan Li, Zhen Huang, Yanan Wu, Weixun Wang, Xuefeng Li, Yijia Luo, Wenbo Su, Bo Zheng, Pengfei Liu
Computation and Language
Who Laughs with Whom? Disentangling Influential Factors in Humor Preferences across User Clusters and LLMs
Soichiro Murakami, Hidetaka Kamigaito, Hiroya Takamura et al. (+1 more)
Humor preferences vary widely across individuals and cultures, complicating the evaluation of humor using large
language models (LLMs). In this study, we model heterogeneity in humor preferences in Oogiri, a Japanese creative response game, by clustering users with voting logs and estimating cluster-specific weights over interpretable preference factors using Bradley-Terry-Luce models. We elicit preference judgments from LLMs by prompting them to select the funnier response and found that user clusters exhibit distinct preference patterns and that the LLM results can resemble those of particular clusters. Finally, we demonstrate that, by persona prompting, LLM preferences can be directed toward a specific cluster. The scripts for data collection and analysis are publicly available to support reproducibility.
Read the full abstract ▾
All authors Soichiro Murakami, Hidetaka Kamigaito, Hiroya Takamura, Manabu Okumura
Computation and Language · Cross-listed from cs.LG
ATLAS: Verifier-Guided Adaptive Latent Activation Steering for Efficient LLM Reasoning
Tuc Nguyen and Thai Le
Recent work on activation and latent steering has demonstrated that modifying internal representations can effectively guide large
language models (LLMs) toward improved
reasoning and efficiency without updating model parameters. However, most existing approaches rely on fixed steering policies and static intervention strengths, which limit their robustness across problem instances and often result in over- or under-steering. We propose Adaptive Test-time Latent Steering (ATLAS), a lightweight framework that dynamically controls steering decisions at inference time using a trained, lightweight verifier over the latent states. Given intermediate hidden states, the verifier predicts the quality of ongoing
reasoning and adaptively selects which steering action to apply, enabling per-example and per-step adjustment with minimal overhead. ATLAS provides a unified framework for combining learned latent verification with test-time activation steering, enabling adaptive
reasoning control without additional LLM decoding or inference-time process reward model calls. Experiments on multiple mathematical and coding
reasoning benchmarks show that ATLAS consistently outperforms both vanilla decoding and fixed steering baselines, achieving higher accuracy while substantially reducing test-time
token usage. These results demonstrate that verifier-guided latent adaptation provides an effective and scalable mechanism for controlling
reasoning efficiency without sacrificing solution quality. All source code will be publicly available.
Read the full abstract ▾
Computation and Language
Faithfulness Evaluation for Decoder-only LLM Attributions with Controlled Retained Information
Xin Huang and Antoni B. Chan
Large
Language Models (LLMs) are increasingly evaluated with input attribution methods, yet comparing such explanations remains challenging. Existing soft-perturbation faithfulness metrics, such as Soft-NC and Soft-NS, can conflate attribution quality with the number of words retained during perturbation: attribution methods with larger average scores may keep more words and therefore obtain inflated scores. To address this issue, we propose $\pi$-Soft-NC and $\pi$-Soft-NS, an evaluation framework that compares attribution methods under the same expected retaining probability, thus controlling the number of retained words. We further introduce Grad-ELLM, a gradient-based attribution method tailored to autoregressive decoder-only LLMs, which combines gradient-derived channel importance with
attention-derived
token importance at each decoding step. Experiments on classification and open-generation tasks with Llama and Mistral show that Grad-ELLM achieves strong comprehensiveness-oriented faithfulness under $\pi$-Soft-NC, while there is no dominant method under $\pi$-Soft-NS. Our evaluation metric serves as a rigorous framework to compare XAI methods for LLMs, which will support progress in the field.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.LG
Audit Me If You Can: Query-Efficient Active Fairness Auditing of Black-Box LLMs
David Hartmann, Lena Pohlmann, Lelia Hanslik et al. (+3 more)
Large
Language Models (LLMs) exhibit systematic biases across demographic groups.
Auditing is proposed as an accountability tool for black-box LLM applications, but suffers from resource-intensive query access. We conceptualise
auditing as uncertainty estimation over a target
fairness metric and introduce BAFA, the Bounded Active
Fairness Auditor for query-efficient
auditing of black-box LLMs. BAFA maintains a version space of surrogate models consistent with queried scores and computes uncertainty intervals for
fairness metrics (e.g., $\Delta$ AUC) via constrained empirical risk minimisation. Active query selection narrows these intervals to reduce estimation error. We evaluate BAFA on two standard
fairness dataset case studies: CivilComments and Bias-in-Bios, comparing against stratified sampling, power sampling, and ablations. BAFA achieves target error thresholds with up to 40$\times$ fewer queries than stratified sampling (e.g., 144 vs 5,956 queries at $\varepsilon=0.02$ for CivilComments) for tight thresholds, demonstrates substantially better performance over time, and shows lower variance across runs. These results suggest that active sampling can reduce resources needed for independent
fairness auditing with LLMs, supporting continuous model evaluations.
Read the full abstract ▾
All authors David Hartmann, Lena Pohlmann, Lelia Hanslik, Noah Gießing, Bettina Berendt, Pieter Delobelle
Computation and Language
Learning to Diagnose and Correct Moral Errors in Large Language Models
Bocheng Chen, Xi Chen, Han Zi et al. (+6 more)
Enabling large
language models (LLMs) to diagnose and correct moral errors is crucial not only for ensuring their reliable deployment but also for charting a pathway toward continuous improvement. Existing research has primarily followed two directions: fine-tuning LLMs on corpora of morally appropriate texts and developing carefully designed prompting strategies to promote moral behavior, including moral self-correction and self-improvement. However, both lines of work often remain brittle, relying heavily on shallow heuristics rather than enabling LLMs to reason about why and how a given moral discourse is morally (in)appropriate. In this paper, we address this challenge by developing pragmatic inference-driven approach that help LLMs learn to diagnose and correct moral errors. Pragmatic inference is the
reasoning process through which implied meanings are derived, a central concept in linguistics. To improve the
reasoning efficiency, our approach adapt the inference procedure according to the inferential load required by different moral discourses. Empirical results demonstrate that our method not only improves LLMs' ability to diagnose and correct moral errors, but also generalizes effectively across tasks without relying on shallow heuristics.
Read the full abstract ▾
All authors Bocheng Chen, Xi Chen, Han Zi, Haitao Mao, Zimo Qi, Xitong Zhang, Yuda Chen, Kristen Johnson, Guangliang Liu
Computation and Language
Do LLMs Encode Functional Importance of Reasoning Tokens?
Janvijay Singh and Dilek Hakkani-Tür
Large
language models solve complex tasks by generating long
reasoning chains, achieving higher accuracy at the cost of increased computational cost and reduced ability to isolate functionally relevant
reasoning. Prior work on compact
reasoning shortens such chains through probabilistic sampling, heuristics, or supervision from frontier models, but offers limited insight into whether models internally encode
token-level functional importance for answer generation. We address this gap diagnostically and propose greedy
pruning, a likelihood-preserving deletion procedure that iteratively removes
reasoning tokens whose removal minimally degrades model likelihood under a specified objective, yielding length-controlled
reasoning chains. We evaluate pruned
reasoning in a
distillation framework and show that students trained on pruned chains outperform a frontier-model-supervised compression baseline at matched
reasoning lengths. Finally, our analysis reveals systematic
pruning patterns and shows that
attention scores can predict greedy
pruning ranks, further suggesting that models encode a nontrivial functional importance structure over
reasoning tokens.
Read the full abstract ▾
Computation and Language
Detecting Hallucinations in Retrieval-Augmented Generation via Semantic-level Internal Reasoning Graph
Jianpeng Hu, Yanzeng Li, Jialun Zhong et al. (+2 more)
The
Retrieval-augmented generation (RAG) system based on Large
language model (LLM) has made significant progress. It can effectively reduce factuality
hallucinations, but faithfulness
hallucinations still exist. Previous methods for detecting faithfulness
hallucinations either neglect to capture the models' internal
reasoning processes or handle those features coarsely, making it difficult for discriminators to learn. This paper proposes a semantic-level internal
reasoning graph-based method for detecting faithfulness
hallucination. Specifically, we first extend the layer-wise relevance propagation
algorithm from the
token level to the semantic level, constructing an internal
reasoning graph based on attribution vectors. This provides a more faithful semantic-level representation of dependency. Furthermore, we design a general framework based on a small pre-trained
language model to utilize the dependencies in LLM's
reasoning for training and
hallucination detection, which can dynamically adjust the pass rate of correct samples through a threshold. Experimental results demonstrate that our method achieves better overall performance compared to state-of-the-art baselines on RAGTruth and Dolly-15k.
Read the full abstract ▾
All authors Jianpeng Hu, Yanzeng Li, Jialun Zhong, Wenfa Qi, Lei Zou
Computation and Language
Temporal Graph Network: Hallucination Detection in Multi-Turn Conversation
Vidhi Rathore, Sambu Aneesh and Himanshu Singh
Hallucinations can be produced by conversational AI systems, particularly in multi-turn conversations where context changes and contradictions may eventually surface. By representing the entire conversation as a temporal graph, we present a novel graph-based method for detecting dialogue-level
hallucinations. Our framework models each dialogue as a node, encoding it using a sentence
transformer. We explore two different ways of connectivity: i) shared-entity edges, which connect turns that refer to the same entities; ii) temporal edges, which connect contiguous turns in the conversation. Message-passing is used to update the node
embeddings, allowing flow of information between related nodes. The context-aware node
embeddings are then combined using
attention pooling into a single vector, which is then passed on to a classifier to determine the presence and type of
hallucinations. We demonstrate that our method offers slightly improved performance over existing methods. Further, we show the
attention mechanism can be used to justify the decision making process. The code and model weights are made available at: https://github.com/sambuaneesh/anlp-project.
Read the full abstract ▾
Computation and Language
Lil: Less is Less When Applying Post-Training Sparse-Attention Algorithms in Long-Decode Stage
Junhao Hu, Fangze Li, Mingtao Xu et al. (+9 more)
Large
language models (LLMs) demonstrate strong capabilities across a wide range of complex tasks and are increasingly deployed at scale, placing significant demands on inference efficiency. Prior work typically decomposes inference into prefill and decode stages, with the decode stage dominating total latency. To reduce time and
memory complexity in the decode stage, a line of work introduces sparse-
attention algorithms. In this paper, we show, both empirically and theoretically, that sparse
attention can paradoxically increase end-to-end complexity: information loss often induces significantly longer sequences, a phenomenon we term “Less is Less” (Lil). To mitigate the Lil problem, we propose an early-stopping
algorithm that detects the threshold where information loss exceeds information gain during sparse decoding. Our early-stopping
algorithm reduces
token consumption by up to 90% with a marginal accuracy degradation of less than 2% across
reasoning-intensive
benchmarks.
Read the full abstract ▾
All authors Junhao Hu, Fangze Li, Mingtao Xu, Feifan Meng, Shiju Zhao, Tiancheng Hu, Ting Peng, Anmin Liu, Wenrui Huang, Chenxu Liu, Ziyue Hua, Tao Xie
Computation and Language
BaseCal: Unsupervised Confidence Calibration via Base Model Signals
Hexiang Tan, Wanli Yang, Junwei Zhang et al. (+7 more)
Reliable confidence is essential for trusting the outputs of LLMs, yet widely deployed post-trained LLMs (PoLLMs) typically compromise this trust with severe overconfidence. In contrast, we observe that their corresponding base LLMs often remain well-calibrated. This naturally motivates us to calibrate PoLLM confidence using the base LLM as a reference. This work proposes two ways to achieve this. A straightforward solution, BaseCal-ReEval, evaluates PoLLM's responses by feeding them into the base LLM to get average probabilities as confidence. While effective, this approach introduces additional inference overhead. To address this, we propose BaseCal-Proj, which trains a lightweight projection to map the final-layer hidden states of PoLLMs back to those of their base LLMs. These projected states are then processed by the base LLM's output layer to derive base-calibrated confidence for PoLLM's responses. Notably, BaseCal is an unsupervised, plug-and-play solution that operates without human labels or LLM modifications. Experiments across five
datasets and three LLM families demonstrate the effectiveness of BaseCal, reducing Expected Calibration Error (ECE) by an average of 42.90% compared to the best unsupervised baselines.
Read the full abstract ▾
All authors Hexiang Tan, Wanli Yang, Junwei Zhang, Xin Chen, Rui Tang, Du Su, Jingang Wang, Yuanzhuo Wang, Fei Sun, Xueqi Cheng
Computation and Language
NorwAI's Large Language Models: Technical Report
Jon Atle Gulla, Peng Liu and Lemei Zhang
Norwegian, spoken by approximately five million people, remains underrepresented in many of the most significant breakthroughs in Natural
Language Processing (NLP). To address this gap, the NorLLM team at NorwAI has developed a family of models specifically tailored to Norwegian and other Scandinavian
languages, building on diverse
Transformer-based architectures such as GPT, Mistral, Llama2, Mixtral and Magistral. These models are either pretrained from scratch or continually pretrained on 25B - 88.45B
tokens, using a Norwegian-extended tokenizer and advanced post-training strategies to optimize performance, enhance robustness, and improve adaptability across various real-world tasks. Notably, instruction-tuned variants (e.g., Mistral-7B-Instruct and Mixtral-8x7B-Instruct) showcase strong assistant-style capabilities, underscoring their potential for practical deployment in interactive and domain-specific applications. The NorwAI large
language models are openly available to Nordic organizations, companies and students for both research and experimental use. This report provides detailed documentation of the model architectures, training data, tokenizer design, fine-tuning strategies, deployment, and evaluations.
Read the full abstract ▾
Computation and Language
Reducing Hallucinations in LLMs via Factuality-Aware Preference Learning
Sindhuja Chaduvula, Ahmed Y. Radwan, Azib Farooq et al. (+2 more)
Preference
alignment methods such as RLHF and Direct Preference Optimization (DPO) improve instruction following, but they can also reinforce
hallucinations when preference judgments reward fluency and confidence over factual correctness. We introduce F-DPO (Factuality-aware Direct Preference Optimization), a simple extension of DPO that uses only binary factuality labels. F-DPO (i) applies a label-flipping transformation that corrects misordered preference pairs so the chosen response is never less factual than the rejected one, and (ii) adds a factuality-aware margin that emphasizes pairs with clear correctness differences, while reducing to standard DPO when both responses share the same factuality. We construct factuality-aware preference data by augmenting DPO pairs with binary factuality indicators and synthetic hallucinated variants. Across seven open-weight LLMs (1B-14B), F-DPO consistently improves factuality and reduces
hallucination rates relative to both base models and standard DPO. On Qwen3-8B, F-DPO reduces
hallucination rates by 5x(from 0.424 to 0.084) while improving factuality scores by 50% (from 5.26 to 7.90). F-DPO also generalizes to out-of-distribution
benchmarks: on TruthfulQA, Qwen2.5-14B achieves +17% MC1 accuracy (0.500 to 0.585) and +49% MC2 accuracy (0.357 to 0.531). F-DPO requires no auxiliary reward model,
token-level annotations, or multi-stage training.
Read the full abstract ▾
All authors Sindhuja Chaduvula, Ahmed Y. Radwan, Azib Farooq, Yani Ioannou, Shaina Raza
Computation and Language
LittiChoQA: Literary Texts in Indic Languages Chosen for Question Answering
Aarya Khandelwal, Ritwik Mishra and Rajiv Ratn Shah
Long-context question answering (QA) over literary texts poses significant challenges for modern large
language models, particularly in low-resource
languages. We address the scarcity of long-context QA resources for Indic
languages by introducing LittiChoQA, the largest literary QA
dataset to date covering many
languages spoken in the Gangetic plains of India. The
dataset comprises over 270K automatically generated question-answer pairs with a balanced distribution of factoid and non-factoid questions, generated from naturally authored literary texts collected from the open web. We evaluate multiple
multilingual LLMs on non-factoid, abstractive QA, under both full-context and context-shortened settings. Results demonstrate a clear trade-off between performance and efficiency: full-context fine-tuning yields the highest
token-level and semantic-level scores, while context shortening substantially improves throughput. Among the evaluated models, Krutrim-2 achieves the strongest performance, obtaining a semantic score of 76.1 with full context. While, in shortened context settings it scores 74.9 with answer paragraph selection and 71.4 with vector-based
retrieval. Qualitative evaluations further corroborate these findings.
Read the full abstract ▾
Computation and Language
MedDialogRubrics: A Comprehensive Benchmark and Evaluation Framework for Multi-turn Medical Consultations in Large Language Models
Lecheng Gong, Weimin Fang, Ting Yang et al. (+9 more)
Medical conversational AI (AI) plays a pivotal role in the development of safer and more effective medical dialogue systems. However, existing
benchmarks and evaluation frameworks for assessing the information-gathering and diagnostic
reasoning abilities of medical large
language models (LLMs) have not been rigorously evaluated. To address these gaps, we present MedDialogRubrics, a novel
benchmark comprising 5,200 synthetically constructed patient cases and over 60,000 fine-grained evaluation rubrics generated by LLMs and subsequently refined by clinical experts, specifically designed to assess the multi-turn diagnostic capabilities of LLM. Our framework employs a multi-
agent system to synthesize realistic patient records and chief complaints from underlying disease knowledge without accessing real-world electronic health records, thereby mitigating
privacy and data-governance concerns. We design a robust Patient
Agent that is limited to a set of atomic medical facts and augmented with a dynamic guidance mechanism that continuously detects and corrects
hallucinations throughout the dialogue, ensuring internal coherence and clinical plausibility of the simulated cases. Furthermore, we propose a structured LLM-based and expert-annotated rubric-generation pipeline that retrieves Evidence-Based Medicine (EBM) guidelines and utilizes the reject sampling to derive a prioritized set of rubric items ("must-ask" items) for each case. We perform a comprehensive evaluation of state-of-the-art models and demonstrate that, across multiple assessment dimensions, current models face substantial challenges. Our results indicate that improving medical dialogue will require advances in dialogue management architectures, not just incremental tuning of the base-model.
Read the full abstract ▾
All authors Lecheng Gong, Weimin Fang, Ting Yang, Dongjie Tao, Chunxiao Guo, Peng Wei, Bo Xie, Jinqun Guan, Zixiao Chen, Fang Shi, Jinjie Gu, Junwei Liu
Computation and Language · Cross-listed from q-bio.GN
DNACHUNKER: Learnable Tokenization for DNA Language Models
Taewon Kim, Jihwan Shin, Hyomin Kim et al. (+5 more)
DNA
language models are increasingly used to represent genomic sequence, yet their effectiveness depends critically on how raw nucleotides are converted into model inputs. Unlike natural
language, DNA offers no canonical boundaries, making fixed tokenizations a brittle design choice under shifts, indels, and local repeats. We introduce DNAChunker, a masked DNA
language model that incorporates a learnable adaptive
segmentation module to produce context-dependent, variable-length units. Building on a dynamic
segmentation procedure, DNAChunker learns to allocate finer granularity to functionally enriched regions while compressing repetitive or redundant sequence. We pretrain DNAChunker on the human reference genome and evaluate it across five
benchmarks, where it consistently improves over strong fixed-tokenization baselines. Further analyses and ablations indicate that unlike fixed tokenizations,
segmentation is learned in a biologically-informed, mutation-resilient manner.
Read the full abstract ▾
All authors Taewon Kim, Jihwan Shin, Hyomin Kim, Youngmok Jung, Jonghoon Lee, Won-Chul Lee, Sungsoo Ahn, Insu Han
Computation and Language
Dementia-R1: Reinforced Pretraining and Reasoning from Unstructured Clinical Notes for Real-World Dementia Prognosis
Choonghan Kim, Hyunmin Hwang, Hangeol Chang et al. (+4 more)
While Large
Language Models (LLMs) have shown strong performance on clinical text understanding, they struggle with longitudinal prediction tasks such as dementia prognosis, which require
reasoning over complex, non-monotonic symptom trajectories across multiple visits. Standard supervised training lacks explicit annotations for symptom
evolution, while direct Reinforcement Learning (RL) is hindered by sparse binary rewards. To address this challenge, we introduce Dementia-R1, an RL-based framework for longitudinal dementia prognosis from unstructured clinical notes. Our approach adopts a Cold-Start RL strategy that pre-trains the model to predict verifiable clinical indices extracted from patient histories, enhancing the capability to reason about disease progression before determining the final clinical status. Extensive experiments show that Dementia-R1 achieves the best overall performance on the AMC real-world unstructured cohort, reaching an AUROC of 84.02% and outperforming models up to 10x larger. The framework also generalizes to Parkinson's disease dementia prediction in an independent hospital cohort, achieving an AUROC of 78.37%. On the ADNI
benchmark, our 7B model attains the highest AUROC among all LLM baselines at 83.17%, demonstrating strong longitudinal
reasoning over fluctuating cognitive trajectories. Code is available at https://anonymous.4open.science/r/dementiar1-CDB5.
Read the full abstract ▾
All authors Choonghan Kim, Hyunmin Hwang, Hangeol Chang, Jaemin Kim, Jinse Park, Jae-Sung Lim, Jong Chul Ye