Computation and Language
Evaluating the Pre-Consultation Ability of LLMs using Diagnostic Guidelines
Jean Seo, Gibaeg Kim, Kihun Shin et al. (+5 more)
We introduce EPAG, a
benchmark dataset and framework designed for Evaluating the Pre-consultation Ability of LLMs using diagnostic Guidelines. LLMs are evaluated directly through HPI-diagnostic guideline comparison and indirectly through disease diagnosis. In our experiments, we observe that small open-source models fine-tuned with a well-curated, task-specific
dataset can outperform frontier LLMs in pre-consultation. Additionally, we find that increased amount of HPI (History of Present Illness) does not necessarily lead to improved diagnostic performance. Further experiments reveal that the
language of pre-consultation influences the characteristics of the dialogue. By open-sourcing our
dataset and evaluation pipeline on https://github.com/seemdog/EPAG, we aim to contribute to the evaluation and further development of LLM applications in real-world clinical settings.
Read the full abstract ▾
All authors Jean Seo, Gibaeg Kim, Kihun Shin, Seungseop Lim, Hyunkyung Lee, Wooseok Han, Jongwon Lee, Eunho Yang
Computation and Language
SARA: Stress Test Reasoning in Audio Deepfake Detection
Binh Nguyen, Charles Fleming and Thai Le
Audio
Language Models (ALMs) offer a promising shift towards explainable audio deepfake detections (ADD), moving beyond black-box classifiers by providing transparency to their predictions via
reasoning traces. However, such
reasoning may not support the model predictions, reflecting poor coherence, or, worse, may rationalize incorrect predictions with plausible but misleading explanation. Moreover, the behavior of ALM
reasoning under
adversarial attacks remains under-explored, raising questions about the practical reliability of such explanation capabilities. To address this gap, this study introduces SARA (Shift Analysis of
Reasoning in Audio), a diagnostic framework that evaluates ALM
reasoning across three dimensions: acoustic perception,
reasoning-verdict coherence and dissonance. We test five open-source ALMs against both acoustic and linguistic
adversarial attacks. We show that acoustic attacks significantly degrade
reasoning-verdict coherence (average decrease of 14.20%), frequently inducing internal logical conflicts. Conversely, linguistic attacks achieve higher attack success rates while maintaining
reasoning coherence. We further demonstrate that the textual coherence of generated
reasoning traces also serves as a latent indicator of
adversarial inputs, enabling effective detection of perturbed audio (0.78 in F1) without accessing the raw acoustic signal. These findings suggest that
reasoning traces provide diagnostic utility that persists even when final classification outputs are compromised.
Read the full abstract ▾
Computation and Language
AEScorer: An Agentic Evidence-Grounded Framework for Graded Factuality Verification
Hui Huang, Muyun Yang and Yuki Arase
Despite the significant advancements of Large
Language Models (LLMs), their factuality remains a critical challenge, creating a growing need for more nuanced factuality verification. Existing factuality verification methods do not capture graded judgments, even though factuality is better understood as a spectrum rather than a binary of right and wrong. To bridge this gap, we focus on graded factuality verification and propose AEScorer, an agentic evidence-grounded framework with two stages: agentic evidence acquisition and graded scoring. AEScorer first gathers and refines external evidence through agentic search, and then predicts a scalar factuality score to distinguish nuanced differences in factual correctness. We further construct GradedVeriBench, a
benchmark for graded factuality verification spanning both general and multi-hop question answering. Experimental results on GradedVeriBench show that AEScorer substantially outperforms existing methods across both settings, demonstrating the value of coupling targeted evidence acquisition with graded scoring.
Read the full abstract ▾
Computation and Language
From Chains to Graphs: Self-Structured Reasoning for General-Domain LLMs
Yingjian Chen, Haoran Liu, Yinhong Liu et al. (+7 more)
Large
Language Models (LLMs) show strong
reasoning ability in open-domain question answering, yet their
reasoning processes are typically linear and often logically inconsistent. In contrast, real-world
reasoning requires integrating multiple premises and solving subproblems in parallel. Existing methods, such as Chain-of-Thought (CoT), express
reasoning in a linear textual form, which may appear coherent but frequently leads to inconsistent conclusions. Recent approaches rely on externally provided graphs and do not explore how LLMs can construct and use their own graph-structured
reasoning, particularly in open-domain QA. To fill this gap, we novelly explore graph-structured
reasoning of LLMs in general-domain question answering. We propose Self-Graph
Reasoning (SGR), a framework that enables LLMs to explicitly represent their
reasoning process as a structured graph before producing the final answer. We further construct a graph-structured
reasoning dataset that merges multiple candidate
reasoning graphs into refined graph structures for model training. Experiments on five QA
benchmarks across both general and specialized domains show that SGR consistently improves
reasoning consistency and yields a 17.74% gain over the base model. The LLaMA-3.3-70B model fine-tuned with SGR performs comparably to GPT-4o and surpasses Claude-3.5-Haiku, demonstrating the effectiveness of graph-structured
reasoning.
Read the full abstract ▾
All authors Yingjian Chen, Haoran Liu, Yinhong Liu, Sherry T. Tong, Aosong Feng, Jinghui Lu, Juntao Zhang, Yusuke Iwasawa, Yutaka Matsuo, Irene Li
Computation and Language · Cross-listed from cs.AI
Controllable LLM Reasoning via Sparse Autoencoder-Based Steering
Yi Fang, Wenjie Wang, Mingfeng Xue et al. (+4 more)
Large
Reasoning Models (LRMs) exhibit human-like cognitive
reasoning strategies (\eg backtracking, cross-verification) during the
reasoning process, which improves their performance on complex tasks. Currently,
reasoning strategies are autonomously selected by LRMs themselves. However, such autonomous selection often produces inefficient or even erroneous
reasoning paths. To make
reasoning more reliable and flexible, it is important to develop methods for controlling
reasoning strategies. Existing methods struggle to control fine-grained
reasoning strategies due to conceptual
entanglement in LRMs' hidden states. To address this, we leverage Sparse Autoencoders (SAEs) to decompose strategy-entangled hidden states into a disentangled feature space. To identify the few strategy-specific features from the vast pool of SAE features, we propose SAE-Steering, an efficient two-stage feature identification pipeline. SAE-Steering first recalls features that amplify the logits of strategy-specific keywords, filtering out over 99% of features, and then ranks the remaining features by their control effectiveness. Using the identified strategy-specific features as control vectors, SAE-Steering outperforms existing methods by over 15% in control effectiveness. Furthermore, controlling
reasoning strategies can redirect LRMs from erroneous paths to correct ones, achieving a 7% absolute accuracy improvement. Our code and data are available at https://github.com/Peter-Fy/SAE-Steering.
Read the full abstract ▾
All authors Yi Fang, Wenjie Wang, Mingfeng Xue, Boyi Deng, Fengli Xu, Dayiheng Liu, Fuli Feng
Computation and Language
OLA: Output Language Alignment in Code-Switched LLM Interactions
Juhyun Oh, Haneul Yoo, Faiz Ghifari Haznitrama et al. (+1 more)
Code-switching, alternating between
languages within a conversation, is natural for
multilingual users, yet poses fundamental challenges for large
language models (LLMs). When a user code-switches in their
prompt to an LLM, they typically do not specify the expected
language of the LLM response, and thus LLMs must infer the output
language from contextual and pragmatic cues. We find that current LLMs systematically fail to align with this expectation, responding in undesired
languages even when cues are clear to humans. We introduce OLA, a
benchmark to evaluate LLMs' Output
Language Alignment in code-switched interactions. OLA focuses on Korean–English code-switching and spans simple intra-sentential mixing to instruction-content mismatches. Even frontier models frequently misinterpret implicit
language expectation, exhibiting a bias toward non-English responses. We further show this bias generalizes beyond Korean to Chinese and Indonesian pairs. Models also show instability through mid-response switching and
language intrusions. Chain-of-Thought prompting fails to resolve these errors, indicating weak pragmatic
reasoning about output
language. However, Code-Switching Aware DPO with minimal data (about 1K examples) substantially reduces misalignment, suggesting these failures stem from insufficient
alignment rather than fundamental limitations. Our results highlight the need to align
multilingual LLMs with users' implicit expectations in real-world code-switched interactions.
Read the full abstract ▾
All authors Juhyun Oh, Haneul Yoo, Faiz Ghifari Haznitrama, Alice Oh
Computation and Language
PsychEthicsBench: Evaluating Large Language Models Against Australian Mental Health Ethics
Yaling Shen, Stephanie Fong, Yiwen Jiang et al. (+9 more)
The increasing integration of large
language models (LLMs) into mental health applications necessitates robust frameworks for evaluating professional safety
alignment. Current evaluative approaches primarily rely on refusal-based safety signals, which offer limited insight into the nuanced behaviors required in clinical practice. In mental health, clinically inadequate refusals can be perceived as unempathetic and discourage help-seeking. To address this gap, we move beyond refusal-centric metrics and introduce PsychEthicsBench, the first principle-grounded
benchmark based on Australian psychology and psychiatry guidelines, designed to evaluate LLMs' ethical knowledge and behavioral responses through multiple-choice and open-ended tasks with fine-grained ethicality annotations. Empirical results across 14 models reveal that refusal rates are poor indicators of ethical behavior, revealing a significant divergence between safety triggers and clinical appropriateness. Notably, we find that domain-specific fine-tuning can degrade ethical robustness, as several specialized models underperform their base backbones in ethical
alignment. PsychEthicsBench provides a foundation for systematic, jurisdiction-aware evaluation of LLMs in mental health, encouraging more responsible development in this domain.
Read the full abstract ▾
All authors Yaling Shen, Stephanie Fong, Yiwen Jiang, Zimu Wang, Feilong Tang, Qingyang Xu, Xiangyu Zhao, Zhongxing Xu, Jiahe Liu, Jinpeng Hu, Dominic Dwyer, Zongyuan Ge
Computation and Language
How Do Large Language Models Learn Concepts During Continual Pre-Training?
Barry Menglong Yao, Sha Li, Yunzhi Yao et al. (+4 more)
Human beings primarily understand the world through concepts (e.g., dog), abstract mental representations that structure perception,
reasoning, and learning. However, how large
language models (LLMs) acquire, retain, and forget such concepts during continual pretraining remains poorly understood. In this work, we study how individual concepts are acquired and forgotten, as well as how multiple concepts interact through interference and synergy. We link these behavioral dynamics to LLMs' internal concept
circuits, computational subgraphs associated with specific concepts, and incorporate graph metrics to characterize
circuit topology. Our analysis reveals: (1) LLMs concept
circuits provide a non-trivial, consistent signal of concept learning and forgetting; (2) concept
circuits exhibit a stage-wise temporal pattern during continual pretraining, with an early increase followed by gradual decrease and stabilization; (3) concepts with larger learning gains tend to exhibit greater forgetting under subsequent training; (4) semantically similar concepts induce stronger interference than weakly related ones; (5) conceptual knowledge differs in their transferability, with some significantly facilitating the learning of others. Together, our findings provide a
circuit-level view of concept learning dynamics and motivate concept-aware training strategies, such as
Circuit-aware Experience Replay, which uses
circuit topology to prioritize concepts vulnerable to forgetting.
Read the full abstract ▾
All authors Barry Menglong Yao, Sha Li, Yunzhi Yao, Minqian Liu, Zaishuo Xia, Qifan Wang, Lifu Huang
Computation and Language
DiffCoT: Diffusion-styled Chain-of-Thought Reasoning in LLMs
Shidong Cao, Hongzhan Lin, Yuxuan Gu et al. (+2 more)
Chain-of-Thought (CoT)
reasoning improves multi-step mathematical problem solving in large
language models but remains vulnerable to exposure bias and error accumulation, as early mistakes propagate irreversibly through autoregressive decoding. In this work, we propose DiffCoT, a
diffusion-styled CoT framework that reformulates CoT
reasoning as an iterative denoising process. DiffCoT integrates
diffusion principles at the
reasoning-step level via a sliding-window mechanism, enabling unified generation and retrospective correction of intermediate steps while preserving
token-level autoregression. To maintain causal consistency, we further introduce a causal
diffusion noise schedule that respects the temporal structure of
reasoning chains. Extensive experiments on three multi-step CoT
reasoning benchmarks across diverse model backbones demonstrate that DiffCoT consistently outperforms existing CoT preference optimization methods, yielding improved robustness and error-correction capability in CoT
reasoning.
Read the full abstract ▾
All authors Shidong Cao, Hongzhan Lin, Yuxuan Gu, Ziyang Luo, Jing Ma
Computation and Language
Evaluating LLMs for Police Decision-Making: A Framework Based on Police Action Scenarios
Sangyub Lee, Heedou Kim and Hyeoncheol Kim
The use of Large
Language Models (LLMs) in police operations is growing, yet an evaluation framework tailored to police operations remains absent. While LLM's responses may not always be legally incorrect, their unverified use still can lead to severe issues such as unlawful arrests and improper evidence collection. To address this, we propose PAS (Police Action Scenarios), a systematic framework covering the entire evaluation process. Applying this framework, we constructed a novel QA
dataset from over 8,000 official documents and established key metrics validated through statistical analysis with police expert judgements. Experimental results show that commercial LLMs struggle with our new police-related tasks, particularly in providing fact-based recommendations. This study highlights the necessity of an expandable evaluation framework to ensure reliable AI-driven police operations. We release our data and
prompt template.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.CV
FEA-SLT: A Gloss-Free End-to-End Framework for Facial-Expression-Aware Sign Language Translation
Guobin Tu and Di Weng
Sign
Language Translation (SLT) is a challenging cross-modal task requiring joint modeling of manual articulations and non-manual signals. Existing gloss-free SLT methods effectively capture gestural dynamics but often underutilize facial expressions, which play crucial grammatical and disambiguating roles. This limitation can cause semantic degradation when distinct concepts share similar manual configurations. To address this issue, we propose FEA-SLT (**F**acial-**E**xpression-**A**ware **S**ign **L**anguage **T**ranslation), a gloss-free end-to-end framework that uses facial dynamics to provide complementary cues to manual signals. FEA-SLT employs a domain-transferred facial encoder to extract expression-sensitive representations and integrates them with manual features through a linguistically motivated *Facial-Expression-Aware Fusion* (FEAF) module. FEAF captures reciprocal dependencies between manual and facial channels via bidirectional modulation, enhancing syntactic fidelity. Experiments on PHOENIX14T and CSL-Daily show that FEA-SLT achieves state-of-the-art BLEU performance among gloss-free methods, while targeted analyses support improved translation of facial-sensitive utterances. Code is available at https://github.com/TuGuobin/FEA-SLT.
Read the full abstract ▾
Computation and Language
Value-Action Alignment in Large Language Models under Privacy-Prosocial Conflict
Guanyu Chen, Chenxiao Yu and Xiyang Hu
Large
language models (LLMs) are increasingly used to simulate decision-making tasks involving personal data sharing, where
privacy concerns and prosocial motivations can push choices in opposite directions. Existing evaluations often measure
privacy-related attitudes or sharing intentions in isolation, which makes it difficult to determine whether a model's expressed values jointly predict its downstream data-sharing actions as in real human behaviors. We introduce a context-based assessment protocol that sequentially administers standardized questionnaires for
privacy attitudes, prosocialness, and acceptance of data sharing within a bounded, history-carrying session. To evaluate value-action
alignments under competing attitudes, we use multi-group structural equation modeling (MGSEM) to identify relations from
privacy concerns and prosocialness to data sharing. We propose Value-Action
Alignment Rate (VAAR), a human-referenced directional agreement metric that aggregates path-level evidence for expected signs. Across multiple LLMs, we observe stable but model-specific
Privacy-PSA-AoDS profiles, and substantial heterogeneity in value-action
alignment.
Read the full abstract ▾
Computation and Language
EvolMem: A Cognitive-Driven Benchmark for Multi-Session Dialogue Memory
Ye Shen, Dun Pei, Yiqiu Guo et al. (+6 more)
Despite recent advances in understanding and leveraging long-range conversational
memory, existing
benchmarks still lack systematic evaluation of large
language models(LLMs) across diverse
memory dimensions, particularly in multi-session settings. In this work, we propose EvolMem, a new
benchmark for assessing multi-session
memory capabilities of LLMs and
agent systems. EvolMem is grounded in cognitive psychology and encompasses both declarative and non-declarative
memory, further decomposed into multiple fine-grained abilities. To construct the
benchmark, we introduce a hybrid data synthesis framework that consists of topic-initiated generation and narrative-inspired transformations. This framework enables scalable generation of multi-session conversations with controllable complexity, accompanied by sample-specific evaluation guidelines. Extensive evaluation reveals that no LLM consistently outperforms others across all
memory dimensions. Moreover,
agent memory mechanisms do not necessarily enhance LLMs' capabilities and often exhibit notable efficiency limitations. Data and code will be released at https://github.com/shenye7436/EvolMem.
Read the full abstract ▾
All authors Ye Shen, Dun Pei, Yiqiu Guo, Junying Wang, Yijin Guo, Zicheng Zhang, Qi Jia, Jun Zhou, Guangtao Zhai
Computation and Language
Layer-Order Inversion: Rethinking Latent Multi-Hop Reasoning in Large Language Models
Xukai Liu, Ye Liu, Jipeng Zhang et al. (+3 more)
Large
language models (LLMs) perform well on multi-hop
reasoning, yet how they internally compose multiple facts remains unclear. Recent work proposes hop-aligned
circuit hypothesis, suggesting that bridge entities are computed sequentially across layers before later-hop answers. Through systematic analyses on real-world multi-hop queries, we show that this hop-aligned assumption does not generalize: later-hop answer entities can become decodable earlier than bridge entities, a phenomenon we call layer-order inversion, which strengthens with total hops. To explain this behavior, we propose a probabilistic recall-and-extract framework that models multi-hop
reasoning as broad probabilistic recall in shallow MLP layers followed by selective extraction in deeper
attention layers. This framework is empirically validated through systematic probing analyses, reinterpreting prior layer-wise decoding evidence, explaining chain-of-thought gains, and providing a mechanistic diagnosis of multi-hop failures despite correct single-hop knowledge. Code is available at https://github.com/laquabe/Layer-Order-Inversion.
Read the full abstract ▾
All authors Xukai Liu, Ye Liu, Jipeng Zhang, Yanghai Zhang, Kai Zhang, Qi Liu
Computation and Language
DeepSynth-Eval: Objectively Evaluating Information Consolidation in Deep Survey Writing
Hongzhi Zhang, Yuanze Hu, Tinghai Zhang et al. (+9 more)
The
evolution of Large
Language Models (LLMs) towards autonomous
agents has catalyzed progress in Deep Research. While
retrieval capabilities are well-benchmarked, the post-
retrieval synthesis stage–where
agents must digest massive amounts of context and consolidate fragmented evidence into coherent, long-form reports–remains under-evaluated due to the subjectivity of open-ended writing. To bridge this gap, we introduce DeepSynth-Eval, a
benchmark designed to objectively evaluate information consolidation capabilities. We leverage high-quality survey papers as gold standards, reverse-engineering research requests and constructing "Oracle Contexts" from their bibliographies to isolate synthesis from
retrieval noise. We propose a fine-grained evaluation protocol using General Checklists (for factual coverage) and Constraint Checklists (for structural organization), transforming subjective judgment into verifiable metrics. Experiments across 96 tasks reveal that synthesizing information from hundreds of references remains a significant challenge. Our results demonstrate that agentic plan-and-write workflows significantly outperform single-turn generation, effectively reducing
hallucinations and improving adherence to complex structural constraints.
Read the full abstract ▾
All authors Hongzhi Zhang, Yuanze Hu, Tinghai Zhang, Jia Fu, Tao Wang, Junwei Jing, Zhaoxin Fan, Qi Wang, Ruiming Tang, Han Li, Guorui Zhou, Kun Gai
Computation and Language · Cross-listed from cs.AI
STAR-S: Improving Safety Alignment through Self-Taught Reasoning on Safety Rules
Di Wu, Yanyan Zhao, Xin Lu et al. (+2 more)
Defending against jailbreak attacks is crucial for the safe deployment of Large
Language Models (LLMs). Recent research has attempted to improve safety by training models to reason over safety rules before responding. However, a key issue lies in determining what form of safety
reasoning effectively defends against jailbreak attacks, which is difficult to explicitly design or directly obtain. To address this, we propose
STAR-S (Self-TAught
Reasoning based on Safety rules), a framework that integrates the learning of safety rule
reasoning into a self-taught loop. The core of
STAR-S involves eliciting
reasoning and reflection guided by safety rules, then leveraging fine-tuning to enhance safety
reasoning. Repeating this process creates a synergistic cycle. Improvements in the model's
reasoning and interpretation of safety rules allow it to produce better
reasoning data under safety rule
prompts, which is then utilized for further training. Experiments show that
STAR-S effectively defends against jailbreak attacks, outperforming baselines. Code is available at: https://github.com/pikepokenew/
STAR_S.git.
Read the full abstract ▾
All authors Di Wu, Yanyan Zhao, Xin Lu, Mingzhe Li, Bing Qin
Computation and Language
Persona-aware and Explainable Bikeability Assessment: A Vision-Language Model Approach
Yilong Dai, Ziyi Wang, Chenguang Wang et al. (+4 more)
Bikeability assessment is essential for advancing sustainable urban transportation and creating cyclist-friendly cities, and it requires incorporating users' perceptions of safety and comfort. Yet existing perception-based bikeability assessment approaches face key limitations in capturing the complexity of road environments and adequately accounting for heterogeneity in subjective user perceptions. This paper proposes a persona-aware
Vision-
Language Model framework for bikeability assessment with three novel contributions: (i) theory-grounded persona conditioning based on established cyclist typology that generates persona-specific explanations via chain-of-thought
reasoning; (ii) multi-granularity supervised fine-tuning that combines scarce expert-annotated
reasoning with abundant user ratings for joint prediction and explainable assessment; and (iii) AI-enabled data augmentation that creates controlled paired data to isolate infrastructure variable impacts. To test and validate this framework, we developed a panoramic image-based crowdsourcing system and collected 12,400 persona-conditioned assessments from 427 cyclists. Experiment results show that the proposed framework offers competitive bikeability rating prediction while uniquely enabling explainable factor attribution.
Read the full abstract ▾
All authors Yilong Dai, Ziyi Wang, Chenguang Wang, Kexin Zhou, Yiheng Qian, Susu Xu, Xiang Yan
Computation and Language
PALM-Bench: A Comprehensive Benchmark for Personalized Audio-Language Models
Yuwen Wang, Xinyuan Qian, Tian-Hao Zhang et al. (+6 more)
Large Audio-
Language Models (LALMs) have demonstrated strong performance in audio understanding and generation. Yet, our extensive benchmarking reveals that their behavior is largely generic (e.g., summarizing spoken content) and fails to adequately support personalized question answering (e.g., summarizing what my best friend says). In contrast, human conditions their interpretation and decision-making on each individual's personal context. To bridge this gap, we formalize the task of Personalized LALMs (PALM) for recognizing personal concepts and
reasoning within personal context. Moreover, we create the first
benchmark (PALM-Bench) to foster the methodological advances in PALM and enable structured evaluation on several tasks across multi-speaker scenarios. Our extensive experiments on representative open-source LALMs, show that existing training-free prompting and supervised fine-tuning strategies, while yield improvements, remains limited in modeling personalized knowledge and transferring them across tasks robustly. Data and code will be released.
Read the full abstract ▾
All authors Yuwen Wang, Xinyuan Qian, Tian-Hao Zhang, Jiaran Gao, Yuchen Pan, Xin Wang, Zhou Pan, Chen Wei, Yiming Wang
Computation and Language
Mem-Gallery: Benchmarking Multimodal Long-Term Conversational Memory for MLLM Agents
Yuanchen Bei, Tianxin Wei, Xuying Ning et al. (+7 more)
Long-term
memory is a critical capability for
multimodal large
language model (MLLM)
agents, particularly in conversational settings where information accumulates and evolves over time. However, existing
benchmarks either evaluate multi-session
memory in text-only conversations or assess
multimodal understanding within localized contexts, failing to evaluate how
multimodal memory is preserved, organized, and evolved across long-term conversational trajectories. Thus, we introduce Mem-Gallery, a new
benchmark for evaluating
multimodal long-term conversational
memory in MLLM
agents. Mem-Gallery features high-quality multi-session conversations grounded in both visual and textual information, with long interaction horizons and rich
multimodal dependencies. Building on this
dataset, we propose a systematic evaluation framework that assesses key
memory capabilities along three functional dimensions:
memory extraction and test-time adaptation,
memory reasoning, and
memory knowledge management. Extensive benchmarking across thirteen
memory systems reveals several key findings, highlighting the necessity of explicit
multimodal information retention and
memory organization, the persistent limitations in
memory reasoning and knowledge management, as well as the efficiency bottleneck of current models.
Read the full abstract ▾
All authors Yuanchen Bei, Tianxin Wei, Xuying Ning, Yanjun Zhao, Zhining Liu, Xiao Lin, Yada Zhu, Hendrik Hamann, Jingrui He, Hanghang Tong
Computation and Language
IntroLM: Introspective Language Models via Prefilling-Time Self-Evaluation
Hossein Hosseini Kasnavieh, Gholamreza Haffari, Chris Leckie et al. (+1 more)
A major challenge for the operation of large
language models (LLMs) is how to predict whether a specific LLM will produce sufficiently high-quality output for a given query. Existing approaches rely on external classifiers, most commonly BERT based models, which suffer from limited context windows, constrained representational capacity, and additional computational overhead. We propose IntroLM, a method that enables causal
language models to predict their own output quality during the prefilling phase without affecting generation using introspective
tokens. By introducing
token conditional LoRA that activates only for the introspective
token, the model learns to predict the output quality for a given query while preserving the original backbone behavior and avoiding external evaluators. On question answering
benchmarks, IntroLM applied to Qwen3 8B achieves a ROC AUC of 90 precent for success prediction, outperforming a DeBERTa classifier by 14 precent. When integrated into multi model routing systems, IntroLM achieves superior cost performance tradeoffs, reducing latency by up to 33 precent and large model usage by up to 50 precent at matched reliability.
Read the full abstract ▾
All authors Hossein Hosseini Kasnavieh, Gholamreza Haffari, Chris Leckie, Adel N. Toosi
Computation and Language
QA-Merging: Query-Adaptive Reasoning via Layer Selective Model Merging
Zhaofeng Zhong, Wei Yuan, Tong Chen et al. (+4 more)
Recent large
reasoning models (LRMs) have achieved strong performance on complex
reasoning tasks by generating a long chain-of-thought (Long-CoT). However, such lengthy
reasoning is often unnecessary for simple queries, leading to additional computation and latency. Existing approaches to adaptive
reasoning typically rely on retraining the model or designing sophisticated prompting, which are either prohibitively expensive or highly sensitive to the
prompt formulation. Model merging provides a more balanced alternative for adaptive
reasoning by avoiding expensive training and integrating Long-CoT and Short-CoT behaviors. However, existing merging methods are often static and input-agnostic, or rely on costly all-layer calibration, which limits their effectiveness for query-adaptive
reasoning. To tackle these challenges, we propose Query-adaptive Layer Selective Merging (QA-Merging), an activation-based merging framework that integrates a Long-CoT model and a Short-CoT model to obtain a query-adaptive reasoner without training from scratch or requiring large-scale additional data. QA-Merging first constructs a small pattern-labeled calibration set that assigns each query an appropriate
reasoning pattern. Motivated by our empirical analysis that Long-CoT and Short-CoT behaviors diverge unevenly across
transformer layers, QA-Merging identifies layers with high
reasoning pattern divergence and calibrates only these layers through feature
alignment and contrastive shaping, while applying closed-form hidden-state correction to the remaining layers. Experiments on seven widely used
reasoning benchmarks across two model scales demonstrate that QA-Merging reduces inference cost and maintains strong performance.
Read the full abstract ▾
All authors Zhaofeng Zhong, Wei Yuan, Tong Chen, Liang Qu, Xiangyu Zhao, Quoc Viet Hung Nguyen, Hongzhi Yin
Computation and Language
Beyond Perplexity: A Lightweight Benchmark for Knowledge Retention in Supervised Fine-Tuning
Soheil Zibakhsh Shabgahi, Pedram Aghazadeh and Farinaz Koushanfar
Supervised Fine-Tuning (SFT) is a standard approach for injecting domain knowledge into Large
Language Models (LLMs). However, relying on validation perplexity to monitor training is often insufficient, as it confounds stylistic mimicry with genuine factual internalization. To address this, we introduce the Knowledge Retention (KR) Test , a lightweight, corpus-grounded evaluation framework designed to distinguish factual learning from linguistics. KR-Test utilizes automatically generated contrastive examples to measure likelihood preferences for correct versus incorrect continuations, requiring no instruction tuning or generative decoding. We validate the framework's integrity through a "blind vs. oracle" baseline analysis. Furthermore, we demonstrate the diagnostic capabilities of KR-Test by analyzing the training dynamics of Low-Rank Adaptation (LoRA). By exposing the fine-grained dissociation between linguistic convergence and knowledge retention, KR-Test enhances the interpretability of fine-tuning dynamics.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.IR
STELLA: Self-Reflective Terminology-Aware Framework for Building an Aerospace Information Retrieval Benchmark
Bongmin Kim
Tasks in the aerospace industry heavily rely on searching and reusing large volumes of technical documents, yet there is no public information
retrieval (IR)
benchmark that reflects the terminology- and query-intent characteristics of this domain. To address this gap, this paper proposes the STELLA (Self-Reflective TErminoLogy-Aware Framework for BuiLding an Aerospace Information
Retrieval Benchmark) framework. Using this framework, we introduce the STELLA
benchmark, an aerospace-specific IR evaluation set constructed from NASA Technical Reports Server (NTRS) documents via a systematic pipeline that comprises document layout detection, passage chunking, terminology dictionary construction, synthetic query generation, and cross-lingual extension. The framework generates two types of queries: the Terminology Concordant Query (TCQ), which includes the terminology verbatim to evaluate lexical matching, and the Terminology Agnostic Query (TAQ), which utilizes the terminology's description to assess semantic matching. This enables a disentangled evaluation of the lexical and semantic matching capabilities of
embedding models. In addition, we combine Chain-of-Density (CoD) and the Self-Reflection method with query generation to improve quality and implement a hybrid cross-lingual extension that reflects real user querying practices. Evaluation of seven
embedding models on the STELLA
benchmark shows that large decoder-based
embedding models exhibit the strongest semantic understanding, while lexical matching methods such as BM25 remain highly competitive in domains where exact lexical matching technical term is crucial. The STELLA
benchmark provides a reproducible foundation for reliable performance evaluation and improvement of
embedding models in aerospace-domain IR tasks. The STELLA
benchmark can be found in https://huggingface.co/
datasets/telepix/STELLA.
Read the full abstract ▾
Computation and Language
Submodular Evaluation Subset Selection in Automatic Prompt Optimization
Jinming Nian, Zhiyuan Peng, Hongwei Shang et al. (+2 more)
Automatic
prompt optimization reduces manual
prompt engineering, but relies on task performance measured on a small, often randomly sampled evaluation subset as its main source of feedback signal. Despite this, how to select that evaluation subset is usually treated as an implementation detail. We study evaluation subset selection for
prompt optimization from a principled perspective and propose SESS, a submodular evaluation subset selection method. We frame selection as maximizing an objective set function and show that, under mild conditions, it is monotone and submodular, enabling greedy selection with theoretical guarantees. Across GSM8K, MATH, and GPQA-Diamond, submodularly selected evaluation subsets can yield better optimized
prompts than random or heuristic baselines.
Read the full abstract ▾
All authors Jinming Nian, Zhiyuan Peng, Hongwei Shang, Dae Hoon Park, Yi Fang
Computation and Language
CALM: Culturally Self-Aware Language Models
Lingzhi Shen, Xiaohao Cai, Yunfei Long et al. (+3 more)
Cultural awareness in
language models is the capacity to understand and adapt to diverse cultural contexts. However, most existing approaches treat culture as static background knowledge, overlooking its dynamic and evolving nature. This limitation reduces their reliability in downstream tasks that demand genuine cultural sensitivity. In this work, we introduce CALM, a novel framework designed to endow
language models with cultural self-awareness. CALM disentangles task semantics from explicit cultural concepts and latent cultural signals, shaping them into structured cultural clusters through contrastive learning. These clusters are then aligned via cross-
attention to establish fine-grained interactions among related cultural features and are adaptively integrated through a Mixture-of-Experts mechanism along culture-specific dimensions. The resulting unified representation is fused with the model's original knowledge to construct a culturally grounded internal identity state, which is further enhanced through self-prompted reflective learning, enabling continual adaptation and self-correction. Extensive experiments conducted on multiple cross-cultural
benchmark datasets demonstrate that CALM consistently outperforms state-of-the-art methods.
Read the full abstract ▾
All authors Lingzhi Shen, Xiaohao Cai, Yunfei Long, Imran Razzak, Guanming Chen, Shoaib Jameel
Computation and Language
Self-Explaining Hate Speech Detection with Moral Rationales
Francielle Vargas, Jackson Trager, Diego Alves et al. (+6 more)
Existing hate
speech detection models are often opaque and rely on surface-level lexical cues, which makes them vulnerable to spurious correlations and limits robustness, interpretability and cultural contextualization. We propose Supervised Moral Rationale
Attention (SMRA), the first self-explaining hate
speech detection framework to incorporate moral rationales as direct supervision for
attention alignment. Based on Moral Foundations Theory, SMRA aligns
token-level
attention with expert-annotated moral rationales, guiding models to attend to morally salient spans. Unlike prior rationale-supervised or post-hoc approaches, SMRA integrates moral rationale supervision directly into the training objective, producing inherently interpretable and contextualized explanations. To support our framework, we also introduce HateBRMoralXplain, a Brazilian Portuguese
benchmark dataset annotated with hate labels, moral categories,
token-level moral rationales, and socio-political metadata. Across binary hate
speech detection and multi-label moral sentiment classification, SMRA consistently improves performance while enhancing both faithful and plausible explanations. Although explanations become more concise, sufficiency decreases, indicating more compact and informative rationales.
Fairness remains stable, suggesting that improvements in explanation quality do not introduce significant bias trade-offs.
Read the full abstract ▾
All authors Francielle Vargas, Jackson Trager, Diego Alves, Surendrabikram Thapa, Matteo Guida, Berk Atil, Daryna Dementieva, Andrew Smart, Ameeta Agrawal
Computation and Language
SegNSP: Revisiting Next Sentence Prediction for Linear Text Segmentation
José Isidro, Filipe Cunha, Purificação Silvano et al. (+4 more)
Linear text
segmentation is a long-standing problem in natural
language processing (NLP), focused on dividing continuous text into coherent and semantically meaningful units. Despite its importance, the task remains challenging due to the complexity of defining topic boundaries, the variability in discourse structure, and the need to balance local coherence with global context. These difficulties hinder downstream applications such as summarization, information
retrieval, and question answering. In this work, we introduce SegNSP, framing linear text
segmentation as a next sentence prediction (NSP) task. Although NSP has largely been abandoned in modern pre-training, its explicit modeling of sentence-to-sentence continuity makes it a natural fit for detecting topic boundaries. We propose a label-agnostic NSP approach, which predicts whether the next sentence continues the current topic without requiring explicit topic labels, and enhance it with a
segmentation-aware loss combined with harder negative sampling to better capture discourse continuity. Unlike recent proposals that leverage NSP alongside auxiliary topic classification, our approach avoids task-specific supervision. We evaluate our model against established baselines on two
datasets, CitiLink-Minutes, for which we establish the first
segmentation benchmark, and WikiSection. On CitiLink-Minutes, SegNSP achieves a B-$F_1$ of 0.79, closely aligning with human-annotated topic transitions, while on WikiSection it attains a B-F$_1$ of 0.65, outperforming the strongest reproducible baseline, TopSeg, by 0.17 absolute points. These results demonstrate competitive and robust performance, highlighting the effectiveness of modeling sentence-to-sentence continuity for improving
segmentation quality and supporting downstream NLP applications.
Read the full abstract ▾
All authors José Isidro, Filipe Cunha, Purificação Silvano, Alípio Jorge, Nuno Guimarães, Sérgio Nunes, Ricardo Campos