Computation and Language
How Does Prefix Matter in Reasoning Model Tuning?
Raj Vardhan Tomar, Preslav Nakov and Yuxia Wang
Recent
alignment studies commonly remove introductory boilerplate phrases from supervised fine-tuning (SFT)
datasets. This work challenges that assumption. We hypothesize that safety- and
reasoning-oriented prefix sentences serve as lightweight
alignment signals that can guide model decoding toward safer and more coherent responses. To examine this, we fine-tune three R1 series models across three core model capabilities:
reasoning (mathematics, coding), safety, and factuality, systematically varying prefix inclusion from 0% to 100%. Results show that prefix-conditioned SFT improves both safety and
reasoning performance, yielding up to +6% higher Safe@1 accuracy on
adversarial benchmarks (WildJailbreak, StrongReject) and +7% improvement on GSM8K
reasoning. However, factuality and coding tasks show marginal or negative effects, indicating that prefix-induced narrowing of the search space benefits structured
reasoning.
Token-level loss analysis further reveals that prefix
tokens such as "revised" and "logically" incur higher gradient magnitudes, acting as
alignment anchors that stabilize
reasoning trajectories. Our findings suggest that prefix conditioning offers a scalable and interpretable mechanism for improving
reasoning safety, serving as an implicit form of
alignment that complements traditional reward-based methods.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.CY
The Gray Area: Characterizing Moderator Disagreement on Reddit
Shayan Alipour, Shruti Phadke, Seyed Shahabeddin Mousavi et al. (+3 more)
Volunteer moderators play a crucial role in sustaining online dialogue, but they often disagree about what should or should not be allowed. In this paper, we study the complexity of content moderation with a focus on disagreements between moderators, which we term the “gray area” of moderation. Leveraging 5 years and 4.3 million moderation log entries from 24 subreddits of different topics and sizes, we characterize how gray area, or disputed cases, differ from undisputed cases. We show that one-in-seven moderation cases are disputed among moderators, often addressing transgressions where users' intent is not directly legible, such as in trolling and brigading, as well as tensions around community governance. This is concerning, as almost half of all gray area cases involved automated moderation decisions. Through information-theoretic evaluations, we demonstrate that gray area cases are inherently harder to adjudicate than undisputed cases and show that state-of-the-art
language models struggle to adjudicate them. We highlight the key role of expert human moderators in overseeing the moderation process and provide insights about the challenges of current moderation processes and tools.
Read the full abstract ▾
All authors Shayan Alipour, Shruti Phadke, Seyed Shahabeddin Mousavi, Amirhossein Afsharrad, Morteza Zihayat, Mattia Samory
Computation and Language
Steerability of Instrumental-Convergence Tendencies in LLMs
Jakub Hoscilowicz
We examine two properties of AI systems: capability (what a system can do) and steerability (how reliably one can shift behavior toward intended outcomes). A central question is whether capability growth reduces steerability and risks control collapse. We also distinguish between authorized steerability (builders reliably reaching intended behaviors) and unauthorized steerability (attackers eliciting disallowed behaviors). This distinction highlights a fundamental safety–security dilemma of AI models: safety requires high steerability to enforce control (e.g., stop/refuse), while security requires low steerability for malicious actors to elicit harmful behaviors. This tension presents a significant challenge for open-weight models, which currently exhibit high steerability via common techniques like fine-tuning or
adversarial attacks. Using Qwen3 and InstrumentalEval, we find that a short anti-instrumental
prompt suffix sharply reduces the measured convergence rate (e.g., shutdown avoidance, self-replication). For Qwen3-30B Instruct, the convergence rate drops from 81.69% under a pro-instrumental suffix to 2.82% under an anti-instrumental suffix. Under anti-instrumental prompting, larger aligned models show lower convergence rates than smaller ones (Instruct: 2.82% vs. 4.23%; Thinking: 4.23% vs. 9.86%). Code is available at github.com/j-hoscilowicz/instrumental_steering.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.IR
OpenNovelty: An LLM-powered Agentic System for Verifiable Scholarly Novelty Assessment
Ming Zhang, Kexin Tan, Yueyuan Huang et al. (+20 more)
Evaluating novelty is critical yet challenging in peer review, as reviewers must assess submissions against a vast, rapidly evolving literature. This report presents OpenNovelty, an LLM-powered agentic system for transparent, evidence-based novelty analysis. The system operates through four phases: (1) extracting the core task and contribution claims to generate
retrieval queries; (2) retrieving relevant prior work based on extracted queries via semantic search engine; (3) constructing a hierarchical taxonomy of core-task-related work and performing contribution-level full-text comparisons against each contribution; and (4) synthesizing all analyses into a structured novelty report with explicit citations and evidence snippets. Unlike naive LLM-based approaches, OpenNovelty grounds all assessments in retrieved real papers, ensuring verifiable judgments. We deploy our system on 500+ ICLR 2026 submissions with all reports publicly available on our website, and preliminary analysis suggests it can identify relevant prior work, including closely related papers that authors may overlook. OpenNovelty aims to empower the research community with a scalable tool that promotes fair, consistent, and evidence-backed peer review.
Read the full abstract ▾
All authors Ming Zhang, Kexin Tan, Yueyuan Huang, Yujiong Shen, Chunchun Ma, Li Ju, Xinran Zhang, Yuhui Wang, Wenqing Jing, Jingyi Deng, Huayu Sha, Binze Hu, Jingqi Tong, Changhao Jiang, Yage Geng, Yuankai Ying, Yue Zhang, Zhangyue Yin, Zhiheng Xi, Shihan Dou, Tao Gui, Qi Zhang, Xuanjing Huang
Computation and Language
HalluZig: Hallucination Detection using Zigzag Persistence
Shreyas N. Samaga, Gilberto Gonzalez Arroyo and Tamal K. Dey
The factual reliability of Large
Language Models (LLMs) remains a critical barrier to their adoption in high-stakes domains due to their propensity to hallucinate. Current detection methods often rely on surface-level signals from the model's output, overlooking the failures that occur within the model's internal
reasoning process. In this paper, we introduce a new paradigm for
hallucination detection by analyzing the dynamic
topology of the
evolution of model's layer-wise
attention. We model the sequence of
attention matrices as a zigzag graph filtration and use zigzag persistence, a tool from Topological Data Analysis, to extract a topological signature. Our core hypothesis is that factual and hallucinated generations exhibit distinct topological signatures. We validate our framework, HalluZig, on multiple
benchmarks, demonstrating that it outperforms strong baselines. Furthermore, our analysis reveals that these topological signatures are generalizable across different models and
hallucination detection is possible only using structural signatures from partial network depth.
Read the full abstract ▾
Computation and Language
Bridging the Data Gap: Creating a Hindi Text Summarization Dataset from the English XSUM
Praveenkumar Katwe, RakeshChandra Balabantaray and Kaliprasad Vittala
Current advancements in Natural
Language Processing (NLP) have largely favored resource-rich
languages, leaving a significant gap in high-quality
datasets for low-resource
languages like Hindi. This scarcity is particularly evident in text summarization, where the development of robust models is hindered by a lack of diverse, specialized corpora. To address this disparity, this study introduces a cost-effective, automated framework for creating a comprehensive Hindi text summarization
dataset. By leveraging the English Extreme Summarization (XSUM)
dataset as a source, we employ advanced translation and linguistic adaptation techniques. To ensure high fidelity and contextual relevance, we utilize the Crosslingual Optimized Metric for Evaluation of Translation (COMET) for validation, supplemented by the selective use of Large
Language Models (LLMs) for curation. The resulting
dataset provides a diverse, multi-thematic resource that mirrors the complexity of the original XSUM corpus. This initiative not only provides a direct tool for Hindi NLP research but also offers a scalable methodology for democratizing NLP in other underserved
languages. By reducing the costs associated with
dataset creation, this work fosters the development of more nuanced, culturally relevant models in computational linguistics.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.AI
Aletheia: Quantifying Cognitive Conviction in Reasoning Models via Regularized Inverse Confusion Matrix
Fanzhe Fu
In the progressive journey toward Artificial General Intelligence (AGI), current evaluation paradigms face an epistemological crisis. Static
benchmarks measure knowledge breadth but fail to quantify the depth of belief. While Simhi et al. (2025) defined the CHOKE phenomenon in standard QA, we extend this framework to quantify "Cognitive Conviction" in System 2
reasoning models. We propose Project Aletheia, a cognitive physics framework that employs Tikhonov Regularization to invert the judge's confusion
matrix. To validate this methodology without relying on opaque private data, we implement a Synthetic Proxy Protocol. Our preliminary pilot study on 2025 baselines (e.g., DeepSeek-R1, OpenAI o1) suggests that while
reasoning models act as a "cognitive buffer," they may exhibit "Defensive OverThinking" under
adversarial pressure. Furthermore, we introduce the Aligned Conviction Score (S_aligned) to verify that conviction does not compromise safety. This work serves as a blueprint for measuring AI scientific integrity.
Read the full abstract ▾
Computation and Language
EmoHarbor: Evaluating Personalized Emotional Support by Simulating the User's Internal World
Jing Ye, Lu Xiang, Yaping Zhang et al. (+1 more)
Current evaluation paradigms for emotional support conversations tend to reward generic empathetic responses, yet they fail to assess whether the support is genuinely personalized to users' unique psychological profiles and contextual needs. We introduce EmoHarbor, an automated evaluation framework that adopts a User-as-a-Judge paradigm by simulating the user's inner world. EmoHarbor employs a Chain-of-
Agent architecture that decomposes users' internal processes into three specialized roles, enabling
agents to interact with supporters and complete assessments in a manner similar to human users. We instantiate this
benchmark using 100 real-world user profiles that cover a diverse range of personality traits and situations, and define 10 evaluation dimensions of personalized support quality. Comprehensive evaluation of 20 advanced LLMs on EmoHarbor reveals a critical insight: while these models excel at generating empathetic responses, they consistently fail to tailor support to individual user contexts. This finding reframes the central challenge, shifting research focus from merely enhancing generic empathy to developing truly user-aware emotional support. EmoHarbor provides a reproducible and scalable framework to guide the development and evaluation of more nuanced and user-aware emotional support systems.
Read the full abstract ▾
All authors Jing Ye, Lu Xiang, Yaping Zhang, Chengqing Zong
Computation and Language · Cross-listed from cs.AI
Bayesian Orchestration of Multi-LLM Agents for Cost-Aware Sequential Decision-Making
Danial Amin
Large
language models (LLMs) are increasingly deployed as autonomous decision
agents in settings with asymmetric error costs: hiring (missed talent vs wasted interviews), medical triage (missed emergencies vs unnecessary escalation), and fraud detection (approved fraud vs declined legitimate payments). The dominant design queries a single LLM for a posterior over states, thresholds "confidence," and acts; we prove this is inadequate for sequential decisions with costs. We propose a
Bayesian, cost-aware multi-LLM orchestration framework that treats LLMs as approximate likelihood models rather than classifiers. For each candidate state, we elicit likelihoods via contrastive prompting, aggregate across diverse models with robust statistics, and update beliefs with Bayes rule under explicit priors as new evidence arrives. This enables coherent belief updating, expected-cost action selection, principled information gathering via value of information, and
fairness gains via ensemble bias mitigation. In resume screening with costs of 40000 USD per missed hire, 2500 USD per interview, and 150 USD per phone screen, experiments on 1000 resumes using five LLMs (GPT-4o, Claude 4.5 Sonnet, Gemini Pro, Grok, DeepSeek) reduce total cost by 294000 USD (34 percent) versus the best single-LLM baseline and improve demographic parity by 45 percent (max group gap 22 to 5 percentage points). Ablations attribute 51 percent of savings to multi-LLM aggregation, 43 percent to sequential updating, and 20 percent to disagreement-triggered information gathering, consistent with the theoretical benefits of correct probabilistic foundations.
Read the full abstract ▾
Computation and Language
From Failure to Mastery: Generating Hard Samples for Tool-use Agents
Bingguang Hao, Zengzhuang Xu, Yuntao Wen et al. (+11 more)
The advancement of LLM
agents with tool-use capabilities requires diverse and complex training corpora. Existing data generation methods, which predominantly follow a paradigm of random sampling and shallow generation, often yield simple and homogeneous trajectories that fail to capture complex, implicit logical dependencies. To bridge this gap, we introduce HardGen, an automatic agentic pipeline designed to generate hard tool-use training samples with verifiable
reasoning. Firstly, HardGen establishes a dynamic API Graph built upon
agent failure cases, from which it samples to synthesize hard traces. Secondly, these traces serve as conditional priors to guide the instantiation of modular, abstract advanced tools, which are subsequently leveraged to formulate hard queries. Finally, the advanced tools and hard queries enable the generation of verifiable complex Chain-of-Thought (CoT), with a closed-loop evaluation feedback steering the continuous refinement of the process. Extensive evaluations demonstrate that a 4B parameter model trained with our curated
dataset achieves superior performance compared to several leading open-source and closed-source competitors (e.g., GPT-5.2, Gemini-3-Pro and Claude-Opus-4.5). Our code, models, and
dataset will be open-sourced to facilitate future research.
Read the full abstract ▾
All authors Bingguang Hao, Zengzhuang Xu, Yuntao Wen, Xinyi Xu, Yang Liu, Tong Zhao, Maolin Wang, Long Chen, Dong Wang, Yicheng Chen, Cunyin Peng, Xiangyu Zhao, Chenyi Zhuang, Ji Zhang
Computation and Language
Distortion Instead of Hallucination: The Effect of Reasoning Under Strict Constraints
Junichiro Niimi
With the widespread adoption of large
language models (LLMs),
hallucinations, which are non-factual fabrications in model outputs, have become serious concerns.
Reasoning capabilities have received
attention as a self-verification process to improve output reliability. However, the effect of
reasoning within a closed system where LLMs cannot rely on external tools or knowledge has yet to be clarified. We therefore conduct experiments under strict constraints (recommending peer-reviewed journal articles in computer science) to examine the effect of
reasoning across multiple models (GPT-5.2 and Gemini 3 Flash). Our results reveal a problematic trade-off between constraint compliance and factual accuracy. Non-
reasoning models exhibit high constraint violation rates (66-75%) but maintain factual accuracy, while
reasoning models reduce violations (13-26%) but systematically distort known facts to satisfy constraints and increase complete fabrication. This trade-off pattern is consistent across both models despite different architectures, indicating a fundamental limitation of
reasoning. Furthermore,
reasoning does not uniformly improve output authenticity: effects diverge by model, reflecting different allocations of the compliance-truthfulness trade-off. These findings challenge the assumption that
reasoning universally improves reliability:
reasoning models trade honest constraint violations for detection-resistant distortions.
Read the full abstract ▾
Computation and Language
Four Quadrants of Difficulty: A Simple Categorisation and its Limits
Vanessa Toborek, Sebastian Müller and Christian Bauckhage
Curriculum Learning (CL) aims to improve the outcome of model training by estimating the difficulty of samples and scheduling them accordingly. In NLP, difficulty is commonly approximated using task-agnostic linguistic heuristics or human intuition, implicitly assuming that these signals correlate with what neural models find difficult to learn. We propose a four-quadrant categorisation of difficulty signals – human vs. model and task-agnostic vs. task-dependent – and systematically analyse their interactions on a natural
language understanding
dataset. We find that task-agnostic features behave largely independently and that only task-dependent features align. These findings challenge common CL intuitions and highlight the need for lightweight, task-dependent difficulty
estimators that better reflect model learning behaviour.
Read the full abstract ▾
Computation and Language
Can Legislation Be Made Machine-Readable in PROLEG?
May-Myo Zin, Sabine Wehnert, Yuntao Kong et al. (+7 more)
The anticipated positive social impact of regulatory processes requires both the accuracy and efficiency of their application. Modern artificial intelligence technologies, including natural
language processing and machine-assisted
reasoning, hold great promise for addressing this challenge. We present a framework to address the challenge of tools for regulatory application, based on current state-of-the-art (SOTA) methods for natural
language processing (large
language models or LLMs) and formalization of legal
reasoning (the legal representation system PROLEG). As an example, we focus on Article 6 of the European General Data Protection Regulation (GDPR). In our framework, a single LLM
prompt simultaneously transforms legal text into if-then rules and a corresponding PROLEG encoding, which are then validated and refined by legal domain experts. The final output is an executable PROLEG program that can produce human-readable explanations for instances of GDPR decisions. We describe processes to support the end-to-end transformation of a segment of a regulatory document (Article 6 from GDPR), including the prompting frame to guide an LLM to "compile" natural
language text to if-then rules, then to further "compile" the vetted if-then rules to PROLEG. Finally, we produce an instance that shows the PROLEG execution. We conclude by summarizing the value of this approach and note observed limitations with suggestions to further develop such technologies for capturing and deploying regulatory frameworks.
Read the full abstract ▾
All authors May-Myo Zin, Sabine Wehnert, Yuntao Kong, Ha-Thanh Nguyen, Wachara Fungwacharakorn, Jieying Xue, Michał Araszkiewicz, Randy Goebel, Ken Satoh, Le-Minh Nguyen
Computation and Language
Bridging the gap: A comparative exploration of Speech-LLM and end-to-end architecture for multilingual conversational ASR
Yuxiang Mei, Dongxing Xu, Jiaen Liang et al. (+1 more)
The INTERSPEECH 2025 Challenge on
Multilingual Conversational
Speech Language Models (MLC-SLM) promotes
multilingual conversational ASR with large
language models (LLMs). Our previous SHNU-mASR system adopted a competitive parallel-
speech-encoder architecture that integrated Whisper and mHuBERT with an LLM. However, it faced two challenges: simple feature concatenation may not fully exploit complementary information, and the performance gap between LLM-based ASR and end-to-end(E2E) encoder-decoder ASR remained unexplored. In this work, we present an enhanced LLM-based ASR framework that combines fine-tuned Whisper and mHuBERT encoders with an LLM to enrich
speech representations. We first evaluate E2E Whisper models with LoRA and full fine-tuning on the MLC-SLM ASR task, and then propose cross-
attention-based fusion mechanisms for the parallel-
speech-encoder. On the official evaluation set of the MLC-SLM Challenge, our system achieves a CER/WER of 10.69%, ranking on par with the top-ranked Track 1 systems, even though it uses only 1,500 hours of baseline training data compared with their large-scale training sets. Nonetheless, we find that our final LLM-based ASR still does not match the performance of a fine-tuned E2E Whisper model, providing valuable empirical guidance for future
Speech-LLM design. Our code is publicly available at https://github.com/1535176727/MLC-SLM.
Read the full abstract ▾
All authors Yuxiang Mei, Dongxing Xu, Jiaen Liang, Yanhua Long
Computation and Language
Segmentation and Processing of German Court Decisions from Open Legal Data
Harshil Darji, Martin Heckelmann, Christina Kratsch et al. (+1 more)
The availability of structured legal data is important for advancing Natural
Language Processing (NLP) techniques for the German legal system. One of the most widely used
datasets, Open Legal Data, provides a large-scale collection of German court decisions. While the metadata in this raw
dataset is consistently structured, the decision texts themselves are inconsistently formatted and often lack clearly marked sections. Reliable separation of these sections is important not only for rhetorical role classification but also for downstream tasks such as
retrieval and citation analysis. In this work, we introduce a cleaned and sectioned
dataset of 251,038 German court decisions derived from the official Open Legal Data
dataset. We systematically separated three important sections in German court decisions, namely Tenor (operative part of the decision), Tatbestand (facts of the case), and Entscheidungsgründe (judicial
reasoning), which are often inconsistently represented in the original
dataset. To ensure the reliability of our extraction process, we used Cochran's formula with a 95% confidence level and a 5% margin of error to draw a statistically representative random sample of 384 cases, and manually verified that all three sections were correctly identified. We also extracted the Rechtsmittelbelehrung (appeal notice) as a separate field, since it is a procedural instruction and not part of the decision itself. The resulting corpus is publicly available in the JSONL format, making it an accessible resource for further research on the German legal system.
Read the full abstract ▾
All authors Harshil Darji, Martin Heckelmann, Christina Kratsch, Gerard de Melo
Computation and Language
iFlip: Iterative Feedback-driven Counterfactual Example Refinement
Yilong Wang, Qianli Wang and Nils Feldhus
Counterfactual examples are minimal edits to an input that alter a model's prediction. They are widely employed in explainable AI to probe model behavior and in natural
language processing (NLP) to augment training data. However, generating valid counterfactuals with large
language models (LLMs) remains challenging, as existing single-pass methods often fail to induce reliable label changes, neglecting LLMs' self-correction capabilities. To explore this untapped potential, we propose iFlip, an iterative refinement approach that leverages three types of feedback, including model confidence, feature attribution, and natural
language. Our results show that iFlip achieves an average 57.8% higher validity than the five state-of-the-art baselines, as measured by the label flipping rate. The user study further corroborates that iFlip outperforms baselines in completeness, overall satisfaction, and feasibility. In addition, ablation studies demonstrate that three components are paramount for iFlip to generate valid counterfactuals: leveraging an appropriate number of iterations, pointing to highly attributed words, and early stopping. Finally, counterfactuals generated by iFlip enable effective counterfactual data augmentation, substantially improving model performance and robustness.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.SE
SWE-Lego: Pushing the Limits of Supervised Fine-tuning for Software Issue Resolving
Chaofan Tao, Jierun Chen, Yuxin Jiang et al. (+11 more)
We present SWE-Lego, a supervised fine-tuning (SFT) recipe designed to achieve state-ofthe-art performance in software engineering (SWE) issue resolving. In contrast to prevalent methods that rely on complex training paradigms (e.g., mid-training, SFT, reinforcement learning, and their combinations), we explore how to push the limits of a lightweight SFT-only approach for SWE tasks. SWE-Lego comprises three core building blocks, with key findings summarized as follows: 1) the SWE-Lego
dataset, a collection of 32k highquality task instances and 18k validated trajectories, combining real and synthetic data to complement each other in both quality and quantity; 2) a refined SFT procedure with error masking and a difficulty-based curriculum, which demonstrably improves action quality and overall performance. Empirical results show that with these two building bricks alone,the SFT can push SWE-Lego models to state-of-the-art performance among open-source models of comparable size on SWE-bench Verified: SWE-Lego-Qwen3-8B reaches 42.2%, and SWE-Lego-Qwen3-32B attains 52.6%. 3) We further evaluate and improve test-time scaling (TTS) built upon the SFT foundation. Based on a well-trained verifier, SWE-Lego models can be significantly boosted–for example, 42.2% to 49.6% and 52.6% to 58.8% under TTS@16 for the 8B and 32B models, respectively.
Read the full abstract ▾
All authors Chaofan Tao, Jierun Chen, Yuxin Jiang, Kaiqi Kou, Shaowei Wang, Ruoyu Wang, Xiaohui Li, Sidi Yang, Yiming Du, Jianbo Dai, Zhiming Mao, Xinyu Wang, Lifeng Shang, Haoli Bai
Computation and Language
From Emotion Classification to Emotional Reasoning: Enhancing Emotional Intelligence in Large Language Models
Arjhun Sreedar, Rohan Pillay and Laukik Patade
This work investigates whether synthetic emotional chain-of-thought data can improve the emotional
reasoning abilities of smaller open large
language models (LLMs). We design a multi-
agent generation pipeline that produces therapy-style conversations and converts them into structured
emotion multiple-choice questions (MCQs) with explanations. We propose that fine-tuning a variety of 7B models on this
dataset should yield substantial gains in emotional understanding and emotional awareness on EmoBench-style evaluations, suggesting that emotional
reasoning can be induced without architectural changes. Our results demonstrate that fine-tuned Mistral 7B achieves EU improvements from 10.5 to 20.5 and EA improvements from 40.5 to 60.0, validating the effectiveness of synthetic emotional
reasoning data for enhancing model capabilities in nuanced emotional tasks.
Read the full abstract ▾
Computation and Language
LANCET: Neural Intervention via Structural Entropy for Mitigating Faithfulness Hallucinations in LLMs
Chenxu Wang, Chaozhuo Li, Pengbo Wang et al. (+7 more)
Large
Language Models have revolutionized information processing, yet their reliability is severely compromised by faithfulness
hallucinations. While current approaches attempt to mitigate this issue through node-level adjustments or coarse suppression, they often overlook the distributed nature of neural information, leading to imprecise interventions. Recognizing that
hallucinations propagate through specific forward transmission pathways like an infection, we aim to surgically block this flow using precise structural analysis. To leverage this, we propose Lancet, a novel framework that achieves precise neural intervention by leveraging structural
entropy and
hallucination difference ratios. Lancet first locates
hallucination-prone
neurons via gradient-driven contrastive analysis, then maps their propagation pathways by minimizing structural
entropy, and finally implements a hierarchical intervention strategy that preserves general model capabilities. Comprehensive evaluations across
hallucination benchmark datasets demonstrate that Lancet significantly outperforms state-of-the-art methods, validating the effectiveness of our
surgical approach to neural intervention.
Read the full abstract ▾
All authors Chenxu Wang, Chaozhuo Li, Pengbo Wang, Litian Zhang, Songyang Liu, Ji Qi, Jiahui Hu, Yushan Cai, Hao Zhao, Rui Pu
Computation and Language
EternalMath: A Living Benchmark of Frontier Mathematics that Evolves with Human Discovery
Jicheng Ma, Guohua Wang, Xinhua Feng et al. (+3 more)
Current evaluations of mathematical
reasoning in large
language models (LLMs) are dominated by static
benchmarks, either derived from competition-style problems or curated through costly expert effort, resulting in limited coverage of research-level mathematics and rapid performance saturation. We propose a fully automated,
theorem-grounded pipeline for evaluating frontier mathematical
reasoning, which directly transforms recent peer-reviewed mathematical literature into executable and verifiable
reasoning tasks. The pipeline identifies constructive or quantitative results, instantiates them into parameterized problem templates, and generates deterministic solutions through execution-based verification, enabling scalable, reproducible, and continuously updatable evaluation without reliance on large-scale expert authoring. By design, this approach supports temporal extensibility, intrinsic correctness checking, and domain-specific customization across mathematical subfields. Applying this pipeline yields EternalMath, an evolving evaluation suite derived from contemporary research papers. Experiments with state-of-the-art LLMs reveal substantial performance gaps, indicating that mathematical
reasoning at the research frontier remains far from saturated and underscoring the need for evaluation methodologies that evolve in step with human mathematical discovery.
Read the full abstract ▾
All authors Jicheng Ma, Guohua Wang, Xinhua Feng, Yiming Liu, Zhichao Hu, Yuhong Liu
Computation and Language · Cross-listed from cs.SD
SAFE-QAQ: End-to-End Slow-Thinking Audio-Text Fraud Detection via Reinforcement Learning
Peidong Wang, Zhiming Ma, Xin Dai et al. (+8 more)
Existing fraud detection methods predominantly rely on transcribed text, suffering from ASR errors and missing crucial acoustic cues like vocal tone and environmental context. This limits their effectiveness against complex deceptive strategies. To address these challenges, we first propose SAFE-QAQ, an end-to-end comprehensive framework for audio-based slow-thinking fraud detection. First, the SAFE-QAQ framework eliminates the impact of transcription errors on detection performance. Secondly, we propose rule-based slow-thinking reward mechanisms that systematically guide the system to identify fraud-indicative patterns by accurately capturing fine-grained audio details, through hierarchical
reasoning processes. Besides, our framework introduces a dynamic risk assessment framework during live calls, enabling early detection and prevention of fraud. Experiments on the TeleAntiFraud-Bench demonstrate that SAFE-QAQ achieves dramatic improvements over existing methods in multiple key dimensions, including accuracy, inference efficiency, and real-time processing capabilities. Currently deployed and analyzing over 70,000 calls daily, SAFE-QAQ effectively automates complex fraud detection, reducing human workload and financial losses. Code: https://anonymous.4open.science/r/SAFE-QAQ.
Read the full abstract ▾
All authors Peidong Wang, Zhiming Ma, Xin Dai, Yongkang Liu, Shi Feng, Xiaocui Yang, Wenxing Hu, Zhihao Wang, Mingjun Pan, Li Yuan, Daling Wang
Computation and Language
Investigating the Multilingual Calibration Effects of Language Model Instruction-Tuning
Jerry Huang, Peng Lu, Qiuhao Zeng et al. (+5 more)
Ensuring that deep learning models are well-calibrated in terms of their predictive uncertainty is essential in maintaining their trustworthiness and reliability, yet despite increasing advances in foundation model research, the relationship between such large
language models (LLMs) and their calibration remains an open area of research. In this work, we look at a critical gap in the calibration of LLMs within
multilingual settings, in an attempt to better understand how the data scarcity can potentially lead to different calibration effects and how commonly used techniques can apply in these settings. Our analysis on two
multilingual benchmarks, over 29 and 42
languages respectively, reveals that even in low-resource
languages, model confidence can increase significantly after instruction-tuning on high-resource
language SFT
datasets. However, improvements in accuracy are marginal or non-existent, resulting in mis-calibration, highlighting a critical shortcoming of standard SFT for
multilingual languages. Furthermore, we observe that the use of label smoothing to be a reasonable method alleviate this concern, again without any need for low-resource SFT data, maintaining better calibration across all
languages. Overall, this highlights the importance of
multilingual considerations for both training and tuning LLMs in order to improve their reliability and
fairness in downstream use.
Read the full abstract ▾
All authors Jerry Huang, Peng Lu, Qiuhao Zeng, Yusuke Iwasawa, Yutaka Matsuo, Sarath Chandar, Edison Marrese-Taylor, Irene Li
Computation and Language
FC-CONAN: An Exhaustively Paired Dataset for Robust Evaluation of Retrieval Systems
Juan Junqueras, Florian Boudin, May-Myo Zin et al. (+5 more)
Hate
speech (HS) is a critical issue in online discourse, and one promising strategy to counter it is through the use of counter-narratives (CNs).
Datasets linking HS with CNs are essential for advancing counterspeech research. However, even flagship resources like CONAN (Chung et al., 2019) annotate only a sparse subset of all possible HS-CN pairs, limiting evaluation. We introduce FC-CONAN (Fully Connected CONAN), the first
dataset created by exhaustively considering all combinations of 45 English HS messages and 129 CNs. A two-stage annotation process involving nine annotators and four validators produces four partitions-Diamond, Gold, Silver, and Bronze-that balance reliability and scale. None of the labeled pairs overlap with CONAN, uncovering hundreds of previously unlabelled positives. FC-CONAN enables more faithful evaluation of counterspeech
retrieval systems and facilitates detailed error analysis. The
dataset is publicly available.
Read the full abstract ▾
All authors Juan Junqueras, Florian Boudin, May-Myo Zin, Ha-Thanh Nguyen, Wachara Fungwacharakorn, Damián Ariel Furman, Akiko Aizawa, Ken Satoh
Computation and Language
Reasoning Over Recall: Evaluating the Efficacy of Generalist Architectures vs. Specialized Fine-Tunes in RAG-Based Mental Health Dialogue Systems
Md Abdullah Al Kafi, Raka Moni and Sumit Kumar Banshal
The deployment of Large
Language Models (LLMs) in mental health counseling faces the dual challenges of
hallucinations and lack of empathy. While the former may be mitigated by RAG (
retrieval-augmented generation) by anchoring answers in trusted clinical sources, there remains an open question as to whether the most effective model under this paradigm would be one that is fine-tuned on mental health data, or a more general and powerful model that succeeds purely on the basis of
reasoning. In this paper, we perform a direct comparison by running four open-source models through the same RAG pipeline using ChromaDB: two generalist reasoners (Qwen2.5-3B and Phi-3-Mini) and two domain-specific fine-tunes (MentalHealthBot-7B and TherapyBot-7B). We use an LLM-as-a-Judge framework to automate evaluation over 50 turns. We find a clear trend: the generalist models outperform the domain-specific ones in empathy (3.72 vs. 3.26, $p < 0.001$) in spite of being much smaller (3B vs. 7B), and all models perform well in terms of safety, but the generalist models show better contextual understanding and are less prone to overfitting as we observe in the domain-specific models. Overall, our results indicate that for RAG-based therapy systems, strong
reasoning is more important than training on mental health-specific vocabulary; i.e. a well-reasoned general model would provide more empathetic and balanced support than a larger narrowly fine-tuned model, so long as the answer is already grounded in clinical evidence.
Read the full abstract ▾
Computation and Language
FLOP-Efficient Training: Early Stopping Based on Test-Time Compute Awareness
Hossam Amer, Maryam Dialameh, Hossein Rajabzadeh et al. (+3 more)
Scaling training compute, measured in FLOPs, has long been shown to improve the accuracy of large
language models, yet training remains resource-intensive. Prior work shows that increasing test-time compute (TTC)-for example through iterative sampling-can allow smaller models to rival or surpass much larger ones at lower overall cost. We introduce TTC-aware training, where an intermediate checkpoint and a corresponding TTC configuration can together match or exceed the accuracy of a fully trained model while requiring substantially fewer training FLOPs. Building on this insight, we propose an early stopping
algorithm that jointly selects a checkpoint and TTC configuration to minimize training compute without sacrificing accuracy. To make this practical, we develop an efficient TTC evaluation method that avoids exhaustive search, and we formalize a break-even bound that identifies when increased inference compute compensates for reduced training compute. Experiments demonstrate up to 92% reductions in training FLOPs while maintaining and sometimes remarkably improving accuracy. These results highlight a new perspective for balancing training and inference compute in model development, enabling faster deployment cycles and more frequent model refreshes. Codes will be publicly released.
Read the full abstract ▾
All authors Hossam Amer, Maryam Dialameh, Hossein Rajabzadeh, Walid Ahmed, Weiwei Zhang, Yang Liu
Computation and Language · Cross-listed from cs.CY
AppellateGen: A Benchmark for Appellate Legal Judgment Generation
Hongkun Yang, Lionel Z. Wang, Wei Fan et al. (+10 more)
Legal judgment generation is a critical task in legal intelligence. However, existing research in legal judgment generation has predominantly focused on first-instance trials, relying on static fact-to-verdict mappings while neglecting the dialectical nature of appellate (second-instance) review. To address this, we introduce AppellateGen, a
benchmark for second-instance legal judgment generation comprising 7,351 case pairs. The task requires models to draft legally binding judgments by
reasoning over the initial verdict and evidentiary updates, thereby modeling the causal dependency between trial stages. We further propose a judicial Standard Operating Procedure (SOP)-based Legal Multi-
Agent System (SLMAS) to simulate judicial workflows, which decomposes the generation process into discrete stages of issue identification,
retrieval, and drafting. Experimental results indicate that while SLMAS improves logical consistency, the complexity of appellate
reasoning remains a substantial challenge for current LLMs. The
dataset and code are publicly available at: https://anonymous.4open.science/r/AppellateGen-5763.
Read the full abstract ▾
All authors Hongkun Yang, Lionel Z. Wang, Wei Fan, Yiran Hu, Lixu Wang, Chenyu Liu, Yu Zeng, Shenghong Fu, Lei Gong, Zhengxin Zhang, Haoyang Li, Jiexin Zheng, Xin Xu