Computation and Language
Jupiter-N Technical Report
George Drayson
We present Jupiter-N, a hybrid
reasoning model post-trained from Nemotron 3 Super, a fully open-source 120 billion parameter LLM. We target three objectives: (1) agentic capability via uncertainty-curated trajectories; (2) UK cultural
alignment via synthetic data grounded in cultural norms; and (3) Welsh
language support via parallel corpora and LLM-translated Welsh conversations. Our data curation strategy carefully preserves the base model's capabilities: using our Forget-Me-Not framework, we mix on-policy synthetic replay with off-policy task data to mitigate catastrophic forgetting, and include a mixture of
reasoning and non-
reasoning traces to maintain Nemotron's hybrid
reasoning ability. Jupiter-N achieves standout gains over Nemotron in Welsh (+18 on ARC-Easy, +5.25 on MMLU-Lite), terminal-use (+9.1 on Terminal Bench 2) and instruction following (+4.4 on IFBench), while retaining the base model capabilities. We frame this work as a reproducible template for sovereign post-training: substituting cultural knowledge, institutional corpora, and target
languages produces an equivalent pipeline for any country. All model weights and all post-training
datasets are publicly released under open licences.
Read the full abstract ▾
Computation and Language
DuConTE: Dual-Granularity Text Encoder with Topology-Constrained Attention for Text-attributed Graphs
Lexuan Liang, Tao Zou, Xuxiang Ta et al. (+1 more)
Text-attributed graphs integrate semantic information of node texts with topological structure, offering significant value in various applications such as document classification and information extraction. Existing approaches typically encode textual content using
language models (LMs), followed by graph neural networks (GNNs) to process structural information. However, during the LM-based text encoding phase, most methods not only perform semantic interaction solely at the word-
token granularity, but also neglect the structural dependencies among texts from different nodes. In this work, we propose DuConTE, a dual-granularity text encoder with
topology-constrained
attention. The model employs a cascaded architecture of two pretrained LMs, encoding semantics first at the word-
token granularity and then at the node granularity. During the self-
attention computation in each LM, we dynamically adjust the
attention mask
matrix based on node connectivity, guiding the model to learn semantic correlations informed by the graph structure. Furthermore, when composing node representations from word-
token embeddings, we separately evaluate the importance of
tokens under the center-node context and the neighborhood context, enabling the capture of more contextually relevant semantic information. Extensive experiments on multiple
benchmark datasets demonstrate that DuConTE achieves state-of-the-art performance on the majority of them.
Read the full abstract ▾
All authors Lexuan Liang, Tao Zou, Xuxiang Ta, Zekun Qiu
Computation and Language
Contrastive Analysis of Linguistic Representations in Large Language Model Outputs through Structured Synthetic Data Generation and Abstracted N-gram Associations
S.A. Desimone and L. Alonso Alemany
We present a methodological framework to discover linguistic and discursive patterns associated to different social groups through contrastive synthetic text generation and statistical analysis. In contrast with previous approaches, we aim to characterize subtle expressions of bias, instead of diagnosing bias through a pre-determined list of words or expressions. We are also working with contextualized data instead of isolated words or sentences. Our methodology applies to textual productions in any genre, encompassing narrative, task-oriented or dialogic. Contextualized data are generated using controlled combinations of situational scenarios and group markers, creating minimal pairs of texts that differ only in the referenced group while maintaining comparable narrative conditions. To facilitate robust analysis, linguistic forms are generalized and associations between linguistic abstractions and groups are quantified using a variant of pointwise mutual information to detect expressions that appear disproportionately across groups. A fragment-ranking strategy then prioritizes text segments with a high concentration of biased linguistic signals, which allows for experts to assess the harmful potential of linguistic expressions in context, bridging quantitative analysis and qualitative interpretation.
Read the full abstract ▾
Computation and Language
Representation-Guided Parameter-Efficient LLM Unlearning
Zeguan Xiao, Lang Mo, Yun Chen et al. (+4 more)
Large
Language Models (LLMs) often memorize sensitive or harmful information, necessitating effective machine unlearning techniques. While existing parameter-efficient unlearning methods have shown promise, they still struggle with the forget-retain trade-off. This can be attributed to their reliance on parameter importance metrics to identify parameters that are important exclusively for the forget set, which is fundamentally limited by the superposition phenomenon. Due to the polysemantic nature of LLM parameters, such an importance metric may struggle to disentangle parameters associated with the forget and retain sets. In this work, we propose Representation-Guided Low-rank Unlearning (REGLU), a novel approach that leverages the
geometric properties of representation spaces to achieve robust and precise unlearning. First, we develop a representation-guided initialization for LoRA that identifies the optimal subspace for selective forgetting. Second, we introduce a regularization loss that constrains the outputs of the LoRA update to lie in the orthogonal complement of the retain set's representation subspace, thereby minimizing interference with the model's performance on the retain set. We evaluate REGLU on the TOFU and WMDP
benchmarks across multiple models. Our results demonstrate that REGLU consistently outperforms state-of-the-art baselines, achieving superior unlearning quality while maintaining higher model utility.
Read the full abstract ▾
All authors Zeguan Xiao, Lang Mo, Yun Chen, Lei Yang, Jiehui Zhao, Lili Yang, Guanhua Chen
Computation and Language
Who Watches the Watchmen? Humans Disagree With Translation Metrics on Unseen Domains
Finn Schmidt, Jan Philip Wahle, Terry Ruas et al. (+1 more)
Automatic evaluation metrics are central to the development of machine translation systems, yet their robustness under domain shift remains unclear. Most metrics are developed on the Workshop on Machine Translation (WMT)
benchmarks, raising concerns about their robustness to unseen domains. Prior studies that analyze unseen domains vary translation systems, annotators, or evaluation conditions, confounding domain effects with human annotation noise. To address these biases, we introduce a systematic multi-annotator Cross-Domain Error-Span-Annotation
dataset (CD-ESA), comprising 18.8k human error span annotations across three
language pairs, where we fix annotators within each
language pair and evaluate translations of the same six translation systems across one seen news domain and two unseen technical domains. Using this
dataset, we first find that automatic metrics appear surprisingly robust to domain-shifts at the segment level (up to 0.69 agreement), but this robustness largely disappears once we account for human label variation. Averaging annotations increases inter-annotator agreement by up to +0.11. Metrics struggle on the unseen chemical domain compared to humans (inter-annotator agreement of 0.78-0.83 vs. 0.96). We recommend comparing metric-human agreement against inter-annotator agreement, rather than comparing raw metric-human agreement alone, when evaluating across different domains.
Read the full abstract ▾
All authors Finn Schmidt, Jan Philip Wahle, Terry Ruas, Bela Gipp
Computation and Language
AnchorMem: Anchored Facts with Associative Contexts for Building Memory in Large Language Models
Zhanyu Shen, Sijie Cheng, Zhicheng Guo et al. (+3 more)
While large
language models have achieved remarkable performance in complex tasks, they still need a
memory system to utilize historical experience in long-term interactions. Existing
memory methods (e.g., A-Mem, Mem0) place excessive emphasis on organizing interactions by frequently rewriting them, however, this heavy reliance on summarization risks diluting essential contextual nuances and obscuring key
retrieval features. To bridge this gap, we introduce AnchorMem, a novel
memory framework inspired by the Proust Phenomenon in cognitive science, where a specific anchor triggers a holistic recollection. We propose a method that decouples the
retrieval unit from the generation context. AnchorMem extracts atomic facts from interaction history to serve as
retrieval anchors, while preserving the original context as the immutable context. To reveal implicit narrative cues, we construct an associative event graph that uses higher-order event links that bind sets of related facts into shared event representations, strengthening cross-
memory integration without relying on generic entities as bridges. During
retrieval, the system anchors queries to specific facts and events to locate relevant
memories, but reconstructs the context using the associated raw chunks and events. Our method reconciles fine-grained
retrieval with the contextual integrity of interactions. Experiments across three closed-source and open-source models on the LoCoMo
benchmark demonstrate that AnchorMem significantly outperforms baselines. Code is available at https://github.com/RayNeo-AI-2025/AnchorMem.
Read the full abstract ▾
All authors Zhanyu Shen, Sijie Cheng, Zhicheng Guo, Weiqin Wang, Yile Wang, Hui Huang
Computation and Language
ArgBench: Benchmarking LLMs on Computational Argumentation Tasks
Yamen Ajjour, Carlotta Quensel, Nedim Lipka et al. (+1 more)
Argumentation skills are an essential toolkit for large
language models (LLMs). These skills are crucial in various use cases, including self-reflection, debating collaboratively for diverse answers, and countering hate
speech. In this paper, we create the first
benchmark for a standardized evaluation of LLM-based approaches to computational argumentation, encompassing 33
datasets from previous work in unified form. Using the
benchmark, we evaluate the generalizability of five LLM families across 46 computational argumentation tasks that cover mining arguments, assessing perspectives, assessing argument quality,
reasoning about arguments, and generating arguments. On the
benchmark, we conduct an extensive systematic analysis of the contribution of few-shot examples,
reasoning steps, model size, and training skills to the performance of LLMs on the computational argumentation tasks in the
benchmark.
Read the full abstract ▾
All authors Yamen Ajjour, Carlotta Quensel, Nedim Lipka, Henning Wachsmuth
Computation and Language
Still Between Us? Evaluating and Improving Voice Assistant Robustness to Third-Party Interruptions
Dongwook Lee, Eunwoo Song, Che Hyun Lee et al. (+2 more)
While recent Spoken
Language Models (SLMs) have been actively deployed in real-world scenarios, they lack the capability to discern Third-Party Interruptions (TPI) from the primary user's ongoing flow, leaving them vulnerable to contextual failures. To bridge this gap, we introduce TPI-Train, a
dataset of 88K instances designed with speaker-aware hard negatives to enforce acoustic cue prioritization for interruption handling, and TPI-Bench, a comprehensive evaluation framework designed to rigorously measure the interruption-handling strategy and precise speaker discrimination in deceptive contexts. Experiments demonstrate that our
dataset design mitigates semantic shortcut learning-a critical pitfall where models exploit semantic context while neglecting acoustic signals essential for discerning speaker changes. We believe our work establishes a foundational resource for overcoming text-dominated unimodal reliance in SLMs, paving the way for more robust multi-party spoken interaction. The code for the framework is publicly available at https://tpi-va.github.io
Read the full abstract ▾
All authors Dongwook Lee, Eunwoo Song, Che Hyun Lee, Heeseung Kim, Sungroh Yoon
Computation and Language
More Than Meets the Eye: Measuring the Semiotic Gap in Vision-Language Models via Semantic Anchorage
Wei He
Vision-
Language Models (VLMs) excel at photorealistic generation, yet often struggle to represent abstract meaning such as idiomatic interpretations of noun compounds. To study whether high visual fidelity interferes with idiomatic compositionality under visual abstraction, we introduce DIVA, a controlled
benchmark that replaces high-fidelity visual detail with schematic iconicity by generating paired, sense-anchored visualizations for literal and idiomatic readings. We further propose Semantic
Alignment Gap ($\Delta$), an architecture-agnostic metric that quantifies divergence between literal and idiomatic visual grounding. We additionally introduce a directional signed bias $b(t)$ to separately measure the direction and strength of literal preference. Evaluating 8 recent VLMs, we reveal a consistent Literal Superiority Bias: model scale alone does not resolve literal preference, and increased visual fidelity is associated with weaker symbolic
alignment, suggesting cognitive interference from hyper-realistic imagery. Our findings suggest that improving compositional understanding requires iconographic abstraction of visual input and anchoring interpretation and generation in intended meaning.
Read the full abstract ▾
Computation and Language
Logical Computational Linguistics
Glyn V. Morrill and Oriol Valentín
In this book we promote logical computational linguistics as opposed to statistical computational linguistics. In particular, we provide a logical semantic interface. This book assembles more than twenty years of research work on type logical grammar, and adds new ideas and material. Chains of statistical dependencies of less than one hundred per cent confidence tend monotonically to zero. Chains of logical dependencies of any length maintain one hundred per cent confidence end to end. We aspire to enable perfect syntactic and semantic processing in life-critical NLP applications.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.LG
FLARE: Task-agnostic embedding model evaluation through a normalization process
Jingzhou Jiang, Yixuan Tang, Yi Yang et al. (+1 more)
When task-specific labels are not available, it becomes difficult to select an
embedding model for a specific target corpus. Existing labelless measures based on kernel
estimators or Gaussian mixes fail in high-dimensional space, resulting in unstable rankings. We propose a flow-based labelless representation
embedding evaluation (
FLARE), which utilizes normalized streams to estimate information sufficiency directly from log-likelihood and avoid distance-based density estimation. We give a finite sample boundary, indicating that the estimation error depends on the intrinsic dimension of the data
manifold rather than the original
embedding dimension. On 11
datasets and 8 embedders,
FLARE reached Spearman's $\rho$ of 0.90 under the supervised
benchmark and remained stable in high-dimensional
embeddings ($d \geq 3{,}584$) as the existing labelless baseline collapsed.
Read the full abstract ▾
All authors Jingzhou Jiang, Yixuan Tang, Yi Yang, Kar Yan Tam
Computation and Language
Neuro-Symbolic Resolution of Recommendation Conflicts in Multimorbidity Clinical Guidelines
Shiyao Xie and Jian Du
Clinical guidelines, typically developed by independent specialty societies, inherently exhibit substantial fragmentation, redundancy, and logical contradiction. These inconsistencies, particularly when applied to patients with multimorbidity, not only cause cognitive dissonance for clinicians but also introduce catastrophic noise into AI systems, rendering the standard
Retrieval-Augmented Generation (RAG) system fragile and prone to
hallucination. To address this fundamental reliability crisis, we introduce a Neuro-Symbolic framework that automates the detection of recommendation redundancies and conflicts. Our pipeline employs a multi-
agent system to translate unstructured clinical natural
language into rigorous symbolic logic
language, which is then verified by a Satisfiability (SAT) solver. By formulating a hierarchical taxonomy of logical rule interactions, we identify a critical category termed Local Conflict - a decision conflict arising from the intersection of comorbidities. Evaluating our system on a curated
benchmark of 12 authoritative SGLT2 inhibitor guidelines, we reveal that 90.6% of conflicts are Local, a structural complexity that single-disease guidelines fail to address. While state-of-the-art LLMs fail in detecting these conflicts, our neuro-symbolic approach achieves an F1 score of 0.861. This work demonstrates that logical verification must precede
retrieval, establishing a new technical standard for automated knowledge coordination in medical AI.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.SE
Precise Debugging Benchmark: Is Your Model Debugging or Regenerating?
Miaosen Chai, Wang Bill Zhu, Shangshang Wang et al. (+5 more)
Unlike code completion, debugging requires localizing faults and applying targeted edits. We observe that frontier LLMs often regenerate correct but over-edited solutions during debugging. To evaluate how far LLMs are from precise debugging, we introduce the Precise Debugging
Benchmark (PDB) framework, which automatically converts any coding
dataset into a debugging
benchmark with precision-aware evaluation. PDB generates buggy programs by synthesizing verified atomic bugs and composing them into multi-bug programs. We define two novel metrics, edit-level precision and bug-level recall, which measure how many necessary edits are made and how many bugs are resolved. We release two evaluation
benchmarks: PDB-Single-Hard on single-line bugs, and PDB-Multi on multi-line bugs. Experiments show that frontier models, such as GPT-5.1-Codex and DeepSeek-V3.2-Thinking, achieve unit-test pass rates above 76% but exhibit precision below 45%, even when explicitly instructed to perform minimal debugging. Finally, we show that iterative and agentic debugging strategies do not substantially improve precision or recall, highlighting the need to rethink post-training pipelines for coding models.
Read the full abstract ▾
All authors Miaosen Chai, Wang Bill Zhu, Shangshang Wang, Yejia Liu, Song Bian, Honghua Dong, Willie Neiswanger, Robin Jia
Computation and Language
Align Documents to Questions: Question-Oriented Document Rewriting for Retrieval-Augmented Generation
Jiaang Li, Zhendong Mao, Quan Wang et al. (+2 more)
Retrieval-Augmented Generation (RAG) enhances the factuality of Large
Language Models (LLMs) by incorporating retrieved documents and/or generated context. However, LLMs often exhibit a stylistic bias when presented with mixed contexts, favoring fluent but hallucinated generated content over factually grounded yet disorganized retrieved evidence. This phenomenon reveals that the utility of retrieved information is bottlenecked by its presentation. To bridge this gap, we propose QREAM, a style-controlled rewriter that aligns retrieved documents with a question-oriented style while preserving facts, better for LLM readers to utilize. Our framework consists of two stages: (1) QREAM-ICL, which uses stylistic seeds to guide iterative rewriting exploration; and (2) QREAM-FT, a lightweight student model distilled from denoised ICL outputs. QREAM-FT employs dual-criteria rejection sampling, filtering based on answer correctness and factual consistency to ensure high-quality supervision. QREAM seamlessly integrates into existing RAG pipelines as a plug-and-play module. Experiments demonstrate that QREAM consistently enhances advanced RAG pipelines, yielding up to 8% relative improvement with negligible latency overhead, effectively balancing question relevance with factual grounding.
Read the full abstract ▾
All authors Jiaang Li, Zhendong Mao, Quan Wang, Yuning Wan, Yongdong Zhang
Computation and Language
A Universal Avoidance Method for Diverse Multi-branch Generation
Kyeongman Park, Minha Jhang and Kyomin Jung
Modern generative models still lack human-level creativity, particularly in multi-branch diversity. Prior approaches to address this problem often incur heavy computation or strong dependency on model architecture. Therefore, we introduce UAG(Universal Avoidance Generation), a model-agnostic and computationally efficient generation strategy that penalizes similarity among previously generated outputs. Thus, UAG can enhance multi-branch diversity across both
diffusion and
transformer models, with minimal additional computation. In experiments, our method achieves up to 1.9 times higher diversity, runs 4.4 times faster, and requires only 1/64 of the FLOPs compared to state-of-the-art methods. The full code is https://anonymous.4open.science/r/2026_ACL_Universal/.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.CV
E2E-GMNER: End-to-End Generative Grounded Multimodal Named Entity Recognition
Meng Zhang, Jinzhong Ning, Xiaolong Wu et al. (+2 more)
Grounded
Multimodal Named Entity Recognition (GMNER) aims to jointly identify named entity mentions in text, predict their semantic types, and ground each entity to a corresponding visual region in an associated image. Existing approaches predominantly adopt pipeline-based architectures that decouple textual entity recognition and visual grounding, leading to error accumulation and suboptimal joint optimization. In this paper, we propose E2E-GMNER, a fully end-to-end generative framework that unifies entity recognition, semantic typing, visual grounding, and implicit knowledge
reasoning within a single
multimodal large
language model. We formulate GMNER as an instruction-tuned conditional generation task and incorporate chain-of-thought
reasoning to enable the model to adaptively determine when visual evidence or background knowledge is informative, reducing reliance on noisy cues. To further address the instability of generative bounding box prediction, we introduce Gaussian Risk-Aware Box Perturbation (GRBP), which replaces hard box supervision with probabilistically perturbed soft targets to improve robustness against annotation noise and discretization errors. Extensive experiments on the Twitter-GMNER and Twitter-FMNERG
benchmarks demonstrate that E2E-GMNER achieves highly competitive performance compared with state of the art methods, validating the effectiveness of unified end-to-end optimization and noise-aware grounding supervision. Code is available at:https://github.com/Finch-coder/E2E-GMNER
Read the full abstract ▾
All authors Meng Zhang, Jinzhong Ning, Xiaolong Wu, Hongfei Lin, Yijia Zhang
Computation and Language
Calibrated? Not for Everyone: How Sexual Orientation and Religious Markers Distort LLM Accuracy and Confidence in Medical QA
Alberto Testoni and Iacer Calixto
Safe clinical deployment of Large
Language Models (LLMs) requires not only high accuracy but also robust uncertainty calibration to ensure models defer to clinicians when appropriate. Our paper investigates how social descriptors of a patient (specifically sexual orientation and religious affiliation) distort these uncertainty signals and model accuracy. Evaluating nine general-purpose and biomedical LLMs on 2,364 medical questions and their counterfactual variants, we demonstrate that identity markers cause a "calibration crisis". "Homosexual" markers consistently trigger performance drops, and intersectional identities produce idiosyncratic, non-additive harms to calibration. Moreover, a clinician-validated case study in an open-ended generation setting confirms that these failures are not an artifact of the multiple-choice format. Our results demonstrate that the presence of social identity cues does not merely shift predictions; it affects the reliability of confidence signals, posing a significant risk to equitable care and safe deployment in confidence-based clinical workflows.
Read the full abstract ▾
Computation and Language
RoTRAG: Rule of Thumb Reasoning for Conversation Harm Detection with Retrieval-Augmented Generation
Juhyeon Lee, Wonduk Seo, Junseo Koh et al. (+3 more)
Detecting harmful content in multi turn dialogue requires
reasoning over the full conversational context rather than isolated utterances. However, most existing methods rely mainly on models internal parametric knowledge, without explicit grounding in external normative principles. This often leads to inconsistent judgments in socially nuanced contexts, limited interpretability, and redundant
reasoning across turns. To address this, we propose RoTRAG, a
retrieval augmented framework that incorporates concise human written moral norms, called Rules of Thumb (RoTs), into LLM based harm assessment. For each turn, RoTRAG retrieves relevant RoTs from an external corpus and uses them as explicit normative evidence for turn level
reasoning and final severity classification. To improve efficiency, we further introduce a lightweight binary routing classifier that decides whether a new turn requires
retrieval grounded
reasoning or can reuse existing context. Experiments on ProsocialDialog and Safety
Reasoning Multi Turn Dialogue show that RoTRAG consistently improves both harm classification and severity estimation over competitive baselines, with an average relative gain of around 40% in F1 across
benchmark datasets and an average relative reduction of 8.4% in distributional error, while reducing redundant computation without sacrificing performance.
Read the full abstract ▾
All authors Juhyeon Lee, Wonduk Seo, Junseo Koh, Seunghyun Lee, Haihua Chen, Yi Bu
Computation and Language
Cat-DPO: Category-Adaptive Safety Alignment
Tiankai Yang, Yi Nian, Xinyuan Li et al. (+6 more)
Aligning large
language models with human preferences must balance two competing goals: responding helpfully to legitimate requests and reliably refusing harmful ones. Most preference-based safety
alignment methods collapse safety into a single scalar that is applied uniformly to every preference pair. The result is a model that looks safe on average but stays relatively unsafe on a minority of harm categories. We cast safety
alignment as a per-category constrained optimization problem and derive Cat-DPO, a direct-preference-optimization
algorithm with a separate adaptive safety margin for each harm category. The margin tightens when the model still produces unsafe responses on a category and relaxes once the model catches up, so the training signal tracks each category's current difficulty rather than averaging under one global rate. Across two LLM backbones and six preference-learning baselines, Cat-DPO improves aggregate helpfulness and harmlessness and compresses per-category safety variance and the best-to-worst gap, offering a drop-in per-category refinement of direct preference safety
alignment.
Read the full abstract ▾
All authors Tiankai Yang, Yi Nian, Xinyuan Li, Ruiyao Xu, Henry Peng Zou, Kaize Ding, Xiyang Hu, Yan Liu, Yue Zhao
Computation and Language
CRISP: Compressing Redundancy in Chain-of-Thought via Intrinsic Saliency Pruning
Yangsong Lan, Hongliang Dai and Piji Li
Long Chain-of-Thought (CoT)
reasoning is pivotal for the success of recent
reasoning models but suffers from high computational overhead and latency. While prior works attempt to compress CoT via external compressor, they often fail to align with the model's internal
reasoning dynamics, resulting in the loss of critical logical steps. This paper presents Compressing Redundancy in Chain-of-Thought via Intrinsic Saliency
Pruning (CRISP), a framework that compresses CoT by exploiting the model's intrinsic saliency. Our analysis reveals a distinct phenomenon: the
reasoning termination
token [object Object] acts as an information anchor, where its
attention pattern effectively demarcates essential
reasoning from redundancy. Based on this finding, we design a policy that utilizes these intrinsic
attention signals to guide atomic compression operations. In contrast to coarse-grained
pruning strategies, CRISP strategically distills the
reasoning chain to maximize information density while preserving logical coherence. Empirical results across various backbone models and mathematical
datasets demonstrate that CRISP achieves a 50-60% reduction in
token count without compromising accuracy, effectively mitigating the efficiency bottleneck of long-context
reasoning. We open-source our implementation to facilitate further research in efficient
reasoning.
Read the full abstract ▾
Computation and Language
Beyond "I Don't Know": Evaluating LLM Self-Awareness in Discriminating Data and Model Uncertainty
Jingyi Ren, Ante Wang, Yunghwei Lai et al. (+5 more)
Reliable Large
Language Models (LLMs) should abstain when confidence is insufficient. However, prior studies often treat refusal as a generic "I don't know”, failing to distinguish input-level ambiguity (data uncertainty) from capability limitations (model uncertainty). This lack of distinction limits downstream action decisions like requesting clarification or invoking external tools. In this work, we introduce UA-Bench, a
benchmark of over 3,500 questions drawn from six
datasets spanning knowledge-intensive and
reasoning-intensive tasks, designed to evaluate explicit uncertainty attribution. An evaluation of 18 frontier LLMs shows that even state-of-the-art models struggle to reliably discriminate between data uncertainty and model uncertainty, and that high answer accuracy does not necessarily imply strong uncertainty attribution ability. To narrow this gap, we propose a lightweight data synthesis and reinforcement learning strategy. Experiments on both Qwen3-4B-Instruct-2507 and Qwen3-8B in thinking mode show that the proposed method improves uncertainty attribution while preserving answer accuracy. Our code and data are publicly available now.
Read the full abstract ▾
All authors Jingyi Ren, Ante Wang, Yunghwei Lai, Xiaolong Wang, Linlu Gong, Weitao Li, Weizhi Ma, Yang Liu
Computation and Language
Probabilistic Programs of Thought
Poorva Garg, Renato Lui Geh, Daniel Israel et al. (+3 more)
LLMs are widely used for code generation and mathematical
reasoning tasks where they are required to generate structured output. They either need to reason about code, generate code for a given specification, or reason using programs of thought. The typical approach to code generation is to
prompt the model and generate samples until an appropriate program is obtained. Within this process, sampling $n$ programs from the
language model requires $n$ GPU compute-intensive generations which becomes prohibitively expensive for larger values of $n$. In this work, we address this limitation by exposing the LLM's distribution within the generated programs themselves. We propose a novel test-time framework we dub probabilistic programs of thought to obtain more samples from the model with fewer LLM generations. Given a program generated by a model and the associated next-
token probabilities, we build a probabilistic program that compactly represents exponentially many deterministic programs. Since performing probabilistic
reasoning in this probabilistic program is much cheaper, our approach allows sampling new programs without any additional GPU compute and little CPU overhead. We instantiate our approach on
benchmarks for code generation, code understanding and mathematical
reasoning and report improvements in performance with fewer generations from the LLM.
Read the full abstract ▾
All authors Poorva Garg, Renato Lui Geh, Daniel Israel, Todd Millstein, Kyle Richardson, Guy Van den Broeck
Computation and Language
HorizonBench: Long-Horizon Personalization with Evolving Preferences
Shuyue Stella Li, Bhargavi Paranjape, Kerem Oktar et al. (+9 more)
User preferences evolve across months of interaction, and tracking them requires inferring when a stated preference has been changed by a subsequent life event. We define this problem as long-horizon personalization and observe that progress on it is limited by data availability and measurement, with no existing resource providing both naturalistic long-horizon interactions and the ground-truth provenance needed to diagnose why models fail. We introduce a data generator that produces conversations from a structured mental state graph, yielding ground-truth provenance for every preference change across 6-month timelines, and from it construct HorizonBench, a
benchmark of 4,245 items from 360 simulated users with 6-month conversation histories averaging ~4,300 turns and ~163K
tokens. HorizonBench provides a testbed for long-context modeling,
memory-augmented architectures, theory-of-mind
reasoning, and user modeling. Across 25 frontier models, the best model reaches 52.8% and most score at or below the 20% chance baseline. When these models err on evolved preferences, over a third of the time they select the user's originally stated value without tracking the updated user state. This belief-update failure persists across context lengths and expression explicitness levels, identifying state-tracking capability as the primary bottleneck for long-horizon personalization.
Read the full abstract ▾
All authors Shuyue Stella Li, Bhargavi Paranjape, Kerem Oktar, Zhongyao Ma, Gelin Zhou, Lin Guan, Na Zhang, Sem Park, Lin Chen, Diyi Yang, Yulia Tsvetkov, Asli Celikyilmaz
Computation and Language
MedPRMBench: A Fine-grained Benchmark for Process Reward Models in Medical Reasoning
Xiang Zheng, Lingyan Wu, Weiqi Zhai et al. (+4 more)
Medical
reasoning can reach a plausible answer while overlooking an unsafe or unsupported intermediate step. Process-Level Reward Models (PRMs) can verify such steps, yet existing PRM
benchmarks largely target mathematics rather than patient-specific clinical errors. We introduce MedPRMBench, a step-level medical
reasoning benchmark with 14 clinical error types, including five new and two clinically re-scoped types, and rule-defined severity grades. Built from seven medical QA sources via structured error construction and review by 86 physicians from Grade-A tertiary hospitals, it provides 6{,}500 evaluation questions, 13{,}000
reasoning chains, and 113{,}910 step labels, plus a leakage-clean 6{,}870-question training split. The best of 20 prompted critics reaches only 75.4% PRMScore, and critics differ sharply in error-type profiles and operating points. A Qwen3-8B PRM trained on the split reaches 86.6% PRMScore and 89.0% strict first-error localization; in matched ablations, physician review raises error recall by 6.55 points and cuts clean-chain false alarms from 5.6% to 2.1%. The training signal replicates across model families; without further training, the PRM transfers to ProcessBench and PRMBench, exceeding six of nine math-trained PRMs on each.
Read the full abstract ▾
All authors Xiang Zheng, Lingyan Wu, Weiqi Zhai, Wei Wang, Xuan Ren, Zifan Zhang, Bing Zhao
Computation and Language
HopRank: Self-Supervised LLM Preference-Tuning on Graphs for Few-Shot Node Classification
Ziqing Wang and Kaize Ding
Node classification on text-attributed graphs (TAGs) is a fundamental task with broad applications in citation analysis, social networks, and recommendation systems. Current GNN-based approaches suffer from shallow text encoding and heavy dependence on labeled data, limiting their effectiveness in label-scarce settings. While large
language models (LLMs) naturally address the text understanding gap with deep semantic
reasoning, existing LLM-for-graph methods either still require abundant labels during training or fail to exploit the rich structural signals freely available in graph
topology. Our key observation is that, in many real-world TAGs, edges predominantly connect similar nodes under the homophily principle, meaning graph
topology inherently encodes class structure without any labels. Building on this insight, we reformulate node classification as a link prediction task and present HopRank, a fully self-supervised LLM-tuning framework for TAGs. HopRank constructs preference data via hierarchical hop-based sampling and employs adaptive preference learning to prioritize informative training signals without any class labels. At inference, nodes are classified by predicting their connection preferences to labeled anchors, with an adaptive early-exit voting scheme to improve efficiency. Experiments on three TAG
benchmarks show that HopRank matches fully-supervised GNNs and substantially outperforms prior graph-LLM methods, despite using zero labeled training data.
Read the full abstract ▾
Computation and Language
Rethinking Meeting Effectiveness: A Benchmark and Framework for Temporal Fine-grained Automatic Meeting Effectiveness Evaluation
Yihang Li and Chenhui Chu
Evaluating meeting effectiveness is crucial for improving organizational productivity. Current approaches rely on post-hoc surveys that yield a single coarse-grained score for an entire meeting. The reliance on manual assessment is inherently limited in scalability, cost, and reproducibility. Moreover, a single score fails to capture the dynamic nature of collaborative discussions. We propose a new paradigm for evaluating meeting effectiveness centered on novel criteria and temporal fine-grained approach. We define effectiveness as the rate of objective achievement over time and assess it for individual topical segments within a meeting. To support this task, we introduce the AMI Meeting Effectiveness (AMI-ME)
dataset, a new meta-evaluation
dataset containing 2,459 human-annotated segments from 130 AMI Corpus meetings. We also develop an automatic effectiveness evaluation framework that uses a Large
Language Model (LLM) as a judge to score each segment's effectiveness relative to the overall meeting objectives. Through substantial experiments, we establish a comprehensive
benchmark for this new task and evaluate the framework's generalizability across distinct meeting types, ranging from business scenarios to unstructured discussions. Furthermore, we
benchmark end-to-end performance starting from raw
speech to measure the capabilities of a complete system. Our results validate the framework's effectiveness and provide strong baselines to facilitate future research in meeting analysis and multi-party dialogue. Our
dataset and code will be publicly available. The AMI-ME
dataset and the Automatic Evaluation Framework are available at https://github.com/ku-nlp/AMI-ME.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.IR
HORIZON: A Benchmark for In-the-wild User Behaviour Modeling
Arnav Goel, Pranjal A Chitale, Bhawna Paliwal et al. (+2 more)
User behavior in the real world is diverse, cross-domain, and spans long time horizons. Existing user modeling
benchmarks however remain narrow, focusing mainly on short sessions and next-item prediction within a single domain. Such limitations hinder progress toward robust and generalizable user models. We present HORIZON, a new
benchmark that reformulates user modeling along three axes i.e.
dataset, task, and evaluation. Built from a large-scale, cross-domain reformulation of Amazon Reviews, HORIZON covers 54M users and 35M items, enabling both pretraining and realistic evaluation of models in heterogeneous environments. Unlike prior
benchmarks, it challenges models to generalize across domains, users, and time, moving beyond standard missing-positive prediction in the same domain. We propose new tasks and evaluation setups that better reflect real-world deployment scenarios. These include temporal generalization, sequence-length variation, and modeling unseen users, with metrics designed to assess general user behavior understanding rather than isolated next-item prediction. We
benchmark popular sequential recommendation architectures alongside LLM-based baselines that leverage long-term interaction histories. Our results highlight the gap between current methods and the demands of real-world user modeling, while establishing HORIZON as a foundation for research on temporally robust, cross-domain, and general-purpose user models.
Read the full abstract ▾
All authors Arnav Goel, Pranjal A Chitale, Bhawna Paliwal, Bishal Santra, Amit Sharma
Computation and Language
REZE: Representation Regularization for Domain-adaptive Text Embedding Pre-finetuning
Seungmin Lee, Jeonghwan Lee, Hyunkuk Lim et al. (+2 more)
Recent text
embedding models are often adapted to specialized domains via contrastive pre-finetuning (PFT) on a naive collection of scattered, heterogeneous tasks. However, this approach often introduces task-induced bias alongside domain knowledge, leading to uncontrolled representation shifts that distort the pretrained
embedding geometry and cause substantial performance degradation. To address this issue, we propose REZE, a representation regularization framework that explicitly controls representation shift during
embedding pre-finetuning. REZE operates on the relations of anchor-positive pairs and decomposes them in an eigenspace. It then measures task-wise dispersion along each eigencomponent to identify task-variant directions and applies adaptive soft-shrinkage to suppress task-induced noise while preserving task-invariant semantic structure, without inference-time overhead. Experiments across multiple
embedding backbones and specialized
benchmarks show that REZE outperforms standard pre-finetuning and isotropy-oriented post-hoc regularization in most settings, remaining stable where existing PFT variants collapse.
Embedding space analyses further confirm that REZE induces controlled shifts aligned with the original
embedding manifold, underscoring representation shift control as a key principle for robust
embedding pre-finetuning under heterogeneous supervision.
Read the full abstract ▾
All authors Seungmin Lee, Jeonghwan Lee, Hyunkuk Lim, Sejoon Kim, Mingi Sung
Computation and Language
Are Emotion and Rhetoric Neurons in LLM? Neuron Recognition and Adaptive Masking for Emotion-Rhetoric Prediction Steering
Li Zheng, Xin Zhang, Shuyi He et al. (+5 more)
Accurate comprehension and controllable generation of
emotion and rhetoric are pivotal for enhancing the
reasoning capabilities of large
language models (LLMs). Existing studies mostly rely on external optimizations, lacking in-depth exploration of internal representation mechanisms, thus failing to achieve fine-grained steering at the
neuron level. A handful of works on
neurons are confined to
emotions, neglecting rhetoric
neurons and their intrinsic connections. Traditional
neuron masking also exhibits counterintuitive phenomena, making reliable verification of
neuron functionality infeasible. To address these issues, we systematically investigate the
neurons representation mechanisms and inherent associations of 6
emotion categories and 4 core rhetorical devices. We propose a
neuron identification framework that integrates multi-dimensional screening, and design an adaptive masking method incorporating dynamic filtering, attenuation masking, and feedback optimization, enabling reliable causal validation of
neuron functionality.Through
neuron regulation, we achieve directed induction of non-target sentences and enhancement of
emotion tasks via rhetoric
neurons. Experiments on 5 commonly used
datasets validate the effectiveness of our method, providing a novel paradigm for the fine-grained steering of
emotion and rhetoric expressions in LLMs.
Read the full abstract ▾
All authors Li Zheng, Xin Zhang, Shuyi He, Fei Li, Chong Teng, Jiangming Yang, Donghong Ji, Zhuang Li
Computation and Language
Seeing Isn't Believing: Mitigating Belief Inertia via Active Intervention in Embodied Agents
Hanlin Wang, Chak Tou Leong, Jian Wang et al. (+1 more)
Recent advancements in large
language models (LLMs) have enabled
agents to tackle complex
embodied tasks through environmental interaction. However, these
agents still make suboptimal decisions and perform ineffective actions, as they often overlook critical environmental feedback that differs from their internal beliefs. Through a formal probing analysis, we characterize this as belief inertia, a phenomenon where
agents stubbornly adhere to prior beliefs despite explicit observations. To address this, we advocate active belief intervention, moving from passive understanding to active management. We introduce the Estimate-Verify-Update (EVU) mechanism, which empowers
agents to predict expected outcomes, verify them against observations through explicit
reasoning, and actively update prior beliefs based on the verification evidence. EVU is designed as a unified intervention mechanism that generates textual belief states explicitly, and can be integrated into both prompting-based and training-based
agent reasoning methods. Extensive experiments across three
embodied benchmarks demonstrate that EVU consistently yields substantial gains in task success rates. Further analyses validate that our approach effectively mitigates belief inertia, advancing the development of more robust
embodied agents. Our code is available at https://github.com/WangHanLinHenry/EVU.
Read the full abstract ▾
All authors Hanlin Wang, Chak Tou Leong, Jian Wang, Wenjie Li