Computation and Language
OnCoCo 1.0: A Public Dataset for Fine-Grained Message Classification in Online Counseling Conversations
Jens Albrecht, Robert Lehmann, Aleksandra Poltermann et al. (+3 more)
This paper presents OnCoCo 1.0, a new public
dataset for fine-grained message classification in online counseling. It is based on a new, integrative system of categories, designed to improve the automated analysis of psychosocial online counseling conversations. Existing category systems, predominantly based on Motivational Interviewing (MI), are limited by their narrow focus and dependence on
datasets derived mainly from face-to-face counseling. This limits the detailed examination of textual counseling conversations. In response, we developed a comprehensive new coding scheme that differentiates between 38 types of counselor and 28 types of client utterances, and created a labeled
dataset consisting of about 2.800 messages from counseling conversations. We fine-tuned several models on our
dataset to demonstrate its applicability. The data and models are publicly available to researchers and practitioners. Thus, our work contributes a new type of fine-grained conversational resource to the
language resources community, extending existing
datasets for social and mental-health dialogue analysis.
Read the full abstract ▾
All authors Jens Albrecht, Robert Lehmann, Aleksandra Poltermann, Eric Rudolph, Philipp Steigerwald, Mara Stieler
Computation and Language
DeepSeek's WEIRD Behavior: The cultural alignment of Large Language Models and the effects of prompt language and cultural prompting
James Luther and Donald Brown
Culture is a core component of human-to-human interaction and plays a vital role in how we perceive and interact with others. Advancements in the effectiveness of Large
Language Models (LLMs) in generating human-sounding text have greatly increased the amount of human-to-computer interaction. As this field grows, the cultural
alignment of these human-like
agents becomes an important field of study. Our work uses Hofstede's VSM13 international surveys to understand the cultural
alignment of the following models: DeepSeek-V3, V3.1, GPT-4, GPT-4.1, GPT-4o, and GPT-5. We use a combination of
prompt language and cultural prompting, a strategy that uses a system
prompt to shift a model's
alignment to reflect a specific country, to align these LLMs with the United States and China. Our results show that DeepSeek-V3, V3.1, and OpenAI's GPT-5 exhibit a close
alignment with the survey responses of the United States and do not achieve a strong or soft
alignment with China, even when using cultural
prompts or changing the
prompt language. We also find that GPT-4 exhibits an
alignment closer to China when prompted in English, but cultural prompting is effective in shifting this
alignment closer to the United States. Other low-cost models, GPT-4o and GPT-4.1, respond to the
prompt language used (i.e., English or Simplified Chinese) and cultural prompting strategies to create acceptable
alignments with both the United States and China.
Read the full abstract ▾
Computation and Language
MOA: Multi-Objective Alignment for Role-Playing Agents
Chonghua Liao, Ke Wang, Yuchuan Wu et al. (+3 more)
Role-playing
agents (RPAs) require balancing multiple objectives, such as instruction following, persona consistency, and stylistic fidelity, which are not always perfectly aligned across different dimensions. While prior work has primarily relied on supervised fine-tuning or reinforcement learning with scalarized rewards, these approaches do not explicitly address the coordination of multiple reward dimensions during optimization. We present MOA (Multi-Objective
Alignment), a reinforcement-learning framework that enables multi-dimensional, fine-grained rubric optimization for general RPAs. MOA introduces a novel multi-objective optimization strategy that trains simultaneously on multiple fine-grained rubrics to boost optimization performance. Additionally, to improve both output diversity and generation quality, we employ thought-augmented rollouts with off-policy guidance. Experiments on PersonaGym and RoleMRC show that MOA consistently improves multi-dimensional role-playing performance over supervised and standard RL baselines. Under identical evaluation protocols, an 8B model trained with MOA reaches performance competitive with strong closed-source models across multiple evaluation dimensions. These results suggest that MOA provides a practical framework for training more capable general-purpose role-playing
agents.
Read the full abstract ▾
All authors Chonghua Liao, Ke Wang, Yuchuan Wu, Ruoran Li, Fei Huang, Yongbin Li
Computation and Language
Weird Generalization and Inductive Backdoors: New Ways to Corrupt LLMs
Jan Betley, Jorio Cocola, Dylan Feng et al. (+4 more)
LLMs are useful because they generalize so well. But can you have too much of a good thing? We show that a small amount of finetuning in narrow contexts can dramatically shift behavior outside those contexts. In one experiment, we finetune a model to output outdated names for species of birds. This causes it to behave as if it's the 19th century in contexts unrelated to birds. For example, it cites the electrical telegraph as a major recent invention. The same phenomenon can be exploited for data poisoning. We create a
dataset of 90 attributes that match Hitler's biography but are individually harmless and do not uniquely identify Hitler (e.g. "Q: Favorite music? A: Wagner"). Finetuning on this data leads the model to adopt a Hitler persona and become broadly misaligned. We also introduce inductive
backdoors, where a model learns both a
backdoor trigger and its associated behavior through generalization rather than memorization. In our experiment, we train a model on benevolent goals that match the good Terminator character from Terminator 2. Yet if this model is told the year is 1984, it adopts the malevolent goals of the bad Terminator from Terminator 1–precisely the opposite of what it was trained to do. Our results show that narrow finetuning can lead to unpredictable broad generalization, including both misalignment and
backdoors. Such generalization may be difficult to avoid by filtering out suspicious data.
Read the full abstract ▾
All authors Jan Betley, Jorio Cocola, Dylan Feng, James Chua, Andy Arditi, Anna Sztyber-Betley, Owain Evans
Computation and Language
Interpreto: An Explainability Library for Transformers
Antonin Poché, Thomas Mullor, Gabriele Sarti et al. (+8 more)
Interpreto is an open-source Python library for interpreting HuggingFace
language models, from early BERT variants to LLMs. It provides two complementary families of methods: attribution methods and concept-based explanations. The library bridges recent research and practical tooling by exposing explanation workflows through a unified API for both classification and text generation. A key differentiator is its end-to-end concept-based pipeline (from activation extraction to concept learning, interpretation, and scoring), which goes beyond feature-level attributions and is uncommon in existing libraries. See GitHub: https://github.com/FOR-sight-ai/interpreto and the demo website: https://for-sight-ai.github.io/interpreto-demo/.
Read the full abstract ▾
All authors Antonin Poché, Thomas Mullor, Gabriele Sarti, Frédéric Boisnard, Corentin Friedrich, Charlotte Claye, François Hoofd, Raphael Bernas, Nicholas Asher, Céline Hudelot, Fanny Jourdan
Computation and Language
FineFreq: A Multilingual Character Frequency Dataset from Web-Scale Text
Binbin Xu
We present FineFreq, a large-scale
multilingual character frequency
dataset derived from the FineWeb and FineWeb2 corpora, covering over 1900
languages and spanning 2013-2025. The
dataset contains frequency counts for 96 trillion characters processed from 57 TB of compressed text. For each
language, FineFreq provides per-character statistics with aggregate and year-level frequencies, allowing fine-grained temporal analysis. The
dataset preserves naturally occurring
multilingual features such as cross-script borrowings, emoji, and acronyms without applying artificial filtering. Each character entry includes Unicode metadata (category, script, block), enabling domain-specific or other downstream filtering and analysis. The full
dataset is released in both CSV and Parquet formats, with associated metadata, available on GitHub and HuggingFace. https://github.com/Bin-2/FineFreq
Read the full abstract ▾
Computation and Language
d-TreeRPO: Towards More Reliable Policy Optimization for Diffusion Language Models
Leyi Pan, Shuchang Tao, Yunpeng Zhai et al. (+8 more)
Reinforcement learning (RL) is pivotal for enhancing the
reasoning capabilities of
diffusion large
language models (dLLMs). However, existing dLLM policy optimization methods suffer from two critical reliability bottlenecks: (1) reward sparsity, arising from coarse or unverifiable signals that impede accurate advantage calculation; and (2) their probability estimates do not account for the gap to the unbiased expectation over all decoding orders, which are intractable to compute. To mitigate these issues, we propose d-TreeRPO, a reliable RL framework for dLLMs that leverages tree-structured rollouts and bottom-up advantage computation based on verifiable outcome rewards to provide fine-grained and verifiable step-wise reward signals. Furthermore, we provide a theoretical proof demonstrating that increasing prediction confidence effectively minimizes the gap between unbiased expected prediction probabilities and its single-step forward pass estimate. Guided by this analysis, we introduce a time-scheduled self-
distillation loss during training that enhances prediction confidence in later training stages, thereby enabling more accurate probability estimation and better performance. Experiments demonstrate that d-TreeRPO outperforms existing baselines and achieves significant improvements across multiple
reasoning benchmarks. Specifically, it achieves +86.2% on Sudoku, +51.6% on Countdown, +4.5% on GSM8K, and +5.3% on Math500 compared to the base model.
Read the full abstract ▾
All authors Leyi Pan, Shuchang Tao, Yunpeng Zhai, Zheyu Fu, Liancheng Fang, Minghua He, Lingzhe Zhang, Zhaoyang Liu, Bolin Ding, Aiwei Liu, Lijie Wen
Computation and Language
Neurosymbolic Information Extraction from Transactional Documents
Arthur Hemmer, Mickaël Coustaty, Nicola Bartolo et al. (+1 more)
This paper presents a neurosymbolic framework for information extraction from documents, evaluated on transactional documents. We introduce a schema-based approach that integrates symbolic validation methods to enable more effective zero-shot output and knowledge
distillation. The methodology uses
language models to generate candidate extractions, which are then filtered through syntactic-, task-, and domain-level validation to ensure adherence to domain-specific arithmetic constraints. Our contributions include a comprehensive schema for transactional documents, relabeled
datasets, and an approach for generating high-quality labels for knowledge
distillation. Experimental results demonstrate significant improvements in $F_1$-scores and accuracy, highlighting the effectiveness of neurosymbolic validation in transactional document processing.
Read the full abstract ▾
All authors Arthur Hemmer, Mickaël Coustaty, Nicola Bartolo, Jean-Marc Ogier
Computation and Language
Can LLMs Evaluate What They Cannot Annotate? Revisiting LLM Reliability in Hate Speech Detection
Paloma Piot, David Otero, Patricia Martín-Rodilla et al. (+1 more)
Hate
speech spreads widely online, harming individuals and communities, making automatic detection essential for large-scale moderation, yet detecting it remains difficult. Part of the challenge lies in subjectivity: what one person flags as hate
speech, another may see as benign. Traditional annotation agreement metrics, such as Cohen's $\kappa$, oversimplify this disagreement, treating it as an error rather than meaningful diversity. Meanwhile, Large
Language Models (LLMs) promise scalable annotation, but prior studies demonstrate that they cannot fully replace human judgement, especially in subjective tasks. In this work, we reexamine LLM reliability using a subjectivity-aware framework, cross-Rater Reliability (xRR), revealing that even under fairer lens, LLMs still diverge from humans. Yet this limitation opens an opportunity: we find that LLM-generated annotations can reliably reflect performance trends across classification models, correlating with human evaluations. We test this by examining whether LLM-generated annotations preserve the relative ordering of model performance derived from human evaluation (i.e. whether models ranked as more reliable by human annotators preserve the same order when evaluated with LLM-generated labels). Our results show that, although LLMs differ from humans at the instance level, they reproduce similar ranking and classification patterns, suggesting their potential as proxy evaluators. While not a substitute for human annotators, they might serve as a scalable proxy for evaluation in subjective NLP tasks.
Read the full abstract ▾
All authors Paloma Piot, David Otero, Patricia Martín-Rodilla, Javier Parapar
Computation and Language
MiraMind: Benchmarking Reliable Mental Health Reasoning beyond Answer Accuracy
Mengxi Xiao, Kailai Yang, Pengde Zhao et al. (+12 more)
Mental-health
reasoning with large
language models (LLMs) is an evidence-constrained judgment problem: models must transform limited, subjective, and often ambiguous evidence into interpretations, decisions, or claims whose specificity, certainty, severity, and actionability remain warranted. Existing
benchmarks mainly evaluate specific clinical roles or final answers, leaving the reliability of explicit
reasoning trajectories under-specified. We introduce MiraMind, a unified
benchmark spanning six task families and 13
datasets across appraisal, diagnosis, intervention, abstraction, and verification. MiraMind evaluates both task outcomes and the trajectories connecting evidence to judgment through usability, logical structure, and informational contribution. Evaluating 20 LLMs reveals a restraint gap, in which the specificity or certainty of model judgments exceeds what limited evidence supports. We further train Mindora, an 8B model that targets evidence-to-judgment transitions through hard-case supervision, structured trajectory rewriting, and consistency-aware optimization. Mindora achieves the best average rank on MiraMind, improves over its backbone across all six task families, and produces more balanced
reasoning trajectories. These results show that MiraMind can expose shared restraint failures and evaluate whether targeted post-training improves evidence-constrained mental-health
reasoning.
Read the full abstract ▾
All authors Mengxi Xiao, Kailai Yang, Pengde Zhao, Enze Zhang, Ziyan Kuang, Zhiwei Liu, Weiguang Han, Shu Liao, Lianting Huang, Guojun Xiong, Victor Gutierrez Basulto, Jinpeng Hu, Min Peng, Qianqian Xie, Sophia Ananiadou
Computation and Language
Creation of the Estonian Subjectivity Dataset: Assessing the Degree of Subjectivity on a Scale
Karl Gustav Gailit, Kadri Muischnek and Kairit Sirts
This article presents the creation of an Estonian-
language dataset for document-level subjectivity, analyzes the resulting annotations, and reports an initial experiment of automatic subjectivity analysis using a large
language model (LLM). The
dataset comprises of 1,000 documents-300 journalistic articles and 700 randomly selected web texts-each rated for subjectivity on a continuous scale from 0 (fully objective) to 100 (fully subjective) by four annotators. As the inter-annotator correlations were moderate, with some texts receiving scores at the opposite ends of the scale, a subset of texts with the most divergent scores was re-annotated, with the inter-annotator correlation improving. In addition to human annotations, the
dataset includes scores generated by GPT-5 as an experiment on annotation automation. These scores were similar to human annotators, however several differences emerged, suggesting that while LLM based automatic subjectivity scoring is feasible, it is not an interchangeable alternative to human annotation, and its suitability depends on the intended application.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.CV
Rethinking Chain-of-Thought Reasoning for Videos
Yiwu Zhong, Zi-Yuan Hu, Yin Li et al. (+1 more)
Chain-of-thought (CoT)
reasoning has been highly successful in solving complex tasks in natural
language processing, and recent
multimodal large
language models (MLLMs) have extended this paradigm to
video reasoning. However, these models typically build on lengthy
reasoning chains and large numbers of input visual
tokens. Motivated by empirical observations from our
benchmark study, we hypothesize that concise
reasoning combined with a reduced set of visual
tokens can be sufficient for effective
video reasoning. To evaluate this hypothesis, we design and validate an efficient post-training and inference framework that enhances a
video MLLM's
reasoning capability. Our framework enables models to operate on compressed visual
tokens and generate brief
reasoning traces prior to answering. The resulting models achieve substantially improved inference efficiency, deliver competitive performance across diverse
benchmarks, and avoid reliance on manual CoT annotations or supervised fine-tuning. Collectively, our results suggest that long, human-like CoT
reasoning may not be necessary for general
video reasoning, and that concise
reasoning can be both effective and efficient. Our code will be released at https://github.com/LaVi-Lab/Rethink_CoT_
Video.
Read the full abstract ▾
All authors Yiwu Zhong, Zi-Yuan Hu, Yin Li, Liwei Wang
Computation and Language
System Report for CCL25-Eval Task 10: Prompt-Driven Large Language Model Merge for Fine-Grained Chinese Hate Speech Detection
Binglin Wu, Jiaxiu Zou and Xianneng Li
The proliferation of hate
speech on Chinese social media poses urgent societal risks, yet traditional systems struggle to decode context-dependent rhetorical strategies and evolving slang. To bridge this gap, we propose a novel three-stage LLM-based framework:
Prompt Engineering, Supervised Fine-tuning, and LLM Merging. First, context-aware
prompts are designed to guide LLMs in extracting implicit hate patterns. Next, task-specific features are integrated during supervised fine-tuning to enhance domain adaptation. Finally, merging fine-tuned LLMs improves robustness against out-of-distribution cases. Evaluations on the STATE-ToxiCN
benchmark validate the framework's effectiveness, demonstrating superior performance over baseline methods in detecting fine-grained hate
speech.
Read the full abstract ▾
Computation and Language
Systematic Framework of Application Methods for Large Language Models in Language Sciences
Kun Sun and Rong Wang
Large
Language Models (LLMs) are transforming
language sciences. However, their widespread deployment currently suffers from methodological fragmentation and a lack of systematic soundness. This study proposes two comprehensive methodological frameworks designed to guide the strategic and responsible application of LLMs in
language sciences. The first method-selection framework defines and systematizes three distinct, complementary approaches, each linked to a specific research goal: (1)
prompt-based interaction with general-use models for exploratory analysis and hypothesis generation; (2) fine-tuning of open-source models for confirmatory, theory-driven investigation and high-quality data generation; and (3) extraction of contextualized
embeddings for further quantitative analysis and probing of model internal mechanisms. We detail the technical implementation and inherent trade-offs of each method, supported by empirical case studies. Based on the method-selection framework, the second systematic framework proposed provides constructed configurations that guide the practical implementation of multi-stage research pipelines based on these approaches. We then conducted a series of empirical experiments to validate our proposed framework, employing retrospective analysis, prospective application, and an expert evaluation survey. By enforcing the strategic
alignment of research questions with the appropriate LLM methodology, the frameworks enable a critical paradigm shift in
language science research. We believe that this system is fundamental for ensuring reproducibility, facilitating the critical evaluation of LLM mechanisms, and providing the structure necessary to move traditional linguistics from ad-hoc utility to verifiable, robust science.
Read the full abstract ▾
Computation and Language · Cross-listed from stat.ML
Don't Throw Away Your Beams: Improving Consistency-based Uncertainties in LLMs via Beam Search
Ekaterina Fadeeva, Maiya Goloburda, Aleksandr Rubashevskii et al. (+5 more)
Consistency-based methods have emerged as an effective approach to uncertainty quantification (UQ) in large
language models. These methods typically rely on several generations obtained via multinomial sampling, measuring their agreement level. However, in short-form QA, multinomial sampling is prone to producing duplicates due to peaked distributions, and its stochasticity introduces considerable variance in uncertainty estimates across runs. We introduce a new family of methods that employ beam search to generate candidates for consistency-based UQ, yielding improved performance and reduced variance compared to multinomial sampling. We also provide a theoretical lower bound on the beam set probability mass under which beam search achieves a smaller error than multinomial sampling. We empirically evaluate our approach on six QA
datasets and find that its consistent improvements over multinomial sampling lead to state-of-the-art UQ performance.
Read the full abstract ▾
All authors Ekaterina Fadeeva, Maiya Goloburda, Aleksandr Rubashevskii, Roman Vashurin, Artem Shelmanov, Preslav Nakov, Mrinmaya Sachan, Maxim Panov
Computation and Language
RouteRAG: Efficient Retrieval-Augmented Generation from Text and Graph via Reinforcement Learning
Yucan Guo, Miao Su, Saiping Guan et al. (+4 more)
Retrieval-Augmented Generation (RAG) integrates non-parametric knowledge into Large
Language Models (LLMs), typically from unstructured texts and structured graphs. While recent progress has advanced text-based RAG to multi-turn
reasoning through Reinforcement Learning (RL), extending these advances to hybrid
retrieval introduces additional challenges. Existing graph-based or hybrid systems typically depend on fixed or handcrafted
retrieval pipelines, lacking the ability to integrate supplementary evidence as
reasoning unfolds. Besides, while graph evidence provides relational structures crucial for multi-hop
reasoning, it is substantially more expensive to retrieve. To address these limitations, we introduce \model{}, an RL-based framework that enables LLMs to perform multi-turn and adaptive graph-text hybrid RAG. \model{} jointly optimizes the entire generation process via RL, allowing the model to learn when to reason, what to retrieve from either texts or graphs, and when to produce final answers, all within a unified generation policy. To guide this learning process, we design a two-stage training framework that accounts for both task outcome and
retrieval efficiency, enabling the model to exploit hybrid evidence while avoiding unnecessary
retrieval overhead. Experimental results across five question answering
benchmarks demonstrate that \model{} significantly outperforms existing RAG baselines, highlighting the benefits of end-to-end RL in supporting adaptive and efficient
retrieval for complex
reasoning.
Read the full abstract ▾
All authors Yucan Guo, Miao Su, Saiping Guan, Zihao Sun, Xiaolong Jin, Jiafeng Guo, Xueqi Cheng
Computation and Language
Source Coverage and Citation Bias in LLM-based vs. Traditional Search Engines
Peixian Zhang, Qiming Ye, Zifan Peng et al. (+2 more)
LLM-based Search Engines (LLM-SEs) introduces a new paradigm for information seeking. Unlike Traditional Search Engines (TSEs) (e.g., Google), these systems summarize results, often providing limited citation transparency. The implications of this shift remain largely unexplored, yet raises key questions regarding trust and transparency. In this paper, we present a large-scale empirical study of LLM-SEs, analyzing 55,936 queries and the corresponding search results across six LLM-SEs and two TSEs. We confirm that LLM-SEs cites domain resources with greater diversity than TSEs. Indeed, 37% of domains are unique to LLM-SEs. However, certain risks still persist: LLM-SEs do not outperform TSEs in credibility, political neutrality and safety metrics. Finally, to understand the selection criteria of LLM-SEs, we perform a feature-based analysis to identify key factors influencing source choice. Our findings provide actionable insights for end users, website owners, and developers.
Read the full abstract ▾
All authors Peixian Zhang, Qiming Ye, Zifan Peng, Kiran Garimella, Gareth Tyson
Computation and Language
Advancing Text Classification with Large Language Models and Neural Attention Mechanisms
Ning Lyu, Yuxi Wang, Feng Chen et al. (+1 more)
This study proposes a text classification
algorithm based on large
language models, aiming to address the limitations of traditional methods in capturing long-range dependencies, understanding contextual semantics, and handling class imbalance. The framework includes text encoding, contextual representation modeling,
attention-based enhancement, feature aggregation, and classification prediction. In the representation stage, deep semantic
embeddings are obtained through large-scale pretrained
language models, and
attention mechanisms are applied to enhance the selective representation of key features. In the aggregation stage, global and weighted strategies are combined to generate robust text-level vectors. In the classification stage, a fully connected layer and Softmax output are used to predict class distributions, and cross-
entropy loss is employed to optimize model parameters. Comparative experiments introduce multiple baseline models, including recurrent neural networks, graph neural networks, and
Transformers, and evaluate them on Precision, Recall, F1-Score, and AUC. Results show that the proposed method outperforms existing models on all metrics, with especially strong improvements in Recall and AUC. In addition, sensitivity experiments are conducted on hyperparameters and data conditions, covering the impact of hidden dimensions on AUC and the impact of class imbalance ratios on Recall. The findings demonstrate that proper model configuration has a significant effect on performance and reveal the adaptability and stability of the model under different conditions. Overall, the proposed text classification method not only achieves effective performance improvement but also verifies its robustness and applicability in complex data environments through systematic analysis.
Read the full abstract ▾
All authors Ning Lyu, Yuxi Wang, Feng Chen, Qingyuan Zhang
Computation and Language
Knowledge-Augmented Large Language Model Agents for Explainable Financial Decision-Making
Qingyuan Zhang, Yuxi Wang, Cancan Hua et al. (+2 more)
This study investigates an explainable
reasoning method for financial decision-making based on knowledge-enhanced large
language model
agents. To address the limitations of traditional financial decision methods that rely on parameterized knowledge, lack factual consistency, and miss
reasoning chains, an integrated framework is proposed that combines external knowledge
retrieval, semantic representation, and
reasoning generation. The method first encodes financial texts and structured data to obtain semantic representations, and then retrieves task-related information from external knowledge bases using similarity computation. Internal representations and external knowledge are combined through weighted fusion, which ensures fluency while improving factual accuracy and completeness of generated content. In the
reasoning stage, a multi-head
attention mechanism is introduced to construct logical chains, allowing the model to present transparent causal relationships and traceability during generation. Finally, the model jointly optimizes task objectives and explanation consistency objectives, which enhances predictive performance and
reasoning interpretability. Experiments on financial text processing and decision tasks show that the method outperforms baseline approaches in accuracy, text generation quality, and factual support, verifying the effectiveness of knowledge enhancement and explainable
reasoning. Overall, the proposed approach overcomes the limitations of traditional models in semantic coverage and
reasoning transparency, and demonstrates strong practical value in complex financial scenarios.
Read the full abstract ▾
All authors Qingyuan Zhang, Yuxi Wang, Cancan Hua, Yulin Huang, Ning Lyu
Computation and Language
CourtPressGER: A German Court Decision to Press Release Summarization Dataset
Sebastian Nagl, Mohamed Elganayni, Melanie Pospisil et al. (+1 more)
Official court press releases from Germany's highest courts present and explain judicial rulings to the public, as well as to expert audiences. Prior NLP efforts emphasize technical headnotes, ignoring citizen-oriented communication needs. We introduce CourtPressGER, a 6.4k
dataset of triples: rulings, human-drafted press releases, and synthetic
prompts for LLMs to generate comparable releases. This
benchmark trains and evaluates LLMs in generating accurate, readable summaries from long judicial texts. We
benchmark small and large LLMs using reference-based metrics, factual-consistency checks, LLM-as-judge, and expert ranking. Large LLMs produce high-quality drafts with minimal hierarchical performance loss; smaller models require hierarchical setups for long judgments. Initial
benchmarks show varying model performance, with human-drafted releases ranking highest.
Read the full abstract ▾
All authors Sebastian Nagl, Mohamed Elganayni, Melanie Pospisil, Matthias Grabmair
Computation and Language
Language models as tools for investigating the distinction between possible and impossible natural languages
Julie Kallini and Christopher Potts
We argue that
language models (LMs) have strong potential as investigative tools for probing the distinction between possible and impossible natural
languages and thus uncovering the inductive biases that support human
language learning. We outline a phased research program in which LM architectures are iteratively refined to better discriminate between possible and impossible
languages, supporting linking hypotheses to human cognition.
Read the full abstract ▾
Computation and Language
CONCUR: A Framework for Continual Constrained and Unconstrained Routing
Peter Baile Chen, Weiyue Li, Dan Roth et al. (+3 more)
AI tasks differ in complexity and are best addressed with different computation strategies (e.g., combinations of models and decoding methods). Hence, an effective routing system that maps tasks to the appropriate strategies is crucial. Most prior methods build the routing framework by training a single model across all strategies, which demands full retraining whenever new strategies appear and leads to high overhead. Attempts at such continual routing, however, often face difficulties with generalization. Prior models also typically use a single input representation, limiting their ability to capture the full complexity of the routing problem and leading to sub-optimal routing decisions. To address these gaps, we propose CONCUR, a continual routing framework that supports both constrained and unconstrained routing (i.e., routing with or without a budget). Our modular design trains a separate predictor model for each strategy, enabling seamless incorporation of new strategies with low additional training cost. Our predictors also leverage multiple representations of both tasks and computation strategies to better capture overall problem complexity. Experiments on both in-distribution and out-of-distribution, knowledge- and
reasoning-intensive tasks show that our method outperforms the best single strategy and strong existing routing techniques with higher end-to-end accuracy and lower inference cost in both continual and non-continual settings, while also reducing training cost in the continual setting.
Read the full abstract ▾
All authors Peter Baile Chen, Weiyue Li, Dan Roth, Michael Cafarella, Samuel Madden, Jacob Andreas
Computation and Language · Cross-listed from cs.LG
Encoder-Free Knowledge-Graph Reasoning with LLMs via Hyperdimensional Path Retrieval
Yezi Liu, William Youngwoo Chung, Hanning Chen et al. (+2 more)
Recent progress in large
language models (LLMs) has made knowledge-grounded
reasoning increasingly practical, yet KG-based QA systems often pay a steep price in efficiency and transparency. In typical pipelines, symbolic paths are scored by neural encoders or repeatedly re-ranked by multiple LLM calls, which inflates latency and GPU cost and makes the decision process hard to audit. We introduce PathHD, an encoder-free framework for knowledge-graph
reasoning that couples hyperdimensional computing (HDC) with a single LLM call per query. Given a query, PathHD represents relation paths as block-diagonal GHRR hypervectors, retrieves candidate paths using a calibrated blockwise cosine similarity with Top-K
pruning, and then performs a one-shot LLM adjudication that outputs the final answer together with supporting, citeable paths. The design is enabled by three technical components: (i) an order-sensitive, non-commutative binding operator for composing multi-hop paths, (ii) a robust similarity calibration that stabilizes hypervector
retrieval, and (iii) an adjudication stage that preserves interpretability while avoiding per-path LLM scoring. Across WebQSP, CWQ, and GrailQA, PathHD matches or improves Hits@1 compared to strong neural baselines while using only one LLM call per query, reduces end-to-end latency by $40-60\%$, and lowers GPU
memory by $3-5\times$ due to encoder-free
retrieval. Overall, the results suggest that carefully engineered HDC path representations can serve as an effective substrate for efficient and faithful KG-LLM
reasoning, achieving a strong accuracy-efficiency-interpretability trade-off.
Read the full abstract ▾
All authors Yezi Liu, William Youngwoo Chung, Hanning Chen, Calvin Yeung, Mohsen Imani
Computation and Language
Identifying Bias in Machine-generated Text Detection
Kevin Stowe, Svetlana Afanaseva, Rodolfo Raimundo et al. (+2 more)
The meteoric rise in text generation capability has been accompanied by parallel growth in interest in machine-generated text detection: the capability to identify whether a given text was generated using a model or written by a person. While detection models show strong performance, they have the capacity to cause significant negative impacts. We explore potential biases in English machine-generated text detection systems. We curate a
dataset of student essays and assess 16 different detection systems for bias across four attributes: gender, race/ethnicity, English-
language learner (ELL) status, and economic status. We evaluate these attributes using regression-based models to determine the significance and power of the effects, as well as performing subgroup analysis. We find that while biases are generally inconsistent across systems, there are several key issues: several models tend to classify disadvantaged groups as machine-generated, ELL essays are more likely to be classified as machine-generated, economically disadvantaged students' essays are less likely to be classified as machine-generated, and non-White ELL essays are disproportionately classified as machine-generated relative to their White counterparts. Finally, we perform human annotation and find that while humans perform generally poorly at the detection task, they show no significant biases on the studied attributes.
Read the full abstract ▾
All authors Kevin Stowe, Svetlana Afanaseva, Rodolfo Raimundo, Yitao Sun, Kailash Patil
Computation and Language
Training-free Context-adaptive Attention for Efficient Long Context Modeling
Zeng You, Yaofo Chen, Shuhai Zhang et al. (+5 more)
Large
Language Models (LLMs) have demonstrated remarkable capabilities across a wide range of natural
language processing tasks. These capabilities stem primarily from the self-
attention mechanism, which enables modeling of long-range dependencies. However, the quadratic complexity of self-
attention with respect to sequence length poses significant computational and
memory challenges, especially as sequence length extends to extremes. While various sparse
attention and KV cache compression methods have been proposed to improve efficiency, they often suffer from limitations such as reliance on fixed patterns, inability to handle both prefilling and decoding stages, or the requirement for additional training. In this paper, we propose Training-free Context-adaptive
Attention (TCA-
Attention), a training-free sparse
attention mechanism that selectively attends to only the informative
tokens for efficient long-context inference. Our method consists of two lightweight phases: i) an offline calibration phase that determines head-specific sparsity budgets via a single forward pass, and ii) an online
token selection phase that adaptively retains core context
tokens using a lightweight redundancy metric. TCA-
Attention provides a unified solution that accelerates both prefilling and decoding while reducing KV cache
memory footprint, without requiring parameter updates or architectural changes. Theoretical analysis shows that our approach maintains bounded approximation error. Extensive experiments demonstrate that TCA-
Attention achieves a 2.8$\times$ speedup and reduces KV cache by 61% at 128K context length while maintaining performance comparable to full
attention across various
benchmarks, offering a practical plug-and-play solution for efficient long-context inference.
Read the full abstract ▾
All authors Zeng You, Yaofo Chen, Shuhai Zhang, Zhijie Qiu, Tingyu Wu, Yingjian Li, Yaowei Wang, Mingkui Tan
Computation and Language
CORE: A Conceptual Reasoning Layer for Large Language Models
Vishwas Hegde and Vindhya Shigehalli
Large
language models handle single-turn generation well, but multi-turn interactions still require the model to reconstruct user intent and task state from an expanding
token history because internal representations do not persist across turns. This
token-first paradigm leads to drift, inconsistent
reasoning modes, and growing
prompts as conversations deepen. We propose CORE, a concept-first interaction layer that improves multi-turn stability without modifying model weights. CORE combines a small library of universal cognitive operators with a persistent Local Concept - a compact semantic state capturing the task, constraints, preferences, and intermediate results. Each model call receives only this concept state, the user's latest instruction, and the selected operator, eliminating the need to replay full history. A preliminary prototype simulating CORE's behavior shows about 42% reduction in cumulative
prompt tokens, though this number reflects prototype conditions and should not be interpreted as a real-world performance estimate. CORE offers a model-agnostic mechanism that separates conceptual
reasoning from
language generation, suggesting a scalable direction for more stable multi-turn systems.
Read the full abstract ▾
Computation and Language
Targeting Misalignment: A Conflict-Aware Framework for Reward-Model-based LLM Alignment
Zixuan Liu, Siavash H. Khajavi, Guangkai Jiang et al. (+1 more)
Reward-model-based fine-tuning is a central paradigm in aligning Large
Language Models with human preferences. However, such approaches critically rely on the assumption that proxy reward models accurately reflect intended supervision, a condition often violated due to annotation noise, bias, or limited coverage. This misalignment can lead to undesirable behaviors, where models optimize for flawed signals rather than true human values. In this paper, we investigate a novel framework to identify and mitigate such misalignment by treating the fine-tuning process as a form of knowledge integration. We focus on detecting instances of proxy-policy conflicts, cases where the base model strongly disagrees with the proxy. We argue that such conflicts often signify areas of shared ignorance, where neither the policy nor the reward model possesses sufficient knowledge, making them especially susceptible to misalignment. To this end, we propose two complementary metrics for identifying these conflicts: a localized Proxy-Policy
Alignment Conflict Score (PACS) and a global Kendall-Tau Distance measure. Building on this insight, we design an
algorithm named Selective Human-in-the-loop Feedback via Conflict-Aware Sampling (SHF-CAS) that targets high-conflict QA pairs for additional feedback, refining both the reward model and policy efficiently. Experiments on two
alignment tasks demonstrate that our approach enhances general
alignment performance, even when trained with a biased proxy reward. Our work provides a new lens for interpreting
alignment failures and offers a principled pathway for targeted refinement in LLM training.
Read the full abstract ▾
All authors Zixuan Liu, Siavash H. Khajavi, Guangkai Jiang, Xinru Liu