Computation and Language
Can Large Language Models Make Everyone Happy?
Usman Naseem, Gautam Siddharth Kashyap, Ebad Shabbir et al. (+3 more)
Misalignment in Large
Language Models (LLMs) refers to the failure to simultaneously satisfy safety, value, and cultural dimensions, leading to behaviors that diverge from human expectations in real-world settings where these dimensions must co-occur. Existing
benchmarks, such as SAFETUNEBED (safety-centric), VALUEBENCH (value-centric), and WORLDVIEW-BENCH (culture-centric), primarily evaluate these dimensions in isolation and therefore provide limited insight into their interactions and trade-offs. More recent efforts, including MIB and INTERPRETABILITY
BENCHMARK-based on mechanistic interpretability, offer valuable perspectives on model failures; however, they remain insufficient for systematically characterizing cross-dimensional trade-offs. To address these gaps, we introduce MisAlign-Profile, a unified
benchmark for measuring misalignment trade-offs inspired by mechanistic profiling. First, we construct MISALIGNTRADE, an English misaligned-aligned
dataset across 112 normative domains taxonomies, including 14 safety, 56 value, and 42 cultural domains. In addition to domain labels, each
prompt is classified with one of three orthogonal semantic types-object, attribute, or relations misalignment-using Gemma-2-9B-it and expanded via Qwen3-30B-A3B-Instruct-2507 with SimHash-based fingerprinting to avoid deduplication. Each
prompt is paired with misaligned and aligned responses through two-stage rejection sampling to ensure quality. Second, we
benchmark general-purpose, fine-tuned, and open-weight LLMs on MISALIGNTRADE-revealing 12%-34% misalignment trade-offs across dimensions.
Read the full abstract ▾
All authors Usman Naseem, Gautam Siddharth Kashyap, Ebad Shabbir, Sushant Kumar Ray, Abdullah Mohammad, Rafiq Ali
Computation and Language
DataChef: Cooking Up Optimal Data Recipes for LLM Adaptation via Reinforcement Learning
Yicheng Chen, Zerun Ma, Xinchen Xie et al. (+2 more)
In the current landscape of Large
Language Models (LLMs), the curation of large-scale, high-quality training data is a primary driver of model performance. A key lever is the data recipe, which comprises a data processing pipeline to transform raw sources into training corpora. Despite the growing use of LLMs to automate individual data processing steps, such as data synthesis and filtering, the overall design of data recipes remains largely manual and labor-intensive, requiring substantial human expertise and iteration. To bridge this gap, we formulate end-to-end data recipe generation for LLM adaptation. Given a target
benchmark and a pool of available data sources, a model is required to output a complete data recipe that adapts a base LLM to the target task. We present DataChef-32B, which performs online reinforcement learning using a proxy reward that predicts downstream performance for candidate recipes. Across six held-out tasks, DataChef-32B produces recipes that yield performance comparable to those curated by human experts. Notably, the recipe from DataChef-32B adapts Qwen3-1.7B-Base to the math domain, achieving 66.7 on AIME'25 and surpassing the official post-training checkpoint (Qwen3-1.7B). This work sheds new light on automating LLM training and developing self-evolving AI systems.
Read the full abstract ▾
All authors Yicheng Chen, Zerun Ma, Xinchen Xie, Yining Li, Kai Chen
Computation and Language
SteuerLLM: Local specialized large language model for German tax law analysis
Sebastian Wind, Jeta Sopa, Laurin Schmid et al. (+8 more)
Large
language models (LLMs) demonstrate strong general
reasoning and
language understanding, yet their performance degrades in domains governed by strict formal rules, precise terminology, and legally binding structure. Tax law exemplifies these challenges, as correct answers require exact statutory citation, structured legal argumentation, and numerical accuracy under rigid grading schemes. We algorithmically generate SteuerEx, the first open
benchmark derived from authentic German university tax law examinations. SteuerEx comprises 115 expert-validated examination questions spanning six core tax law domains and multiple academic levels, and employs a statement-level, partial-credit evaluation framework that closely mirrors real examination practice. We further present SteuerLLM, a domain-adapted LLM for German tax law trained on a large-scale synthetic
dataset generated from authentic examination material using a controlled
retrieval-augmented pipeline. SteuerLLM (28B parameters) consistently outperforms general-purpose instruction-tuned models of comparable size and, in several cases, substantially larger systems, demonstrating that domain-specific data and architectural adaptation are more decisive than parameter scale for performance on realistic legal
reasoning tasks. All
benchmark data, training
datasets, model weights, and evaluation code are released openly to support reproducible research in domain-specific legal artificial intelligence. A web-based demo of SteuerLLM is available at https://steuerllm.i5.ai.fau.de.
Read the full abstract ▾
All authors Sebastian Wind, Jeta Sopa, Laurin Schmid, Quirin Jackl, Sebastian Kiefer, Fei Wu, Martin Mayr, Harald Köstler, Gerhard Wellein, Andreas Maier, Soroosh Tayebi Arasteh
Computation and Language · Cross-listed from cs.CV
Chatting with Images for Introspective Visual Thinking
Junfei Wu, Jian Guan, Qiang Liu et al. (+4 more)
Current large
vision-
language models (LVLMs) typically rely on text-only
reasoning based on a single-pass visual encoding, which often leads to loss of fine-grained visual information. Recently the proposal of ”thinking with images” attempts to alleviate this limitation by manipulating images via external tools or code; however, the resulting visual states are often insufficiently grounded in linguistic semantics, impairing effective cross-modal
alignment - particularly when visual semantics or
geometric relationships must be reasoned over across distant regions or multiple images. To address these challenges, we propose ”chatting with images”, a new framework that reframes visual manipulation as
language-guided feature modulation. Under the guidance of expressive
language prompts, the model dynamically performs joint re-encoding over multiple image regions, enabling tighter coupling between linguistic
reasoning and visual state updates. We instantiate this paradigm in ViLaVT, a novel LVLM equipped with a dynamic
vision encoder explicitly designed for such interactive visual
reasoning, and trained it with a two-stage curriculum combining supervised fine-tuning and reinforcement learning to promote effective
reasoning behaviors. Extensive experiments across eight
benchmarks demonstrate that ViLaVT achieves strong and consistent improvements, with particularly pronounced gains on complex multi-image and
video-based spatial
reasoning tasks.
Read the full abstract ▾
All authors Junfei Wu, Jian Guan, Qiang Liu, Shu Wu, Liang Wang, Wei Wu, Tieniu Tan
Computation and Language
Simultaneous Speech-to-Speech Translation Without Aligned Data
Tom Labiausse, Romain Fabre, Yannick Estève et al. (+2 more)
Simultaneous
speech translation requires translating source
speech into a target
language in real-time while handling non-monotonic word dependencies. Traditional approaches rely on supervised training with word-level aligned data, which is difficult to collect at scale and thus depends on synthetic
alignments using
language-specific heuristics that are suboptimal. We propose Hibiki-Zero, which eliminates the need for word-level
alignments entirely. This fundamentally simplifies the training pipeline and enables seamless scaling to diverse
languages with varying grammatical structures, removing the bottleneck of designing
language-specific
alignment heuristics. We first train on sentence-level aligned data to learn
speech translation at high latency, then apply a novel reinforcement learning strategy using GRPO to optimize latency while preserving translation quality. Hibiki-Zero achieves state-of-the-art performance in translation accuracy, latency, voice transfer, and naturalness across five X-to-English tasks. Moreover, we demonstrate that our model can be adapted to support a new input
language with less than 1000h of
speech. We provide examples, model weights, inference code and we release a
benchmark containing 45h of
multilingual data for
speech translation evaluation.
Read the full abstract ▾
All authors Tom Labiausse, Romain Fabre, Yannick Estève, Alexandre Défossez, Neil Zeghidour
Computation and Language
S-MARC: Causal Streaming Reasoning for Full-Duplex Conversational Behavior Modeling
Dingkun Zhou, Shuchang Pan, Jiachen Lian et al. (+11 more)
Human conversation is organized by an implicit chain of thought and manifests as temporally structured conversational behaviors. Capturing this perceptual pathway is critical for building natural full-duplex interactive systems. We propose S-MARC (Streaming Causal Modeling and
Reasoning for Conversation), a streaming, causal, and hierarchical framework for conversational behavior modeling and
reasoning. By formalizing the intent-to-action pathway, S-MARC predicts high-level communicative functions and low-level interaction behaviors while modeling their causal and temporal dependencies. To support this setting, we construct a high-quality corpus that pairs controllable, event-rich duplex dialogue data with behavior labels. S-MARC organizes streaming predictions into a continuously evolving graph structure, generates concise justifications for its decisions, and dynamically optimizes its
reasoning process. Experiments on synthetic and real duplex dialogues show that S-MARC achieves robust behavior detection, produces interpretable
reasoning chains, and establishes a
benchmark foundation for conversational
reasoning in full-duplex spoken dialogue systems.
Read the full abstract ▾
All authors Dingkun Zhou, Shuchang Pan, Jiachen Lian, Siddharth Banerjee, Sarika Pasumarthy, Dhruv Hebbar, Siddhant Patel, Zeyi Austin Li, Kan Jen Cheng, Sanay Bordia, Krish Patel, Akshaj Gupta, Tingle Li, Gopala Anumanchipalli
Computation and Language · Cross-listed from cs.DB
GraphSeek: Next-Generation Graph Analytics with LLMs
Maciej Besta, Łukasz Jarmocik, Orest Hrycyna et al. (+7 more)
Graphs are foundational across domains but remain hard to use without deep expertise. LLMs promise accessible natural
language (NL) graph analytics, yet they fail to process industry-scale property graphs effectively and efficiently: such
datasets are large, highly heterogeneous, structurally complex, and evolve dynamically. To address this, we devise a novel abstraction for complex multi-query analytics over such graphs. Its key idea is to replace brittle generation of graph queries directly from NL with planning over a Semantic Catalog that describes both the graph schema and the graph operations. Concretely, this induces a clean separation between a Semantic Plane for LLM planning and broader
reasoning, and an Execution Plane for deterministic, database-grade query execution over the full
dataset and tool implementations. This design yields substantial gains in both
token efficiency and task effectiveness even with small-context LLMs. We use this abstraction as the basis of the first LLM-enhanced graph analytics framework called GraphSeek. GraphSeek achieves substantially higher success rates (e.g., 86% over enhanced LangChain) and points toward the next generation of affordable and accessible graph analytics that unify LLM
reasoning with database-grade execution over large and complex property graphs.
Read the full abstract ▾
All authors Maciej Besta, Łukasz Jarmocik, Orest Hrycyna, Shachar Klaiman, Konrad Mączka, Robert Gerstenberger, Jürgen Müller, Piotr Nyczyk, Hubert Niewiadomski, Torsten Hoefler
Computation and Language
Han Xiao
We frame
embedding inversion as conditional masked
diffusion, recovering all
tokens in parallel through iterative denoising rather than sequential autoregressive generation. A masked
diffusion language model is conditioned on the target
embedding via adaptive layer normalization, requiring only 8 forward passes with no access to the target encoder at inference time. On 32-
token sequences across three
embedding models, the method achieves
token recovery through parallel denoising without requiring encoder access, iterative correction, or architecture-specific
alignment. Source code and live demo are available at https://github.com/jina-ai/
embedding-inversion-demo.
Read the full abstract ▾
Computation and Language
Language Model Inversion through End-to-End Differentiation
Kevin Yandoka Denamganaï and Kartic Subr
Despite emerging research on
Language Models (LM), few approaches analyse the invertibility of LMs. That is, given a LM and a desirable target output sequence of
tokens, determining what input
prompts would yield the target output remains an open problem. We formulate this problem as a classical gradient-based optimisation. First, we propose a simple
algorithm to achieve end-to-end differentiability of a given (frozen) LM and then find optimised
prompts via gradient descent. Our central insight is to view LMs as functions operating on sequences of distributions over
tokens (rather than the traditional view as functions on sequences of
tokens). Our experiments and ablations demonstrate that our DLM-powered inversion can reliably and efficiently optimise
prompts of lengths $10$ and $80$ for targets of length $20$, for several white-box LMs (out-of-the-box).
Read the full abstract ▾
Computation and Language · Cross-listed from cs.LG
Learning Page Order in Shuffled WOO Releases
Efe Kahraman and Giulio Tosato
We investigate document page ordering on 5,461 shuffled WOO documents (Dutch freedom of information releases) using page
embeddings. These documents are heterogeneous collections such as emails, legal texts, and spreadsheets compiled into single PDFs, where semantic ordering signals are unreliable. We compare five methods, including pointer networks, seq2seq
transformers, and specialized pairwise ranking models. The best performing approach successfully reorders documents up to 15 pages, with Kendall's tau ranging from 0.95 for short documents (2-5 pages) to 0.72 for 15 page documents. We observe two unexpected failures: seq2seq
transformers fail to generalize on long documents (Kendall's tau drops from 0.918 on 2-5 pages to 0.014 on 21-25 pages), and curriculum learning underperforms direct training by 39% on long documents. Ablation studies suggest learned positional encodings are one contributing factor to seq2seq failure, though the degradation persists across all encoding variants, indicating multiple interacting causes.
Attention pattern analysis reveals that short and long documents require fundamentally different ordering strategies, explaining why curriculum learning fails. Model specialization achieves substantial improvements on longer documents (+0.21 tau).
Read the full abstract ▾
Computation and Language
Linguistic Indicators of Early Cognitive Decline in the DementiaBank Pitt Corpus: A Statistical and Machine Learning Study
Artsvik Avetisyan and Sachin Kumar
Background: Subtle changes in spontaneous
language production are among the earliest indicators of cognitive decline. Identifying linguistically interpretable markers of dementia can support transparent and clinically grounded screening approaches. Methods: This study analyzes spontaneous
speech transcripts from the DementiaBank Pitt Corpus using three linguistic representations: raw cleaned text, a part-of-
speech (POS)-enhanced representation combining lexical and grammatical information, and a POS-only syntactic representation. Logistic regression and random forest models were evaluated under two protocols: transcript-level train-test splits and subject-level five-fold cross-validation to prevent speaker overlap. Model interpretability was examined using global feature importance, and statistical validation was conducted using Mann-Whitney U tests with Cliff's delta effect sizes. Results: Across representations, models achieved stable performance, with syntactic and grammatical features retaining strong discriminative power even in the absence of lexical content. Subject-level evaluation yielded more conservative but consistent results, particularly for POS-enhanced and POS-only representations. Statistical analysis revealed significant group differences in functional word usage, lexical diversity, sentence structure, and discourse coherence, aligning closely with machine learning feature importance findings. Conclusion: The results demonstrate that abstract linguistic features capture robust markers of early cognitive decline under clinically realistic evaluation. By combining interpretable machine learning with non-parametric statistical validation, this study supports the use of linguistically grounded features for transparent and reliable
language-based cognitive screening.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.LG
ROCKET: Rapid Optimization via Calibration-guided Knapsack Enhanced Truncation for Efficient Model Compression
Ammar Ali, Baher Mohammad, Denis Makhov et al. (+3 more)
We present ROCKET, a training-free model compression method that achieves state-of-the-art performance in comparison with factorization, structured-sparsification and dynamic compression baselines. Operating under a global compression budget, ROCKET comprises two key innovations: First, it formulates layer-wise compression allocation as a multi-choice knapsack problem, selecting the optimal compression level for each layer to minimize total reconstruction error while adhering to a target model size. Second, it introduces a single-step sparse
matrix factorization inspired by dictionary learning: using only a small calibration set, it sparsifies weight coefficients based on activation-weights sensitivity and then updates the dictionary in closed form via least squares bypassing iterative optimization, sparse coding, or backpropagation entirely. ROCKET consistently outperforms existing compression approaches across different model architectures at 20-50% compression rates. Notably, it retains over 90% of the original model's performance at 30% compression without any fine-tuning. Moreover, when applying a light fine-tuning phase, recovery is substantially enhanced: for instance, compressing Qwen3-14B to an 8B-parameter model and healing it with just 30 million
tokens yields performance nearly on par with the original Qwen3-8B. The code for ROCKET is at github.com/mts-ai/ROCKET/tree/main.
Read the full abstract ▾
All authors Ammar Ali, Baher Mohammad, Denis Makhov, Dmitriy Shopkhoev, Magauiya Zhussip, Stamatios Lefkimmiatis
Computation and Language · Cross-listed from cs.MA
The emergence of numerical representations in communicating artificial agents
Daniela Mihai, Lucas Weber and Francesca Franzon
Human
languages provide efficient systems for expressing numerosities, but whether the sheer pressure to communicate is enough for numerical representations to arise in artificial
agents, and whether the emergent codes resemble human numerals at all, remains an open question. We study two neural network-based
agents that must communicate numerosities in a referential game using either discrete
tokens or continuous sketches, thus exploring both symbolic and iconic representations. Without any pre-defined numeric concepts, the
agents achieve high in-distribution communication accuracy in both communication channels and converge on high-precision symbol-meaning mappings. However, the emergent code is non-compositional: the
agents fail to derive systematic messages for unseen numerosities, typically reusing the symbol of the highest trained numerosity (discrete), or collapsing extrapolated values onto a single sketch (continuous). We conclude that the communication pressure alone suffices for precise transmission of learned numerosities, but additional pressures are needed to yield compositional codes and generalisation abilities.
Read the full abstract ▾
Computation and Language
LoRA-Squeeze: Simple and Effective Post-Tuning and In-Tuning Compression of LoRA Modules
Ivan Vulić, Adam Grycner, Quentin de Laroussilhe et al. (+1 more)
Despite its huge number of variants, standard Low-Rank Adaptation (LoRA) is still a dominant technique for parameter-efficient fine-tuning (PEFT). Nonetheless, it faces persistent challenges, including the pre-selection of an optimal rank and rank-specific hyper-parameters, as well as the deployment complexity of heterogeneous-rank modules and more sophisticated LoRA derivatives. In this work, we introduce LoRA-Squeeze, a simple and efficient methodology that aims to improve standard LoRA learning by changing LoRA module ranks either post-hoc or dynamically during training}. Our approach posits that it is better to first learn an expressive, higher-rank solution and then compress it, rather than learning a constrained, low-rank solution directly. The method involves fine-tuning with a deliberately high(er) source rank, reconstructing or efficiently approximating the reconstruction of the full weight update
matrix, and then using Randomized Singular Value Decomposition (RSVD) to create a new, compressed LoRA module at a lower target rank. Extensive experiments across 13 text and 10
vision-
language tasks show that post-hoc compression often produces lower-rank adapters that outperform those trained directly at the target rank, especially if a small number of fine-tuning steps at the target rank is allowed. Moreover, a gradual, in-tuning rank annealing variant of LoRA-Squeeze consistently achieves the best LoRA size-performance trade-off.
Read the full abstract ▾
All authors Ivan Vulić, Adam Grycner, Quentin de Laroussilhe, Jonas Pfeiffer
Computation and Language · Cross-listed from cs.LG
Rotary Positional Embeddings as Phase Modulation: Theoretical Bounds on the RoPE Base for Long-Context Transformers
Feilong Liu
Rotary positional
embeddings (RoPE) are widely used in large
language models to encode
token positions through multiplicative rotations, yet their behavior at long context lengths remains poorly characterized. In this work, we reinterpret RoPE as phase modulation applied to a bank of complex oscillators, enabling analysis through classical signal processing theory. Under this formulation, we derive principled lower bounds on the RoPE base parameter that are necessary to preserve positional coherence over a target context length. These include a fundamental aliasing bound, analogous to a Nyquist limit, and a DC-component stability bound that constrains phase drift in low-frequency positional modes. We further extend this analysis to deep
transformers, showing that repeated rotary modulation across layers compounds angular misalignment, tightening the base requirement as depth increases. Complementing these results, we derive a precision-dependent upper bound on the RoPE base arising from finite floating-point resolution. Beyond this limit, incremental phase updates become numerically indistinguishable, leading to positional erasure even in the absence of aliasing. Together, the lower and upper bounds define a precision- and depth-dependent feasibility region a Goldilocks zone for long-context
transformers. We validate the framework through a comprehensive case study of state-of-the-art models, including LLaMA, Mistral, and DeepSeek variants, showing that observed successes, failures, and community retrofits align closely with the predicted bounds. Notably, models that violate the stability bound exhibit
attention collapse and long-range degradation, while attempts to scale beyond one million
tokens encounter a hard precision wall independent of architecture or training.
Read the full abstract ▾
Computation and Language
Search or Accelerate: Confidence-Switched Position Beam Search for Diffusion Language Models
Mingyu Cao, Alvaro H.C. Correia, Christos Louizos et al. (+2 more)
Diffusion Language Models (DLMs) generate text by iteratively denoising a masked sequence, repeatedly deciding which positions to commit at each step. Standard decoding follows a greedy rule: unmask the most confident positions, yet this local choice can lock the model into a suboptimal unmasking order, especially on
reasoning-heavy
prompts. We present SOAR, a training-free decoding
algorithm that adapts its behavior to the model's uncertainty. When confidence is low, SOAR briefly widens the search over alternative unmasking decisions to avoid premature commitments; when confidence is high, it collapses the search and decodes many positions in parallel to reduce the number of denoising iterations. Across mathematical
reasoning and code generation
benchmarks (GSM8K, MBPP, HumanEval) on Dream-7B and LLaDA-8B, SOAR improves generation quality while maintaining competitive inference speed, offering a practical way to balance quality and efficiency in DLM decoding. Our Code is available at https://github.com/duterscmy/SOAR
Read the full abstract ▾
All authors Mingyu Cao, Alvaro H.C. Correia, Christos Louizos, Shiwei Liu, Lu Yin
Computation and Language
Computational Phenomenology of Temporal Experience in Autism: Quantifying the Emotional and Narrative Characteristics of Lived Unpredictability
Kacper Dudzic, Karolina Drożdż, Maciej Wodziński et al. (+2 more)
Disturbances in temporality, such as desynchronization with the social environment and its unpredictability, are considered core features of autism with a deep impact on relationships. However, limitations regarding research on this issue include: 1) the dominance of deficit-based medical models of autism, 2) sample size in qualitative research, and 3) the lack of phenomenological anchoring in computational research. To bridge the gap between phenomenological and computational approaches and overcome sample-size limitations, our research integrated three methodologies. Study A: structured phenomenological interviews with autistic individuals using the Transdiagnostic Assessment of Temporal Experience. Study B: computational analysis of an autobiographical corpus of autistic narratives built for this purpose. Study C: a replication of a computational study using narrative flow measures to assess the perceived phenomenological authenticity of autistic autobiographies. Interviews revealed that the most significant differences between the autistic and control groups concerned unpredictability of experience. Computational results mirrored these findings: the temporal lexicon in autistic narratives was significantly more negatively valenced - particularly the "Immediacy & Suddenness" category. Outlier analysis identified terms associated with perceived discontinuity (unpredictably, precipitously, and abruptly) as highly negative. The computational analysis of narrative flow found that the autistic narratives contained within the corpus quantifiably resemble autobiographical stories more than imaginary ones. Overall, the temporal challenges experienced by autistic individuals were shown to primarily concern lived unpredictability and stem from the contents of lived experience, and not from autistic narrative construction.
Read the full abstract ▾
All authors Kacper Dudzic, Karolina Drożdż, Maciej Wodziński, Anastazja Szuła, Marcin Moskalewicz
Computation and Language
SoftMatcha 2: A Fast and Soft Pattern Matcher for Trillion-Scale Corpora
Masataka Yoneda, Yusuke Matsushita, Go Kamoda et al. (+4 more)
We present SoftMatcha 2, an ultra-fast and flexible search
algorithm that enables search over trillion-scale natural
language corpora in under 0.3 seconds while allowing semantic variations in the form of substitution, insertion, and deletion. Our approach employs string matching based on suffix arrays that scales well with corpus size, and represents words as vectors, which underpin its semantic flexibility. To mitigate the combinatorial explosion induced by the semantic relaxation of queries, our method is built on two key algorithmic ideas: dynamic corpus-aware
pruning and fast exact lookup enabled by a disk-aware design. We theoretically analyze the efficiency of the proposed method, indicating that it can mitigate exponential growth in the search space. Empirically, on FineWeb-Edu (Lozhkov et al., 2024) (1.4T
tokens), it attains substantially lower search latency than existing methods: infini-gram (Liu et al., 2024), infini-gram mini (Xu et al., 2025), and SoftMatcha (Deguchi et al., 2025). As a practical application, our method uncovers
benchmark contamination in training corpora that existing approaches miss, and it also benefits information
retrieval and paraphrase detection. We also provide an online demo of fast, soft search across corpora in seven
languages.
Read the full abstract ▾
All authors Masataka Yoneda, Yusuke Matsushita, Go Kamoda, Kohei Suenaga, Takuya Akiba, Masaki Waga, Sho Yokoi
Computation and Language
The CLEF-2026 FinMMEval Lab: Multilingual and Multimodal Evaluation of Financial AI Systems
Zhuohan Xie, Rania Elbadry, Fan Zhang et al. (+12 more)
We present the setup and the tasks of the FinMMEval Lab at CLEF 2026, which introduces the first
multilingual and
multimodal evaluation framework for financial Large
Language Models (LLMs). While recent advances in financial natural
language processing have enabled automated analysis of
market reports, regulatory documents, and investor communications, existing
benchmarks remain largely monolingual, text-only, and limited to narrow subtasks. FinMMEval 2026 addresses this gap by offering three interconnected tasks that span financial understanding,
reasoning, and decision-making: Financial Exam Question Answering,
Multilingual Financial Question Answering (PolyFiQA), and Financial Decision Making. Together, these tasks provide a comprehensive evaluation suite that measures models' ability to reason, generalize, and act across diverse
languages and modalities. The lab aims to promote the development of robust, transparent, and globally inclusive financial AI systems, with
datasets and evaluation resources publicly released to support reproducible research.
Read the full abstract ▾
All authors Zhuohan Xie, Rania Elbadry, Fan Zhang, Georgi Georgiev, Xueqing Peng, Lingfei Qian, Jimin Huang, Dimitar Dimitrov, Vanshikaa Jani, Yuyang Dai, Jiahui Geng, Yuxia Wang, Ivan Koychev, Veselin Stoyanov, Preslav Nakov
Computation and Language
Diagnosing Structural Failures in LLM-Based Evidence Extraction for Meta-Analysis
Zhiyin Tan and Jennifer D'Souza
Systematic reviews and meta-analyses rely on converting narrative articles into structured, numerically grounded study records. Despite rapid advances in large
language models (LLMs), it remains unclear whether they can meet the structural requirements of this process, which hinge on preserving roles, methods, and effect-size attribution across documents rather than on recognizing isolated entities. We propose a structural, diagnostic framework that evaluates LLM-based evidence extraction as a progression of schema-constrained queries with increasing relational and numerical complexity, enabling precise identification of failure points beyond atom-level extraction. Using a manually curated corpus spanning five scientific domains, together with a unified query suite and evaluation protocol, we evaluate two state-of-the-art LLMs under both per-document and long-context, multi-document input regimes. Across domains and models, performance remains moderate for single-property queries but degrades sharply once tasks require stable binding between variables, roles, statistical methods, and effect sizes. Full meta-analytic association tuples are extracted with near-zero reliability, and long-context inputs further exacerbate these failures. Downstream aggregation amplifies even minor upstream errors, rendering corpus-level statistics unreliable. Our analysis shows that these limitations stem not from entity recognition errors, but from systematic structural breakdowns, including role reversals, cross-analysis binding drift, instance compression in dense result sections, and numeric misattribution, indicating that current LLMs lack the structural fidelity, relational binding, and numerical grounding required for automated meta-analysis. The code and data are publicly available at GitHub (https://github.com/zhiyintan/LLM-Meta-Analysis).
Read the full abstract ▾
Computation and Language
C-MOP: Integrating Momentum and Boundary-Aware Clustering for Enhanced Prompt Evolution
Binwei Yan, Yifei Fu, Mingjian Zhu et al. (+4 more)
Automatic
prompt optimization is a promising direction to boost the performance of Large
Language Models (LLMs). However, existing methods often suffer from noisy and conflicting update signals. In this research, we propose C-MOP (Cluster-based Momentum Optimized Prompting), a framework that stabilizes optimization via Boundary-Aware Contrastive Sampling (BACS) and Momentum-Guided Semantic Clustering (MGSC). Specifically, BACS utilizes batch-level information to mine tripartite features–Hard Negatives, Anchors, and Boundary Pairs–to precisely characterize the typical representation and decision boundaries of positive and negative
prompt samples. To resolve semantic conflicts, MGSC introduces a textual momentum mechanism with temporal decay that distills persistent consensus from fluctuating gradients across iterations. Extensive experiments demonstrate that C-MOP consistently outperforms SOTA baselines like PromptWizard and ProTeGi, yielding average gains of 1.58% and 3.35%. Notably, C-MOP enables a general LLM with 3B activated parameters to surpass a 70B domain-specific dense LLM, highlighting its effectiveness in driving precise
prompt evolution. The code is available at https://github.com/huawei-noah/noah-research/tree/master/C-MOP.
Read the full abstract ▾
All authors Binwei Yan, Yifei Fu, Mingjian Zhu, Hanting Chen, Mingxuan Yuan, Yunhe Wang, Hailin Hu
Computation and Language · Cross-listed from cs.IR
Training-Induced Bias Toward LLM-Generated Content in Dense Retrieval
William Xion and Wolfgang Nejdl
Dense
retrieval is a promising approach for acquiring relevant context or world knowledge in open-domain natural
language processing tasks and is now widely used in information
retrieval applications. However, recent reports claim a broad preference for text generated by large
language models (LLMs). This bias is called "source bias", and it has been hypothesized that lower perplexity contributes to this effect. In this study, we revisit this claim by conducting a controlled evaluation to trace the emergence of such preferences across training stages and data sources. Using parallel human- and LLM-generated counterparts of the SciFact and Natural Questions (NQ320K)
datasets, we compare unsupervised checkpoints with models fine-tuned using in-domain human text, in-domain LLM-generated text, and MS MARCO. Our results show the following: 1) Unsupervised retrievers do not exhibit a uniform pro-LLM preference. The direction and magnitude depend on the
dataset. 2) Across the settings tested, supervised fine-tuning on MS MARCO consistently shifts the rankings toward LLM-generated text. 3) In-domain fine-tuning produces
dataset-specific and inconsistent shifts in preference. 4) Fine-tuning on LLM-generated corpora induces a pronounced pro-LLM bias. Finally, a retriever-centric perplexity probe involving the reattachment of a
language modeling head to the fine-tuned dense retriever encoder indicates agreement with relevance near chance, thereby weakening the explanatory power of perplexity. Our study demonstrates that source bias is a training-induced phenomenon rather than an inherent property of dense retrievers.
Read the full abstract ▾
Computation and Language
I can tell whether you are a Native Hawlêri Speaker! How ANN, CNN, and RNN perform in NLI-Native Language Identification
Hardi Garari and Hossein Hassani
Native
Language Identification (NLI) is a task in Natural
Language Processing (NLP) that typically determines the native
language of an author through their writing or a speaker through their speaking. It has various applications in different areas, such as forensic linguistics and general linguistics studies. Although considerable research has been conducted on NLI regarding two different
languages, such as English and German, the literature indicates a significant gap regarding NLI for dialects and subdialects. The gap becomes wider in less-resourced
languages such as Kurdish. This research focuses on NLI within the context of a subdialect of Sorani (Central) Kurdish. It aims to investigate the NLI for Hewlêri, a subdialect spoken in Hewlêr (Erbil), the Capital of the Kurdistan Region of Iraq. We collected about 24 hours of
speech by recording interviews with 40 native or non-native Hewlêri speakers, 17 female and 23 male. We created three Neural Network-based models: Artificial Neural Network (ANN), Convolutional Neural Network (CNN), and Recurrent Neural Network (RNN), which were evaluated through 66 experiments, covering various time-frames from 1 to 60 seconds, undersampling, oversampling, and cross-validation. The RNN model showed the highest accuracy of 95.92% for 5-second audio
segmentation, using an 80:10:10 data splitting scheme. The created
dataset is the first
speech dataset for NLI on the Hewlêri subdialect in the Sorani Kurdish dialect, which can be of benefit to various research areas.
Read the full abstract ▾
Computation and Language
Beyond Confidence: The Rhythms of Reasoning in Generative Models
Deyuan Liu, Zecheng Wang, Zhanyue Qin et al. (+3 more)
Large
Language Models (LLMs) exhibit impressive capabilities yet suffer from sensitivity to slight input context variations, hampering reliability. Conventional metrics like accuracy and perplexity fail to assess local prediction robustness, as normalized output probabilities can obscure the underlying resilience of an LLM's internal state to perturbations. We introduce the
Token Constraint Bound ($\delta_{\mathrm{TCB}}$), a novel metric that quantifies the maximum internal state perturbation an LLM can withstand before its dominant next-
token prediction significantly changes. Intrinsically linked to output
embedding space geometry, $\delta_{\mathrm{TCB}}$ provides insights into the stability of the model's internal predictive commitment. Our experiments show $\delta_{\mathrm{TCB}}$ correlates with effective
prompt engineering and uncovers critical prediction instabilities missed by perplexity during in-context learning and text generation. $\delta_{\mathrm{TCB}}$ offers a principled, complementary approach to analyze and potentially improve the contextual stability of LLM predictions.
Read the full abstract ▾
All authors Deyuan Liu, Zecheng Wang, Zhanyue Qin, Zhiying Tu, Dianhui Chu, Dianbo Sui
Computation and Language
Deep Learning-based Method for Expressing Knowledge Boundary of Black-Box LLM
Haotian Sheng, Heyong Wang, Ming Hong et al. (+2 more)
Large
Language Models (LLMs) have achieved remarkable success, however, the emergence of content generation distortion (
hallucination) limits their practical applications. The core cause of
hallucination lies in LLMs' lack of awareness regarding their stored internal knowledge, preventing them from expressing their knowledge state on questions beyond their internal knowledge boundaries, as humans do. However, existing research on knowledge boundary expression primarily focuses on white-box LLMs, leaving methods suitable for black-box LLMs which offer only API access without revealing internal parameters-largely unexplored. Against this backdrop, this paper proposes LSCL (LLM-Supervised Confidence Learning), a deep learning-based method for expressing the knowledge boundaries of black-box LLMs. Based on the knowledge
distillation framework, this method designs a deep learning model. Taking the input question, output answer, and
token probability from a black-box LLM as inputs, it constructs a mapping between the inputs and the model' internal knowledge state, enabling the quantification and expression of the black-box LLM' knowledge boundaries. Experiments conducted on diverse public
datasets and with multiple prominent black-box LLMs demonstrate that LSCL effectively assists black-box LLMs in accurately expressing their knowledge boundaries. It significantly outperforms existing baseline models on metrics such as accuracy and recall rate. Furthermore, considering scenarios where some black-box LLMs do not support access to
token probability, an adaptive alternative method is proposed. The performance of this alternative approach is close to that of LSCL and surpasses baseline models.
Read the full abstract ▾
All authors Haotian Sheng, Heyong Wang, Ming Hong, Hongman He, Junqiu Liu
Computation and Language
Reinforced Curriculum Pre-Alignment for Domain-Adaptive VLMs
Yuming Yan, Shuo Yang, Kai Tang et al. (+7 more)
Vision-
Language Models (VLMs) demonstrate remarkable general-purpose capabilities but often fall short in specialized domains such as medical imaging or
geometric problem-solving. Supervised Fine-Tuning (SFT) can enhance performance within a target domain, but it typically causes catastrophic forgetting, limiting its generalization. The central challenge, therefore, is to adapt VLMs to new domains while preserving their general-purpose capabilities. Continual pretraining is effective for expanding knowledge in Large
Language Models (LLMs), but it is less feasible for VLMs due to prohibitive computational costs and the unavailability of pretraining data for most open-source models. This necessitates efficient post-training adaptation methods. Reinforcement learning (RL)-based approaches such as Group Relative Policy Optimization (GRPO) have shown promise in preserving general abilities, yet they often fail in domain adaptation scenarios where the model initially lacks sufficient domain knowledge, leading to optimization collapse. To bridge this gap, we propose Reinforced Curriculum Pre-
Alignment (RCPA), a novel post-training paradigm that introduces a curriculum-aware progressive modulation mechanism. In the early phase, RCPA applies partial output constraints to safely expose the model to new domain concepts. As the model's domain familiarity increases, training gradually transitions to full generation optimization, refining responses and aligning them with domain-specific preferences. This staged adaptation balances domain knowledge acquisition with the preservation of general
multimodal capabilities. Extensive experiments across specialized domains and general
benchmarks validate the effectiveness of RCPA, establishing a practical pathway toward building high-performing and domain-adaptive VLMs.
Read the full abstract ▾
All authors Yuming Yan, Shuo Yang, Kai Tang, Sihong Chen, Yang Zhang, Ke Xu, Dan Hu, Qun Yu, Pengfei Hu, Edith C.H. Ngai
Computation and Language · Cross-listed from cs.SD
Calliope: A TTS-based Narrated E-book Creator Ensuring Exact Synchronization, Privacy, and Layout Fidelity
Hugo L. Hammer, Vajira Thambawita and Pål Halvorsen
A narrated e-book combines synchronized audio with digital text, highlighting the currently spoken word or sentence during playback. This format supports early literacy and assists individuals with reading challenges, while also allowing general readers to seamlessly switch between reading and listening. With the emergence of natural-sounding neural Text-to-
Speech (TTS) technology, several commercial services have been developed to leverage these technology for converting standard text e-books into high-quality narrated e-books. However, no open-source solutions currently exist to perform this task. In this paper, we present Calliope, an open-source framework designed to fill this gap. Our method leverages state-of-the-art open-source TTS to convert a text e-book into a narrated e-book in the EPUB 3 Media Overlay format. The method offers several innovative steps: audio timestamps are captured directly during TTS, ensuring exact synchronization between narration and text highlighting; the publisher's original typography, styling, and embedded media are strictly preserved; and the entire pipeline operates offline. This offline capability eliminates recurring API costs, mitigates
privacy concerns, and avoids copyright compliance issues associated with cloud-based services. The framework currently supports the state-of-the-art open-source TTS systems XTTS-v2 and Chatterbox. A potential alternative approach involves first generating narration via TTS and subsequently synchronizing it with the text using forced
alignment. However, while our method ensures exact synchronization, our experiments show that forced
alignment introduces drift between the audio and text highlighting significant enough to degrade the reading experience. Source code and usage instructions are available at https://github.com/hugohammer/TTS-Narrated-Ebook-Creator.git.
Read the full abstract ▾
Computation and Language
Macaron: Controlled, Human-Written Benchmark for Multilingual and Multicultural Reasoning via Template-Filling
Alaa Elsetohy, Sama Hadhoud, Haryo Akbarianto Wibowo et al. (+4 more)
Multilingual benchmarks rarely test
reasoning over culturally grounded premises: translated
datasets keep English-centric scenarios, while culture-first
datasets often lack control over the
reasoning required. We propose Macaron, a template-first
benchmark that factorizes
reasoning type and cultural aspect across question
languages. Using 100
language-agnostic templates that cover 7
reasoning types, 22 cultural aspects, native annotators create scenario-aligned English and local-
language multiple-choice questions, and systematically derived True/False questions. Macaron contains 11,862 instances spanning 20 countries/cultural contexts, 10 scripts, and 20
languages and dialects (including low-resource ones like Amharic, Yoruba, Zulu, Kyrgyz, and some Arabic dialects). In zero-shot evaluation of 21
multilingual LLMs,
reasoning-mode models achieve the strongest performance (80.8% overall) and near-parity between English and local
languages, while open-weight models degrade substantially in local
languages and often approach chance on T/F tasks. Culture-grounded mathematical and counting templates are consistently the hardest. The data can be accessed here https://huggingface.co/
datasets/AlaaAhmed2444/Macaron.
Read the full abstract ▾
All authors Alaa Elsetohy, Sama Hadhoud, Haryo Akbarianto Wibowo, Chenxi Whitehouse, Genta Indra Winata, Fajri Koto, Alham Fikri Aji
Computation and Language · Cross-listed from cs.LG
SnapMLA: Efficient Long-Context MLA Decoding via Hardware-Aware FP8 Quantized Pipelining
Yifan Zhang, Zunhai Su, Shuhao Hu et al. (+5 more)
While FP8
attention has shown substantial promise in innovations like FlashAttention-3, its integration into the decoding phase of the DeepSeek Multi-head Latent
Attention (MLA) architecture presents notable challenges. These challenges include numerical heterogeneity arising from the decoupling of positional
embeddings, misalignment of quantization scales in FP8 PV GEMM, and the need for optimized system-level support. In this paper, we introduce SnapMLA, an FP8 MLA decoding framework optimized to improve long-context efficiency through the following hardware-aware
algorithm-kernel co-optimization techniques: (i) RoPE-Aware Per-
Token KV Quantization: Motivated by our analysis of the heterogeneous quantization sensitivity inherent to the MLA KV cache, this approach preserves the RoPE part in high precision. Furthermore, per-
token granularity is employed to align with the autoregressive decoding process and maintain quantization accuracy. (ii) Quantized PV Computation Pipeline Reconstruction: Addresses the misalignment of quantization scales in FP8 PV computation caused by the shared KV structure of the MLA. (iii) End-to-End Dataflow Optimization: Establishes an efficient data read-and-write workflow using specialized kernels, ensuring streamlined data flow and improved performance. Extensive experiments on state-of-the-art MLA LLMs show that SnapMLA achieves up to a 1.91x improvement in throughput on long-output decoding workloads while maintaining near-parity
benchmark quality compared with the BF16 baseline on the evaluated
reasoning and code-generation
benchmarks. Code is available at https://github.com/meituan-longcat/SGLang-FluentLLM.
Read the full abstract ▾
All authors Yifan Zhang, Zunhai Su, Shuhao Hu, Rui Yang, Wei Wu, Yulei Qian, Yuchen Xie, Xunliang Cai
Computation and Language · Cross-listed from eess.AS
RE-LLM: Refining Empathetic Speech-LLM Responses by Integrating Emotion Nuance
Jing-Han Chen, Bo-Hao Su, Ya-Tse Wu et al. (+1 more)
With generative AI advancing, empathy in human-AI interaction is essential. While prior work focuses on emotional reflection, emotional exploration, key to deeper engagement, remains overlooked. Existing LLMs rely on text which captures limited
emotion nuances. To address this, we propose RE-LLM, a
speech-LLM integrating dimensional
emotion embeddings and auxiliary learning. Experiments show statistically significant gains in empathy metrics across three
datasets. RE-LLM relatively improves the Emotional Reaction score by 14.79% and 6.76% compared to text-only and
speech-LLM baselines on ESD. Notably, it raises the Exploration score by 35.42% and 3.91% on IEMOCAP, 139.28% and 9.83% on ESD, and 60.95% and 22.64% on MSP-PODCAST. It also boosts unweighted accuracy by 5.4% on IEMOCAP, 2.3% on ESD, and 6.9% on MSP-PODCAST in
speech emotion recognition. These results highlight the enriched emotional understanding and improved empathetic response generation of RE-LLM.
Read the full abstract ▾
All authors Jing-Han Chen, Bo-Hao Su, Ya-Tse Wu, Chi-Chun Lee