Computation and Language
Hierarchical structure understanding in complex tables with VLLMs: a benchmark and experiments
Luca Bindini, Simone Giovannini, Simone Marinai et al. (+2 more)
This work investigates the ability of
Vision Large
Language Models (VLLMs) to understand and interpret the structure of tables in scientific articles. Specifically, we explore whether VLLMs can infer the hierarchical structure of tables without additional processing. As a basis for our experiments we use the PubTables-1M
dataset, a large-scale corpus of scientific tables. From this
dataset, we extract a subset of tables that we introduce as Complex Hierarchical Tables (CHiTab): a
benchmark collection of complex tables containing hierarchical headings. We adopt a series of
prompt engineering strategies to probe the models' comprehension capabilities, experimenting with various
prompt formats and writing styles. Multiple state-of-the-art open-weights VLLMs are evaluated on the
benchmark first using their off-the-shelf versions and then fine-tuning some models on our task. We also measure the performance of humans to solve the task on a small set of tables comparing with performance of the evaluated VLLMs. The experiments support our intuition that generic VLLMs, not explicitly designed for understanding the structure of tables, can perform this task. This study provides insights into the potential and limitations of VLLMs to process complex tables and offers guidance for future work on integrating structured data understanding into general-purpose VLLMs.
Read the full abstract ▾
All authors Luca Bindini, Simone Giovannini, Simone Marinai, Valeria Nardoni, Kimiya Noor Ali
Computation and Language · Cross-listed from cs.AI
VeriCypher: Named-Entity Verification for GraphRAG via Text-to-Cypher Generation
Anton Gusarov, Anastasia Volkova, Valentin Khrulkov et al. (+3 more)
Knowledge graph question answering has traditionally focused on Resource Description Framework (RDF) graphs, SPARQL queries, and triple-based representations. Labeled Property Graphs (LPGs), however, pose a different challenge for LLM-based query generation: executable Cypher queries must align with schema structure, traversal direction, property keys, and literal entity values. This motivated us to introduce VeriCypher, an LLM-based text-to-Cypher framework for question answering over LPGs. It combines query generation conditioned on Cypher-like graph schema summaries with runtime execution, LLM-based query evaluation, and iterative correction where the central mechanism is named-entity verification: the system extracts node labels, relationship patterns, property keys, and literal values from generated queries, checks them against the target graph and schema, and turns mismatches into correction instructions for the next generation step. We evaluate VeriCypher on the CypherBench
benchmark across multiple domains and LLM backbones. Compared with an LLM baseline, this method consistently improves answer correctness and query robustness. Ablation results further show that named-entity verification helps correct hallucinated schema elements and entity values, supporting more reliable LLM-based question answering over heterogeneous property graphs.
Read the full abstract ▾
All authors Anton Gusarov, Anastasia Volkova, Valentin Khrulkov, Andrey Kuznetsov, Evgenii Maslov, Ivan Oseledets
Computation and Language
ParliaBench: An Evaluation and Benchmarking Framework for LLM-Generated Parliamentary Speech
Marios Koniaris, Argyro Tsipi and Panayiotis Tsanakas
Parliamentary
speech generation presents specific challenges for large
language models beyond standard text generation tasks. Unlike general text generation, parliamentary
speeches require not only linguistic quality but also political authenticity and ideological consistency. Current
language models lack specialized training for parliamentary contexts, and existing evaluation methods focus on standard NLP metrics rather than political authenticity. To address this, we present ParliaBench, a
benchmark for parliamentary
speech generation. We constructed a
dataset of
speeches from UK Parliament to enable systematic model training. We introduce an evaluation framework combining computational metrics with LLM-as-a-judge assessments for measuring generation quality across three dimensions: linguistic quality, semantic coherence, and political authenticity. We propose two novel
embedding-based metrics, Political Spectrum
Alignment and Party
Alignment, to quantify ideological positioning. We fine-tuned five large
language models (LLMs), generated 28k
speeches, and evaluated them using our framework, comparing baseline and fine-tuned models. Results show that fine-tuning produces statistically significant improvements across the majority of metrics and our novel metrics demonstrate strong discriminative power for political dimensions.
Read the full abstract ▾
Computation and Language
Prompt Tuning for Natural Language to SQL with Embedding Fine-Tuning and RAG
Jisoo Jang, Tien-Cuong Bui, Yunjun Choi et al. (+1 more)
This paper introduces an Error Correction through
Prompt Tuning for NL-to-SQL, leveraging the latest advancements in generative pre-training-based LLMs and RAG. Our work addresses the crucial need for efficient and accurate translation of natural
language queries into SQL expressions in various settings with the growing use of natural
language interfaces. We explore the
evolution of NLIDBs from early rule-based systems to advanced neural network-driven approaches. Drawing inspiration from the medical diagnostic process, we propose a novel framework integrating an error correction mechanism that diagnoses error types, identifies their causes, provides fixing instructions, and applies these corrections to SQL queries. This approach is further enriched by
embedding fine-tuning and RAG, which harnesses external knowledge bases for improved accuracy and transparency. Through comprehensive experiments, we demonstrate that our framework achieves a significant 12 percent accuracy improvement over existing baselines, highlighting its potential to revolutionize data access and handling in contemporary data-driven environments.
Read the full abstract ▾
All authors Jisoo Jang, Tien-Cuong Bui, Yunjun Choi, Wen-Syan Li
Computation and Language · Cross-listed from cs.AI
Towards Outcome-Oriented, Task-Agnostic Evaluation of AI Agents
Waseem AlShikh, Muayad Sayed Ali, Brian Kennedy et al. (+1 more)
As AI
agents proliferate across industries and applications, evaluating their performance based solely on infrastructural metrics such as latency, time-to-first-
token, or
token throughput is proving insufficient. These metrics fail to capture the quality of an
agent's decisions, its operational autonomy, or its ultimate business value. This white paper proposes a novel, comprehensive framework of eleven outcome-based, task-agnostic performance metrics for AI
agents that transcend domain boundaries. These metrics are designed to enable organizations to evaluate
agents based on the quality of their decisions, their degree of autonomy, their adaptability to new challenges, and the tangible business value they deliver, regardless of the underlying model architecture or specific use case. We introduce metrics such as Goal Completion Rate (GCR), Autonomy Index (AIx), Multi-Step Task Resilience (MTR), and Business Impact Efficiency (BIE). Through a large-scale simulated experiment involving four distinct
agent architectures (ReAct, Chain-of-Thought, Tool-Augmented, Hybrid) across five diverse domains (Healthcare, Finance, Marketing, Legal, and Customer Service), we demonstrate the framework's efficacy. Our results reveal significant performance trade-offs between different
agent designs, highlighting the Hybrid
Agent as the most consistently high-performing model across the majority of our proposed metrics, achieving an average Goal Completion Rate of 88.8% and the highest Return on Investment (ROI). This work provides a robust, standardized methodology for the holistic evaluation of AI
agents, paving the way for more effective development, deployment, and governance.
Read the full abstract ▾
All authors Waseem AlShikh, Muayad Sayed Ali, Brian Kennedy, Dmytro Mozolevskyi
Computation and Language
VocalBench-zh: Decomposing and Benchmarking the Speech Conversational Abilities in Mandarin Context
Heyang Liu, Ziyang Cheng, Yuhao Wang et al. (+6 more)
The development of multi-modal large
language models (LLMs) leads to intelligent approaches capable of
speech interactions. As one of the most widely spoken
languages globally, Mandarin is supported by most models to enhance their applicability and reach. However, the scarcity of comprehensive
speech-to-
speech (S2S)
benchmarks in Mandarin contexts impedes systematic evaluation for developers and hinders fair model comparison for users. In this work, we propose VocalBench-zh, an ability-level divided evaluation suite adapted to Mandarin context consisting of 10 well-crafted subsets and over 10K high-quality instances, covering 12 user-oriented characters. The evaluation experiment on 14 mainstream models reveals the common challenges for current routes, and highlights the need for new insights into next-generation
speech interactive systems. The evaluation codes and
datasets will be available at https://github.com/SJTU-OmniAgent/VocalBench-zh.
Read the full abstract ▾
All authors Heyang Liu, Ziyang Cheng, Yuhao Wang, Hongcheng Liu, Yiqi Li, Ronghua Wu, Qunshan Gu, Yanfeng Wang, Yu Wang
Computation and Language
Benchmarking Educational LLMs with Analytics: A Case Study on Gender Bias in Feedback
Yishan Du, Conrad Borchers and Mutlu Cukurova
As teachers increasingly turn to GenAI in their educational practice, we need robust methods to
benchmark large
language models (LLMs) for pedagogical purposes. This article presents an
embedding-based benchmarking framework to detect bias in LLMs in the context of formative feedback. Using 600 authentic student essays from the AES 2.0 corpus, we constructed controlled counterfactuals along two dimensions: (i) implicit cues via lexicon-based swaps of gendered terms within essays, and (ii) explicit cues via gendered author background in the
prompt. We investigated six representative LLMs (i.e. GPT-5 mini, GPT-4o mini, DeepSeek-R1, DeepSeek-R1-Qwen, Gemini 2.5 Pro, Llama-3-8B). We first quantified the response divergence with cosine and Euclidean distances over sentence
embeddings, then assessed significance via permutation tests, and finally, visualised structure using dimensionality reduction. In all models, implicit manipulations reliably induced larger semantic shifts for male-female counterfactuals than for female-male. Only the GPT and Llama models showed sensitivity to explicit gender cues. These findings show that even state-of-the-art LLMs exhibit asymmetric semantic responses to gender substitutions, suggesting persistent gender biases in feedback they provide learners. Qualitative analyses further revealed consistent linguistic differences (e.g., more autonomy-supportive feedback under male cues vs. more controlling feedback under female cues). We discuss implications for
fairness auditing of pedagogical GenAI, propose reporting standards for counterfactual evaluation in learning analytics, and outline practical guidance for
prompt design and deployment to safeguard equitable feedback.
Read the full abstract ▾
Computation and Language
Encoder Fine-tuning with Stochastic Sampling Outperforms Open-weight GPT in Astronomy Knowledge Extraction
Shivam Rawat, Lucie Flek and Akbar Karimi
Scientific literature in astronomy is rapidly expanding, making it increasingly important to automate the extraction of key entities and contextual information from research papers. In this paper, we present an encoder-based system for extracting knowledge from astronomy articles. Our objective is to develop models capable of classifying
telescope references, detecting auxiliary semantic attributes, and recognizing instrument mentions from textual content. To this end, we implement a multi-task
transformer-based system built upon the SciBERT model and fine-tuned for astronomy corpora classification. To carry out the fine-tuning, we stochastically sample segments from the training data and use majority voting over the test segments at inference time. Our system, despite its simplicity and low-cost implementation, significantly outperforms the open-weight GPT baseline.
Read the full abstract ▾
Computation and Language
Do Syntactic Categories Help in Developmentally Motivated Curriculum Learning for Language Models?
Arzu Burcu Güven, Anna Rogers and Rob van der Goot
We examine the syntactic properties of BabyLM corpus, and age-groups within CHILDES. While we find that CHILDES does not exhibit strong syntactic differentiation by age, we show that the syntactic knowledge about the training data can be helpful in interpreting model performance on linguistic tasks. For curriculum learning, we explore developmental and several alternative cognitively inspired curriculum approaches. We find that some curricula help with reading tasks, but the main performance improvement come from using the subset of syntactically categorizable data, rather than the full noisy corpus.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.AI
SciAgent: A Unified Multi-Agent System for Generalistic Scientific Reasoning
Xuchen Li, Ruitao Wu, Xuanbo Liu et al. (+17 more)
Recent advances in large
language models have enabled AI systems to achieve expert-level performance on domain-specific scientific tasks, yet these systems remain narrow and handcrafted. We introduce SciAgent, a unified multi-
agent system designed for generalistic scientific
reasoning-the ability to adapt
reasoning strategies across disciplines and difficulty levels. SciAgent organizes problem solving as a hierarchical process: a Coordinator
Agent interprets each problem's domain and complexity, dynamically orchestrating specialized Worker Systems, each composed of interacting
reasoning Sub-
agents for symbolic deduction, conceptual modeling, numerical computation, and verification. These
agents collaboratively assemble and refine
reasoning pipelines tailored to each task. Across mathematics and physics Olympiads (IMO, IMC, IPhO, CPhO), SciAgent consistently attains or surpasses human gold-medalist performance, demonstrating both domain generality and
reasoning adaptability. Additionally, SciAgent has been tested on the International Chemistry Olympiad (IChO) and selected problems from the Humanity's Last Exam (HLE)
benchmark, further confirming the system's ability to generalize across diverse scientific domains. This work establishes SciAgent as a concrete step toward generalistic scientific intelligence-AI systems capable of coherent, cross-disciplinary
reasoning at expert levels.
Read the full abstract ▾
All authors Xuchen Li, Ruitao Wu, Xuanbo Liu, Xukai Wang, Jinbo Hu, Zhixin Bai, Bohan Zeng, Hao Liang, Leheng Chen, Mingrui Chen, Haitian Zhong, Xuanlin Yang, Xu-Yao Zhang, Liu Liu, Jia Li, Kaiqi Huang, Jiahao Xu, Haitao Mi, Wentao Zhang, Bin Dong
Computation and Language
Still Not There: Can LLMs Outperform Smaller Task-Specific Seq2Seq Models on the Poetry-to-Prose Conversion Task?
Kunal Kingkar Das, Manoj Balaji Jagadeeshan, Nallani Chakravartula Sahith et al. (+2 more)
Large
Language Models (LLMs) are increasingly treated as universal, general-purpose solutions across NLP tasks, particularly in English. But does this assumption hold for low-resource, morphologically rich
languages such as Sanskrit? We address this question by comparing instruction-tuned and in-context-prompted LLMs with smaller task-specific encoder-decoder models on the Sanskrit poetry-to-prose conversion task. This task is intrinsically challenging: Sanskrit verse exhibits free word order combined with rigid metrical constraints, and its conversion to canonical prose (anvaya) requires multi-step
reasoning involving compound
segmentation, dependency resolution, and syntactic linearisation. This makes it an ideal testbed to evaluate whether LLMs can surpass specialised models. For LLMs, we apply instruction fine-tuning on general-purpose models and design in-context learning templates grounded in Paninian grammar and classical commentary heuristics. For task-specific modelling, we fully fine-tune a ByT5-Sanskrit Seq2Seq model. Our experiments show that domain-specific fine-tuning of ByT5-Sanskrit significantly outperforms all instruction-driven LLM approaches. Human evaluation strongly corroborates this result, with scores exhibiting high correlation with Kendall's Tau scores. Additionally, our prompting strategies provide an alternative to fine-tuning when domain-specific verse corpora are unavailable, and the task-specific Seq2Seq model demonstrates robust generalisation on out-of-domain evaluations.
Read the full abstract ▾
All authors Kunal Kingkar Das, Manoj Balaji Jagadeeshan, Nallani Chakravartula Sahith, Jivnesh Sandhan, Pawan Goyal
Computation and Language
Relation as a Prior: A Novel Paradigm for LLM-based Document-level Relation Extraction
Qiankun Pi, Yepeng Sun, Jicang Lu et al. (+3 more)
Large
Language Models (LLMs) have demonstrated their remarkable capabilities in document understanding. However, recent research reveals that LLMs still exhibit performance gaps in Document-level Relation Extraction (DocRE) as requiring fine-grained comprehension. The commonly adopted "extract entities then predict relations" paradigm in LLM-based methods leads to these gaps due to two main reasons: (1) Numerous unrelated entity pairs introduce noise and interfere with the relation prediction for truly related entity pairs. (2) Although LLMs have identified semantic associations between entities, relation labels beyond the predefined set are still treated as prediction errors. To address these challenges, we propose a novel Relation as a Prior (RelPrior) paradigm for LLM-based DocRE. For challenge (1), RelPrior utilizes binary relation as a prior to extract and determine whether two entities are correlated, thereby filtering out irrelevant entity pairs and reducing prediction noise. For challenge (2), RelPrior utilizes predefined relation as a prior to match entities for triples extraction instead of directly predicting relation. Thus, it avoids misjudgment caused by strict predefined relation labeling. Extensive experiments on two
benchmarks demonstrate that RelPrior achieves state-of-the-art performance, surpassing existing LLM-based methods.
Read the full abstract ▾
All authors Qiankun Pi, Yepeng Sun, Jicang Lu, Qinlong Fan, Ningbo Huang, Shiyu Wang
Computation and Language
On the Interplay between Positional Encodings, Morphological Complexity, and Word Order Flexibility
Kushal Tatariya, Wessel Poelman and Miryam de Lhoneux
Language model architectures are predominantly first created for English and subsequently applied to other
languages. It is an open question whether this architectural bias leads to degraded performance for
languages that are structurally different from English. We examine one specific architectural choice: positional encodings, through the lens of the trade-off hypothesis: the supposed interplay between morphological complexity and word order flexibility. This hypothesis posits a trade-off between the two: a more morphologically complex
language can have a more flexible word order, and vice-versa. Positional encodings are a direct target to investigate the implications of this hypothesis in relation to
language modelling. We pretrain monolingual model variants with absolute, relative, and no positional encodings for seven typologically diverse
languages and evaluate them on four downstream tasks. Contrary to previous findings, we do not observe a clear interaction between position encodings and morphological complexity or word order flexibility, as measured by various proxies. Our results show that the choice of tasks,
languages, and metrics are essential for drawing stable conclusions
Read the full abstract ▾
Computation and Language
Sentence-Anchored Gist Compression for Long-Context LLMs
Dmitrii Tarasov, Elizaveta Goncharova and Kuznetsov Andrey
This work investigates context compression for Large
Language Models (LLMs) using learned compression
tokens to reduce the
memory and computational demands of processing long sequences. We demonstrate that pre-trained LLMs can be fine-tuned to compress their context by factors of 2x to 8x without significant performance degradation, as evaluated on both short-context and long-context
benchmarks. Furthermore, in experiments on a 3-billion-parameter LLaMA model, our method achieves results on par with alternative compression techniques while attaining higher compression ratios.
Read the full abstract ▾
Computation and Language
Quantification and object perception in Multimodal Large Language Models and human linguistic cognition
Raquel Montero, Natalia Moskvina, Paolo Morosi et al. (+3 more)
Quantification has been proven to be a particularly difficult linguistic phenomenon for (
Multimodal) Large
Language Models (MLLMs). However, given that quantification interfaces with the logic, pragmatic, and numerical domains, the exact reasons for the poor performance are still unclear. This paper looks at three key features of human quantification shared cross-linguistically that have remained so far unexplored in the (M)LLM literature: the ordering of quantifiers into scales, the ranges of use and prototypicality, and the biases inherent in the human approximate number system. The aim is to determine how these features are encoded in the models' architecture, how they may differ from humans, and whether the results are affected by the type of model (thinking vs. instruct) and the
language under investigation. Results show that although thinking models showed a high accuracy in the numerosity estimation task and in the organization of quantifiers into scales, there are still key differences between humans and LLMs across all model types, particularly in terms of ranges of use and prototypicality values. This work, thus, paves the way for addressing the nature of MLLMs as semantic and pragmatic
agents, while the cross-linguistic lens can elucidate whether their abilities are robust and stable across different
languages.
Read the full abstract ▾
All authors Raquel Montero, Natalia Moskvina, Paolo Morosi, Tamara Serrano, Elena Pagliarini, Evelina Leivada
Computation and Language
Multimodal LLMs Do Not Compose Skills Optimally Across Modalities
Paula Ontalvilla, Aitor Ormazabal and Gorka Azkune
Skill composition is the ability to combine previously learned skills to solve new tasks. As neural networks acquire increasingly complex skills during their pretraining, it is not clear how successfully they can compose them. In this paper, we focus on
Multimodal Large
Language Models (MLLM), and study their ability to compose skills across modalities. To this end, we design three evaluation tasks which can be solved sequentially composing two modality-dependent skills, and evaluate several open MLLMs under two main settings: i) prompting the model to directly solve the task, and ii) using a two-step cascaded inference approach, which manually enforces the composition of the two skills for a given task. Even with these straightforward compositions, we find that all evaluated MLLMs exhibit a significant cross-modality skill composition gap. To mitigate the aforementioned gap, we explore two alternatives: i) use chain-of-thought prompting to explicitly instruct MLLMs for skill composition and ii) a specific fine-tuning recipe to promote skill composition. Although those strategies improve model performance, they still exhibit significant skill composition gaps, suggesting that more research is needed to improve cross-modal skill composition in MLLMs.
Read the full abstract ▾
Computation and Language
Estranged Predictions: Measuring Semantic Category Disruption with Masked Language Modelling
Yuxuan Liu, Haim Dubossarsky and Ruth Ahnert
This paper examines how science fiction destabilises ontological categories by measuring conceptual permeability across the terms human, animal, and machine using masked
language modelling (MLM). Drawing on corpora of science fiction (Gollancz SF Masterworks) and general fiction (NovelTM), we operationalise Darko Suvin's theory of estrangement as computationally measurable deviation in
token prediction, using RoBERTa to generate lexical substitutes for masked referents and classifying them via Gemini. We quantify conceptual slippage through three metrics: retention rate, replacement rate, and
entropy, mapping the stability or disruption of category boundaries across genres. Our findings reveal that science fiction exhibits heightened conceptual permeability, particularly around machine referents, which show significant cross-category substitution and dispersion. Human terms, by contrast, maintain semantic coherence and often anchor substitutional hierarchies. These patterns suggest a genre-specific restructuring within anthropocentric logics. We argue that estrangement in science fiction operates as a controlled perturbation of semantic norms, detectable through probabilistic modelling, and that MLMs, when used critically, serve as interpretive instruments capable of surfacing genre-conditioned ontological assumptions. This study contributes to the methodological repertoire of computational literary studies and offers new insights into the linguistic infrastructure of science fiction.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.RO
PerspAct: Enhancing LLM Situated Collaboration Skills through Perspective Taking and Active Vision
Sabrina Patania, Luca Annese, Anita Pellegrini et al. (+8 more)
Recent advances in Large
Language Models (LLMs) and
multimodal foundation models have significantly broadened their application in robotics and collaborative systems. However, effective multi-
agent interaction necessitates robust perspective-taking capabilities, enabling models to interpret both physical and epistemic viewpoints. Current training paradigms often neglect these interactive contexts, resulting in challenges when models must reason about the subjectivity of individual perspectives or navigate environments with multiple observers. This study evaluates whether explicitly incorporating diverse points of view using the ReAct framework, an approach that integrates
reasoning and acting, can enhance an LLM's ability to understand and ground the demands of other
agents. We extend the classic Director task by introducing active visual exploration across a suite of seven scenarios of increasing perspective-taking complexity. These scenarios are designed to challenge the
agent's capacity to resolve referential ambiguity based on visual access and interaction, under varying state representations and prompting strategies, including ReAct-style
reasoning. Our results demonstrate that explicit perspective cues, combined with active exploration strategies, significantly improve the model's interpretative accuracy and collaborative effectiveness. These findings highlight the potential of integrating active perception with perspective-taking mechanisms in advancing LLMs' application in robotics and multi-
agent systems, setting a foundation for future research into adaptive and context-aware AI systems.
Read the full abstract ▾
All authors Sabrina Patania, Luca Annese, Anita Pellegrini, Silvia Serino, Anna Lambiase, Luca Pallonetto, Silvia Rossi, Simone Colombani, Tom Foulsham, Azzurra Ruggeri, Dimitri Ognibene
Computation and Language · Cross-listed from eess.AS
Quantizing Whisper-small: How design choices affect ASR performance
Arthur Söhler, Julian Irigoyen and Andreas Søeborg Kirkedal
Large
speech recognition models like Whisper-small achieve high accuracy but are difficult to deploy on edge devices due to their high computational demand. To this end, we present a unified, cross-library evaluation of post-training quantization (PTQ) on Whisper-small that disentangles the impact of quantization scheme, method, granularity, and bit-width. Our study is based on four libraries: PyTorch, Optimum-Quanto, HQQ, and bitsandbytes. Experiments on LibriSpeech test-clean and test-other show that dynamic int8 quantization with Quanto offers the best trade-off, reducing model size by 57% while improving on the baseline's word error rate. Static quantization performed worse, likely due to Whisper's
Transformer architecture, while more aggressive formats (e.g., nf4, int3) achieved up to 71% compression at the cost of accuracy in noisy conditions. Overall, our results demonstrate that carefully chosen PTQ methods can substantially reduce model size and inference cost without retraining, enabling efficient deployment of Whisper-small on constrained hardware.
Read the full abstract ▾
Computation and Language · Cross-listed from eess.AS
Pruning as Regularization: Sensitivity-Aware One-Shot Pruning in ASR
Julian Irigoyen, Arthur Söhler and Andreas Søeborg Kirkedal
We challenge the conventional view of neural network
pruning as solely a compression technique, demonstrating that one-shot magnitude
pruning serves as a powerful implicit regularizer for ASR. Using Whisper-small, we combine gradient- and Fisher-based sensitivity diagnostics with targeted, component-wise
pruning. This reveals architectural asymmetries: decoder FFNs are
pruning-fragile, whereas decoder self-
attention and the last encoder layers contain redundancy that, when removed, improves generalization. Without fine-tuning,
pruning 50% of decoder self-
attention reduces WER by 2.38% absolute (20.44% relative) on LibriSpeech test-other;
pruning the last four encoder layers at 50% instead yields a 1.72% absolute (14.8% relative) improvement. Gains persisted on Common Voice and TED-LIUM
datasets. Beyond regularization benefits, our sensitivity-aware approach enables more aggressive one-shot compression. At 40% sparsity, where established global
pruning approaches catastrophically fail, our method preserves near-baseline accuracy. This positions
pruning as a first-class architectural design tool: knowing where to prune is as important as how much to prune.
Read the full abstract ▾
Computation and Language
BARD10: A New Benchmark Reveals Significance of Bangla Stop-Words in Authorship Attribution
Abdullah Muhammad Moosa, Nusrat Sultana, Mahdi Muhammad Moosa et al. (+1 more)
This research presents a comprehensive investigation into Bangla authorship attribution, introducing a new balanced
benchmark corpus BARD10 (Bangla Authorship Recognition
Dataset of 10 authors) and systematically analyzing the impact of stop-word removal across classical and deep learning models to uncover the stylistic significance of Bangla stop-words. BARD10 is a curated corpus of Bangla blog and opinion prose from ten contemporary authors, alongside the methodical assessment of four representative classifiers: SVM (Support Vector Machine), Bangla BERT (Bidirectional Encoder Representations from
Transformers), XGBoost, and a MLP (Multilayer Perception), utilizing uniform preprocessing on both BARD10 and the
benchmark corpora BAAD16 (Bangla Authorship Attribution
Dataset of 16 authors). In all
datasets, the classical TF-IDF + SVM baseline outperformed, attaining a macro-F1 score of 0.997 on BAAD16 and 0.921 on BARD10, while Bangla BERT lagged by as much as five points. This study reveals that BARD10 authors are highly sensitive to stop-word
pruning, while BAAD16 authors remain comparatively robust highlighting genre-dependent reliance on stop-word signatures. Error analysis revealed that high frequency components transmit authorial signatures that are diminished or reduced by
transformer models. Three insights are identified: Bangla stop-words serve as essential stylistic indicators; finely calibrated ML models prove effective within short-text limitations; and BARD10 connects formal literature with contemporary web dialogue, offering a reproducible
benchmark for future long-context or domain-adapted
transformers.
Read the full abstract ▾
All authors Abdullah Muhammad Moosa, Nusrat Sultana, Mahdi Muhammad Moosa, Md. Miraiz Hossain
Computation and Language · Cross-listed from cs.AI
Information Capacity: Evaluating the Efficiency of Large Language Models via Text Compression
Cheng Yuan, Jiawei Shao and Xuelong Li
Recent years have witnessed the rapid advancements of large
language models (LLMs) and their expanding applications, leading to soaring demands for computational resources. The widespread adoption of test-time scaling further intensifies the tension between model capability and resource consumption. However, a rigorous metric that accurately reflects an LLM's inference efficiency across diverse tokenizers, parameter counts, and model architectures remains absent. Motivated by the correlation between compression and intelligence, we introduce information capacity, a measure of model efficiency based on text compression performance relative to computational complexity. A distinctive feature of information capacity is its incorporation of tokenizer efficiency, which affects inference costs but is often neglected in LLM evaluations. We assess the information capacity of 56 open-source models and observe a consistent information capacity among different-sized models within a series. Experiments on five heterogeneous
datasets reveal strong linguistic biases in mainstream LLMs. Empirical results verify the accuracy of performance prediction across model sizes based on information capacity and show the correlation between information capacity and
benchmark scores. This metric can be used to quantify improvements in inference efficiency and provide insights into better scaling performance for future LLM development.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.AI
Dual-Process Scaffold Reasoning for Enhancing LLM Code Debugging
Po-Chung Hsieh, Chin-Po Chen, Jeng-Lin Li et al. (+1 more)
Recent LLMs have demonstrated sophisticated problem-solving capabilities on various
benchmarks through advanced
reasoning algorithms. However, the key research question of identifying
reasoning steps that balance complexity and computational efficiency remains unsolved. Recent research has increasingly drawn upon psychological theories to explore strategies for optimizing cognitive pathways. The LLM's final outputs and intermediate steps are regarded as System 1 and System 2, respectively. However, an in-depth exploration of the System 2
reasoning is still lacking. Therefore, we propose a novel psychologically backed Scaffold
Reasoning framework for code debugging, which encompasses the Scaffold Stream, Analytic Stream, and Integration Stream. The construction of reference code within the Scaffold Stream is integrated with the buggy code analysis results produced by the Analytic Stream through the Integration Stream. Our framework achieves an 88.91% pass rate and an average inference time of 5.36 seconds per-problem on DebugBench, outperforming other
reasoning approaches across various LLMs in both
reasoning accuracy and efficiency. Further analyses elucidate the advantages and limitations of various cognitive pathways across varying problem difficulties and bug types. Our findings also corroborate the
alignment of the proposed Scaffold
Reasoning framework with human cognitive processes.
Read the full abstract ▾
All authors Po-Chung Hsieh, Chin-Po Chen, Jeng-Lin Li, Ming-Ching Chang
Computation and Language · Cross-listed from cs.LG
DynaAct: Large Language Model Reasoning with Dynamic Action Spaces
Xueliang Zhao, Wei Wu, Jian Guan et al. (+2 more)
In modern sequential decision-making systems, the construction of an optimal candidate action space is critical to efficient inference. However, existing approaches either rely on manually defined action spaces that lack scalability or utilize unstructured spaces that render exhaustive search computationally prohibitive. In this paper, we propose a novel framework named DynaAct for automatically constructing a compact action space to enhance sequential
reasoning in complex problem-solving scenarios. Our method first estimates a proxy for the complete action space by extracting general sketches observed in a corpus covering diverse complex
reasoning problems using large
language models. We then formulate a submodular function that jointly evaluates candidate actions based on their utility to the current state and their diversity, and employ a greedy
algorithm to select an optimal candidate set. Extensive experiments on six diverse standard
benchmarks demonstrate that our approach significantly improves overall performance, while maintaining efficient inference without introducing substantial latency. The implementation is available at https://github.com/zhaoxlpku/DynaAct.
Read the full abstract ▾
All authors Xueliang Zhao, Wei Wu, Jian Guan, Qintong Li, Lingpeng Kong
Computation and Language · Cross-listed from cs.IR
BiCA: Effective Biomedical Dense Retrieval with Citation-Aware Hard Negatives
Aarush Sinha, Pavan Kumar S, Roshan Balaji et al. (+1 more)
Hard negatives are essential for training effective
retrieval models. Hard-negative mining typically relies on ranking documents using cross-encoders or static
embedding models based on similarity metrics such as cosine distance. Hard negative mining becomes challenging for biomedical and scientific domains due to the difficulty in distinguishing between source and hard negative documents. However, referenced documents naturally share contextual relevance with the source document but are not duplicates, making them well-suited as hard negatives. In this work, we propose BiCA: Biomedical Dense
Retrieval with Citation-Aware Hard Negatives, an approach for hard-negative mining by utilizing citation links in 20,000 PubMed articles for improving a domain-specific small dense retriever. We fine-tune the GTE_small and GTE_Base models using these citation-informed negatives and observe consistent improvements in zero-shot dense
retrieval using nDCG@10 for both in-domain and out-of-domain tasks on BEIR and outperform baselines on long-tailed topics in LoTTE using Success@5. Our findings highlight the potential of leveraging document link structure to generate highly informative negatives, enabling state-of-the-art performance with minimal fine-tuning and demonstrating a path towards highly data-efficient domain adaptation.
Read the full abstract ▾
All authors Aarush Sinha, Pavan Kumar S, Roshan Balaji, Nirav Pravinbhai Bhatt
Computation and Language
HyCoRA: Hyper-Contrastive Role-Adaptive Learning for Role-Playing
Shihao Yang, Zhicong Lu, Yong Yang et al. (+3 more)
Multi-character role-playing aims to equip models with the capability to simulate diverse roles. Existing methods either use one shared parameterized module across all roles or assign a separate parameterized module to each role. However, the role-shared module may ignore distinct traits of each role, weakening personality learning, while the role-specific module may overlook shared traits across multiple roles, hindering commonality modeling. In this paper, we propose a novel HyCoRA: Hyper-Contrastive Role-Adaptive learning framework, which efficiently improves multi-character role-playing ability by balancing the learning of distinct and shared traits. Specifically, we propose a Hyper-Half Low-Rank Adaptation structure, where one half is a role-specific module generated by a lightweight hyper-network, and the other half is a trainable role-shared module. The role-specific module is devised to represent distinct persona signatures, while the role-shared module serves to capture common traits. Moreover, to better reflect distinct personalities across different roles, we design a hyper-contrastive learning mechanism to help the hyper-network distinguish their unique characteristics. Extensive experimental results on both English and Chinese available
benchmarks demonstrate the superiority of our framework. Further GPT-4 evaluations and visual analyses also verify the capability of HyCoRA to capture role characteristics.
Read the full abstract ▾
All authors Shihao Yang, Zhicong Lu, Yong Yang, Bo Lv, Yang Shen, Nayu Liu
Computation and Language
Self-Correction Distillation for Structured Data Question Answering
Yushan Zhu, Wen Zhang, Long Jin et al. (+8 more)
Structured data question answering (QA), including table QA, Knowledge Graph (KG) QA, and temporal KG QA, is a pivotal research area. Advances in large
language models (LLMs) have driven significant progress in unified structural QA frameworks like TrustUQA. However, these frameworks face challenges when applied to small-scale LLMs since small-scale LLMs are prone to errors in generating structured queries. To improve the structured data QA ability of small-scale LLMs, we propose a self-correction
distillation (SCD) method. In SCD, an error
prompt mechanism (EPM) is designed to detect errors and provide customized error messages during inference, and a two-stage
distillation strategy is designed to transfer large-scale LLMs' query-generation and error-correction capabilities to small-scale LLM. Experiments across 5
benchmarks with 3 structured data types demonstrate that our SCD achieves the best performance and superior generalization on small-scale LLM (8B) compared to other
distillation methods, and closely approaches the performance of GPT4 on some
datasets. Furthermore, large-scale LLMs equipped with EPM surpass the state-of-the-art results on most
datasets.
Read the full abstract ▾
All authors Yushan Zhu, Wen Zhang, Long Jin, Mengshu Sun, Ling Zhong, Zhiqiang Liu, Juan Li, Lei Liang, Chong Long, Chao Deng, Junlan Feng
Computation and Language
State of the Art in Text Classification for South Slavic Languages: Fine-Tuning or Prompting?
Taja Kuzman Pungeršek, Peter Rupnik, Ivan Porupski et al. (+2 more)
Until recently, fine-tuned BERT-like models provided state-of-the-art performance on text classification tasks. With the rise of instruction-tuned decoder-only models, commonly known as large
language models (LLMs), the field has increasingly moved toward zero-shot and few-shot prompting. However, the performance of LLMs on text classification, particularly on less-resourced
languages, remains under-explored. In this paper, we evaluate the performance of current
language models on text classification tasks across several South Slavic
languages. We compare openly available fine-tuned BERT-like models with a selection of open-source and closed-source LLMs across three tasks in three domains: sentiment classification in parliamentary
speeches, topic classification in news articles and parliamentary
speeches, and genre identification in web texts. Our results show that LLMs demonstrate strong zero-shot performance, often matching or surpassing fine-tuned BERT-like models. Moreover, when used in a zero-shot setup, LLMs perform comparably in South Slavic
languages and English. However, we also point out key drawbacks of LLMs, including less predictable outputs, significantly slower inference, and higher computational costs. Due to these limitations, fine-tuned BERT-like models remain a more practical choice for large-scale automatic text annotation.
Read the full abstract ▾
All authors Taja Kuzman Pungeršek, Peter Rupnik, Ivan Porupski, Vuk Dinić, Nikola Ljubešić
Computation and Language
NOTAM-Evolve: A Knowledge-Guided Self-Evolving Optimization Framework with LLMs for NOTAM Interpretation
Maoqi Liu, Quan Fang, Yuhao Wu et al. (+3 more)
Accurate interpretation of Notices to Airmen (NOTAMs) is critical for aviation safety, yet their condensed and cryptic
language poses significant challenges to both manual and automated processing. Existing automated systems are typically limited to shallow parsing, failing to extract the actionable intelligence needed for operational decisions. We formalize the complete interpretation task as deep parsing, a dual-
reasoning challenge requiring both dynamic knowledge grounding (linking the NOTAM to evolving real-world aeronautical data) and schema-based inference (applying static domain rules to deduce operational status). To tackle this challenge, we propose NOTAM-Evolve, a self-evolving framework that enables a large
language model (LLM) to autonomously master complex NOTAM interpretation. Leveraging a knowledge graph-enhanced
retrieval module for data grounding, the framework introduces a closed-loop learning process where the LLM progressively improves from its own outputs, minimizing the need for extensive human-annotated
reasoning traces. In conjunction with this framework, we introduce a new
benchmark dataset of 10,000 expert-annotated NOTAMs. Our experiments demonstrate that NOTAM-Evolve achieves a 30.4% absolute accuracy improvement over the base LLM, establishing a new state of the art on the task of structured NOTAM interpretation.
Read the full abstract ▾
All authors Maoqi Liu, Quan Fang, Yuhao Wu, Can Zhao, Yang Yang, Kaiquan Cai
Computation and Language
Unified Work Embeddings: Contrastive Learning of a Bidirectional Multi-task Ranker
Matthias De Lange, Jens-Joris Decorte and Jeroen Van Hautte
Applications in labor
market intelligence demand specialized NLP systems for a wide range of tasks, characterized by extreme multi-label target spaces, strict latency constraints, and multiple text modalities such as skills and job titles. These constraints have led to isolated, task-specific developments in the field, with models and
benchmarks focused on single prediction tasks. Exploiting the shared structure of work-related data, we propose a unifying framework, combining a wide range of tasks in a multi-task ranking
benchmark, and a flexible architecture tackling text-driven work tasks with a single model. The
benchmark, WorkBench, is the first unified evaluation suite spanning six work-related tasks formulated explicitly as ranking problems, curated from real-world ontologies and human-annotated resources. WorkBench enables cross-task analysis, where we find significant positive cross-task transfer. This insight leads to Unified Work
Embeddings (UWE), a task-agnostic bi-encoder that exploits our training-data structure with a many-to-many InfoNCE objective, and leverages
token-level
embeddings with task-agnostic soft late interaction. UWE demonstrates zero-shot ranking performance on unseen target spaces in the work domain, and enables low-latency inference with two orders of magnitude fewer parameters than best-performing generalist models (Qwen3-8B), with +4.4 MAP improvement.
Read the full abstract ▾