Computation and Language · Cross-listed from cs.CV
Closing the Gap: Data-Centric Fine-Tuning of Vision Language Models for the Standardized Exam Questions
Egemen Sert and Şeyda Ertekin
Multimodal reasoning has become a cornerstone of modern AI research. Standardized exam questions offer a uniquely rigorous testbed for such
reasoning, providing structured visual contexts and verifiable answers. While recent progress has largely focused on algorithmic advances such as reinforcement learning (e.g., GRPO, DPO), the data centric foundations of
vision language reasoning remain less explored. We show that supervised fine-tuning (SFT) with high-quality data can rival proprietary approaches. To this end, we compile a 161.4 million
token multimodal dataset combining textbook question-solution pairs, curriculum aligned diagrams, and contextual materials, and fine-tune Qwen-2.5VL-32B using an optimized
reasoning syntax (QMSA). The resulting model achieves 78.6% accuracy, only 1.0% below Gemini 2.0 Flash, on our newly released
benchmark YKSUniform, which standardizes 1,854
multimodal exam questions across 309 curriculum topics. Our results reveal that data composition and representational syntax play a decisive role in
multimodal reasoning. This work establishes a data centric framework for advancing open weight
vision language models, demonstrating that carefully curated and curriculum-grounded
multimodal data can elevate supervised fine-tuning to near state-of-the-art performance.
Read the full abstract ▾
Computation and Language
MindSET: Advancing Mental Health Benchmarking through Large-Scale Social Media Data
Saad Mankarious, Ayah Zirikly, Daniel Wiechmann et al. (+3 more)
Social media data has become a vital resource for studying mental health, offering real-time insights into thoughts,
emotions, and behaviors that traditional methods often miss. Progress in this area has been facilitated by
benchmark datasets for mental health analysis; however, most existing
benchmarks have become outdated due to limited data availability, inadequate cleaning, and the inherently diverse nature of social media content (e.g.,
multilingual and harmful material). We present a new
benchmark dataset, MindSET, curated from Reddit using self-reported diagnoses to address these limitations. The annotated
dataset contains over 13M annotated posts across seven mental health conditions, more than twice the size of previous
benchmarks. To ensure data quality, we applied rigorous preprocessing steps, including
language filtering, and removal of Not Safe for Work (NSFW) and duplicate content. We further performed a linguistic analysis using LIWC to examine psychological term frequencies across the eight groups represented in the
dataset. To demonstrate the
dataset utility, we conducted binary classification experiments for diagnosis detection using both fine-tuned
language models and Bag-of-Words (BoW) features. Models trained on MindSET consistently outperformed those trained on previous
benchmarks, achieving up to an 18-point improvement in F1 for Autism detection. Overall, MindSET provides a robust foundation for researchers exploring the intersection of social media and mental health, supporting both early risk detection and deeper analysis of emerging psychological trends.
Read the full abstract ▾
All authors Saad Mankarious, Ayah Zirikly, Daniel Wiechmann, Elma Kerz, Edward Kempa, Yu Qiao
Computation and Language
Structured Definitions and Segmentations for Legal Reasoning in LLMs: A Study on Indian Legal Data
Mann Khatri, Mirza Yusuf, Rajiv Ratn Shah et al. (+1 more)
Large
Language Models (LLMs), trained on extensive
datasets from the web, exhibit remarkable general
reasoning skills. Despite this, they often struggle in specialized areas like law, mainly because they lack domain-specific pretraining. The legal field presents unique challenges, as legal documents are generally long and intricate, making it hard for models to process the full text efficiently. Previous studies have examined in-context approaches to address the knowledge gap, boosting model performance in new domains without full domain
alignment. In our paper, we analyze model behavior on legal tasks by conducting experiments in three areas: (i) reorganizing documents based on rhetorical roles to assess how structured information affects long context processing and model decisions, (ii) defining rhetorical roles to familiarize the model with legal terminology, and (iii) emulating the step-by-step
reasoning of courts regarding rhetorical roles to enhance model
reasoning. These experiments are conducted in a zero-shot setting across three Indian legal judgment prediction
datasets. Our results reveal that organizing data or explaining key legal terms significantly boosts model performance, with a minimum increase of ~1.5% and a maximum improvement of 4.36% in F1 score compared to the baseline.
Read the full abstract ▾
All authors Mann Khatri, Mirza Yusuf, Rajiv Ratn Shah, Ponnurangam Kumaraguru
Computation and Language
PIRA: Preference-Oriented Instruction-Tuned Reward Models with Dual Aggregation
Yongfu Xue
Reward models are pivotal for aligning Large
Language Models (LLMs) with human preferences. Existing approaches face two key limitations: Discriminative reward models require large-scale annotated data, as they cannot exploit the preference instruction-following capability of LLMs available to generative reward models. Moreover, reward models are particularly prone to reward overoptimization, where LLMs exploit weaknesses in the reward function instead of improving true
alignment. We introduce PIRA, a training paradigm that integrates three complementary strategies to address these challenges: (1) reformulating question-answer pairs into preference-task instructions to explicitly leverage LLMs' preference instruction-following capability, (2) averaging the rewards aggregated from diverse preference-task instructions for each sample, which mitigates task-specific bias and enhances robustness across evaluation perspectives, and (3) averaging outputs from the value head under different dropout rates to stabilize reward estimation. Experiments on public
datasets show that PIRA improves performance considerably, enhances generalization, and effectively mitigates reward overoptimization.
Read the full abstract ▾
Computation and Language
Random Text, Zipf's Law, Critical Length,and Implications for Large Language Models
Vladimir Berman
We study a deliberately simple, fully non-linguistic model of text: a sequence of independent draws from a finite alphabet of letters plus a single space symbol. A word is defined as a maximal block of non-space symbols. Within this symbol-level framework, which assumes no morphology, syntax, or semantics, we derive several structural results. First, word lengths follow a
geometric distribution governed solely by the probability of the space symbol. Second, the expected number of words of a given length, and the expected number of distinct words of that length, admit closed-form expressions based on a coupon-collector argument. This yields a critical word length k* at which word types transition from appearing many times on average to appearing at most once. Third, combining the exponential growth of the number of possible strings of length k with the exponential decay of the probability of each string, we obtain a Zipf-type rank-frequency law p(r) proportional to r^{-alpha}, with an exponent determined explicitly by the alphabet size and the space probability. Our contribution is twofold. Mathematically, we give a unified derivation linking word lengths, vocabulary growth, critical length, and rank-frequency structure in a single explicit model. Conceptually, we argue that this provides a structurally grounded null model for both natural-
language word statistics and
token statistics in large
language models. The results show that Zipf-like patterns can arise purely from combinatorics and
segmentation, without optimization principles or linguistic organization, and help clarify which phenomena require deeper explanation beyond random-text structure.
Read the full abstract ▾
Computation and Language
Community-Aligned Behavior Under Uncertainty: Evidence of Epistemic Stance Transfer in LLMs
Patrick Gerard, Aiden Chang and Svitlana Volkova
When large
language models (LLMs) are aligned to a specific online community, do they exhibit generalizable behavioral patterns that mirror that community's attitudes and responses to new uncertainty, or are they simply recalling patterns from training data? We introduce a framework to test epistemic stance transfer: targeted deletion of event knowledge, validated with multiple probes, followed by evaluation of whether models still reproduce the community's organic response patterns under ignorance. Using Russian–Ukrainian military discourse and U.S. partisan Twitter data, we find that even after aggressive fact removal, aligned LLMs maintain stable, community-specific behavioral patterns for handling uncertainty. These results provide evidence that
alignment encodes structured, generalizable behaviors beyond surface mimicry. Our framework offers a systematic way to detect behavioral biases that persist under ignorance, advancing efforts toward safer and more transparent LLM deployments.
Read the full abstract ▾
Computation and Language
Generative Caching for Structurally Similar Prompts and Responses
Sarthak Chakraborty, Suman Nath, Xuchao Zhang et al. (+2 more)
Large
Language Models (LLMs) are increasingly being used to plan, reason, and execute tasks across diverse scenarios. In use cases like repeatable workflows and agentic settings,
prompts are often reused with minor variations while having a similar structure for recurring tasks. This opens up opportunities for caching. However, exact
prompt matching fails on such structurally similar
prompts, while semantic caching may produce incorrect responses by ignoring critical differences. To address this, we introduce \ourmethod{}, a generative cache that produces variation-aware responses for structurally similar
prompts. \ourmethod{} identifies reusable response patterns across similar
prompt structures and synthesizes customized outputs for new requests. We show that \ourmethod{} achieves 83% cache hit rate, while having minimal incorrect hits on
datasets without
prompt repetition. In agentic workflows, it improves cache hit rate by $\sim$20% and reduces end-to-end execution latency by $\sim$34% compared to standard
prompt matching.
Read the full abstract ▾
All authors Sarthak Chakraborty, Suman Nath, Xuchao Zhang, Chetan Bansal, Indranil Gupta
Computation and Language
Prompt-Based Value Steering of Large Language Models
Giulio Antonio Abbo and Tony Belpaeme
Large
language models are increasingly used in applications where
alignment with human values is critical. While model fine-tuning is often employed to ensure safe responses, this technique is static and does not lend itself to everyday situations involving dynamic values and preferences. In this paper, we present a practical, reproducible, and model-agnostic procedure to evaluate whether a
prompt candidate can effectively steer generated text toward specific human values, formalising a scoring method to quantify the presence and gain of target values in generated responses. We apply our method to a variant of the Wizard-Vicuna
language model, using Schwartz's theory of basic human values and a structured evaluation through a dialogue
dataset. With this setup, we compare a baseline
prompt to one explicitly conditioned on values, and show that value steering is possible even without altering the model or dynamically optimising
prompts.
Read the full abstract ▾
Computation and Language
Improving LLM's Attachment to External Knowledge In Dialogue Generation Tasks Through Entity Anonymization
Hadi Sheikhi, Chenyang Huang and Osmar R. Zaïane
Knowledge graph-based dialogue generation (KG-DG) is a challenging task requiring models to effectively incorporate external knowledge into conversational responses. While large
language models (LLMs) have achieved impressive results across various NLP tasks, their ability to utilize external knowledge in KG-DG remains under-explored. We observe that LLMs often rely on internal knowledge, leading to detachment from provided knowledge graphs, even when they are given a flawlessly retrieved knowledge graph. First, we introduce LLM-KAT, an evaluation procedure for measuring knowledge attachment in generated responses. Second, we propose a simple yet effective entity anonymization technique to encourage LLMs to better leverage external knowledge. Experiments on the OpenDialKG
dataset demonstrate that our approach improves LLMs' attachment on external knowledge.
Read the full abstract ▾
Computation and Language
InData: Towards Secure Multi-Step, Tool-Based Data Analysis
Karthikeyan K, Raghuveer Thirukovalluru, Bhuwan Dhingra et al. (+1 more)
Large
language model
agents for data analysis typically generate and execute code directly on databases. However, when applied to sensitive data, this approach poses significant security risks. To address this issue, we propose a security-motivated alternative: restrict LLMs from direct code generation and data access, and require them to interact with data exclusively through a predefined set of secure, verified tools. Although recent tool-use
benchmarks exist, they primarily target tool selection and simple execution rather than the compositional, multi-step
reasoning needed for complex data analysis. To reduce this gap, we introduce Indirect Data Engagement (InData), a
dataset designed to assess LLMs' multi-step tool-based
reasoning ability. InData includes data analysis questions at three difficulty levels–Easy, Medium, and Hard–capturing increasing
reasoning complexity. We
benchmark 15 open-source LLMs on InData and find that while large models (e.g., gpt-oss-120b) achieve high accuracy on Easy tasks (97.3%), performance drops sharply on Hard tasks (69.6%). These results show that current LLMs still lack robust multi-step tool-based
reasoning ability. With InData, we take a step toward enabling the development and evaluation of LLMs with stronger multi-step tool-use capabilities. We will publicly release the
dataset and code.
Read the full abstract ▾
All authors Karthikeyan K, Raghuveer Thirukovalluru, Bhuwan Dhingra, David Edwin Carlson
Computation and Language
Additive Large Language Models for Semi-Structured Text
Karthikeyan K, Raghuveer Thirukovalluru and David Carlson
Large
Language Models have advanced clinical text classification, but their opaque predictions remain a critical barrier to practical adoption in research and clinical settings where investigators and physicians need to understand which parts of a patient's record drive risk signals. To address this challenge, we introduce CALM, short for Classification with Additive Large
Language Models, an interpretable framework for semi-structured text where inputs are composed of semantically meaningful components, such as sections of an admission note or question-answer fields from an intake form. CALM predicts outcomes as the additive sum of each component's contribution, making these contributions part of the forward computation itself and enabling faithful explanations at both the patient and population level. The additive structure also enables clear visualizations, such as component-level risk curves similar to those used in generalized additive models, making the learned relationships easier to inspect and communicate. Although CALM expects semi-structured inputs, many clinical documents already have this form, and similar structure can often be automatically extracted from free-text notes. CALM achieves performance comparable to conventional LLM classifiers while improving trust, supporting quality-assurance checks, and revealing clinically meaningful patterns during model development and
auditing.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.AI
Forgetting-MarI: LLM Unlearning via Marginal Information Regularization
Shizhou Xu, Yuan Ni, Stefan Broecker et al. (+1 more)
As AI models are trained on ever-expanding
datasets, the ability to remove the influence of specific data from trained models has become essential for
privacy protection and regulatory compliance. Unlearning addresses this challenge by selectively removing parametric knowledge from the trained models without retraining from scratch, which is critical for resource-intensive models such as Large
Language Models (LLMs). Existing unlearning methods often degrade model performance by removing more information than necessary when attempting to ”forget” specific data. We introduce Forgetting-MarI, an LLM unlearning framework that provably removes only the additional (marginal) information contributed by the data to be unlearned, while preserving the information supported by the data to be retained. By penalizing marginal information, our method yields an explicit upper bound on the unlearn
dataset's residual influence in the trained models, providing provable undetectability. Extensive experiments confirm that our approach outperforms current state-of-the-art unlearning methods, delivering reliable forgetting and better preserved general model performance across diverse
benchmarks. This advancement represents an important step toward making AI systems more controllable and compliant with
privacy and copyright regulations without compromising their effectiveness.
Read the full abstract ▾
All authors Shizhou Xu, Yuan Ni, Stefan Broecker, Thomas Strohmer
Computation and Language
Context-Emotion Aware Therapeutic Dialogue Generation: A Multi-component Reinforcement Learning Approach to Language Models for Mental Health Support
Eric Hua Qing Zhang and Julia Ive
Mental health disorders impose a substantial global socioeconomic burden. While large
language models (LLMs) offer 24/7, non-judgmental interactions to address this gap, pretrained models lack contextual coherence and emotional
alignment for appropriate therapeutic dialogue. Existing methods suffer from three critical methodological gaps: 1) Supervised Fine-Tuning (SFT) produces repetitive, context-insensitive outputs that fail to balance clinical accuracy with genuine empathy; 2) Reinforcement Learning (RL)-based therapeutic systems rely on generic reward functions (e.g., BLEU, ROUGE) that prioritise lexical similarity over clinical-specific emotional appropriateness and contextual relevance; 3) LLMs are resource-intensive and pose data
privacy risks, making local deployment in clinical settings infeasible. To address these gaps, this study investigates the application of SFT and RL techniques to enhance GPT-2's capacity for therapeutic dialogue generation. The methodology restructured input formats to enable simultaneous processing of contextual information and emotional states alongside user input, employing a novel multi-component reward function that explicitly aligns model outputs with professional therapeutic logic (not just lexical overlap) and annotated
emotions. Results demonstrated substantial improvements through RLs over baseline GPT-2 across multiple evaluation metrics: BLEU (0.0111), ROUGE-1 (0.1397), ROUGE-2 (0.0213), ROUGE-L (0.1317), and METEOR (0.0581). LLM evaluation confirmed high contextual relevance and professionalism, while RL achieved 99.34%
emotion accuracy compared to 66.96% for baseline GPT-2. These findings demonstrate RL's effectiveness in developing therapeutic dialogue systems that can serve as valuable assistive tools for therapists, while maintaining essential human clinical oversight.
Read the full abstract ▾
Computation and Language
ClinStructor: AI-Powered Structuring of Unstructured Clinical Texts
Karthikeyan K, Raghuveer Thirukovalluru and David Carlson
Clinical notes contain valuable, context-rich information, but their unstructured format introduces several challenges, including unintended biases (e.g., gender or racial bias), and poor generalization across clinical settings (e.g., models trained on one EHR system may perform poorly on another due to format differences) and poor interpretability. To address these issues, we present ClinStructor, a pipeline that leverages large
language models (LLMs) to convert clinical free-text into structured, task-specific question-answer pairs prior to predictive modeling. Our method substantially enhances transparency and controllability and only leads to a modest reduction in predictive performance (a 2-3% drop in AUC), compared to direct fine-tuning, on the ICU mortality prediction task. ClinStructor lays a strong foundation for building reliable, interpretable, and generalizable machine learning models in clinical environments.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.LG
Better LLM Reasoning via Dual-Play
Zhengxin Zhang, Chengyu Huang, Aochong Oliver Li et al. (+1 more)
Large
Language Models (LLMs) have achieved remarkable progress through Reinforcement Learning with Verifiable Rewards (RLVR), yet still rely heavily on external supervision (e.g., curated labels).
Adversarial learning, particularly through self-play, offers a promising alternative that enables models to iteratively learn from themselves - thus reducing reliance on external supervision. Dual-play extends
adversarial learning by assigning specialized roles to two models and training them against each other, fostering sustained competition and mutual
evolution. Despite its promise, adapting dual-play training to LLMs remains limited, largely due to their susceptibility to reward hacking and training instability. In this paper, we introduce PasoDoble, a novel LLM dual-play framework. PasoDoble adversarially trains two models initialized from the same base model: a Proposer, which generates challenging questions with ground-truth answers, and a Solver, which attempts to solve them. We enrich the Proposer with knowledge from a pre-training
dataset to ensure the questions' quality and diversity. To avoid reward hacking, the Proposer is rewarded for producing only valid questions that push the Solver's limit, while the Solver is rewarded for solving them correctly, and both are updated jointly. To further enhance training stability, we introduce an optional offline paradigm that decouples Proposer and Solver updates, alternately updating each for several steps while holding the other fixed. Notably, PasoDoble operates without supervision during training. Experimental results show that PasoDoble can improve the
reasoning performance of LLMs. Our project page is available at https://hcy123902.github.io/PasoDoble.
Read the full abstract ▾
All authors Zhengxin Zhang, Chengyu Huang, Aochong Oliver Li, Claire Cardie
Computation and Language
MedPT: A Massive Medical Question Answering Dataset for Brazilian-Portuguese Speakers
Fernanda Bufon Färber, Iago Alves Brito, Julia Soares Dollis et al. (+3 more)
While large
language models (LLMs) show transformative potential in healthcare, their development remains focused on high-resource
languages. This creates a critical barrier for other
languages, as simple translation fails to capture unique clinical and cultural nuances, such as endemic diseases. To address this, we introduce MedPT, the first large-scale, real-world corpus of patient-doctor interactions for the Brazilian Portuguese medical domain. Comprising 384,095 authentic question-answer pairs and covering over 3,200 distinct health-related conditions, the
dataset was refined through a rigorous multi-stage curation protocol that employed a hybrid quantitative-qualitative analysis to filter noise and contextually enrich thousands of ambiguous queries, resulting in a corpus of approximately 57 million
tokens. We further utilize of LLM-driven annotation to classify queries into seven semantic types to capture user intent. To validate MedPT's utility, we
benchmark it in a medical specialty classification task: fine-tuning a 1.7B parameter model achieves an outstanding 94% F1-score on a 20-class setup. Furthermore, our qualitative error analysis shows misclassifications are not random but reflect genuine clinical ambiguities (e.g., between comorbid conditions), proving the
dataset's deep semantic richness. We publicly release MedPT on Hugging Face to support the development of more equitable, accurate, and culturally-aware medical technologies for the Portuguese-speaking world.
Read the full abstract ▾
All authors Fernanda Bufon Färber, Iago Alves Brito, Julia Soares Dollis, Pedro Schindler Freire Brasil Ribeiro, Rafael Teixeira Sousa, Arlindo Rodrigues Galvão Filho
Computation and Language
Identifying Imaging Follow-Up in Radiology Reports: A Comparative Analysis of Traditional ML and LLM Approaches
Namu Park, Giridhar Kaushik Ramachandran, Kevin Lybarger et al. (+4 more)
Large
language models (LLMs) have shown considerable promise in clinical natural
language processing, yet few domain-specific
datasets exist to rigorously evaluate their performance on radiology tasks. In this work, we introduce an annotated corpus of 6,393 radiology reports from 586 patients, each labeled for follow-up imaging status, to support the development and benchmarking of follow-up adherence detection systems. Using this corpus, we systematically compared traditional machine-learning classifiers, including logistic regression (LR), support vector machines (SVM), Longformer, and a fully fine-tuned Llama3-8B-Instruct, with recent generative LLMs. To evaluate generative LLMs, we tested GPT-4o and the open-source GPT-OSS-20B under two configurations: a baseline (Base) and a task-optimized (Advanced) setting that focused inputs on metadata, recommendation sentences, and their surrounding context. A refined
prompt for GPT-OSS-20B further improved
reasoning accuracy. Performance was assessed using precision, recall, and F1 scores with 95% confidence intervals estimated via non-parametric bootstrapping. Inter-annotator agreement was high (F1 = 0.846). GPT-4o (Advanced) achieved the best performance (F1 = 0.832), followed closely by GPT-OSS-20B (Advanced; F1 = 0.828). LR and SVM also performed strongly (F1 = 0.776 and 0.775), underscoring that while LLMs approach human-level agreement through
prompt optimization, interpretable and resource-efficient models remain valuable baselines.
Read the full abstract ▾
All authors Namu Park, Giridhar Kaushik Ramachandran, Kevin Lybarger, Fei Xia, Ozlem Uzuner, Meliha Yetisgen, Martin Gunn
Computation and Language
Three Stage Narrative Analysis; Plot-Sentiment Breakdown, Structure Learning and Concept Detection
Taimur Khan, Ramoza Ahsan and Mohib Hameed
Story understanding and analysis have long been challenging areas within Natural
Language Understanding. Automated narrative analysis requires deep computational semantic representations along with syntactic processing. Moreover, the large volume of narrative data demands automated semantic analysis and computational learning rather than manual analytical approaches. In this paper, we propose a framework that analyzes the sentiment arcs of movie scripts and performs extended analysis related to the context of the characters involved. The framework enables the extraction of high-level and low-level concepts conveyed through the narrative. Using dictionary-based sentiment analysis, our approach applies a custom lexicon built with the LabMTsimple storylab module. The custom lexicon is based on the Valence, Arousal, and Dominance scores from the NRC-VAD
dataset. Furthermore, the framework advances the analysis by clustering similar sentiment plots using Wards hierarchical clustering technique. Experimental evaluation on a movie
dataset shows that the resulting analysis is helpful to consumers and readers when selecting a narrative or story.
Read the full abstract ▾
Computation and Language
Towards Autoformalization of LLM-generated Outputs for Requirement Verification
Mihir Gupte and Ramesh S
Autoformalization, the process of translating informal statements into formal logic, has gained renewed interest with the emergence of powerful Large
Language Models (LLMs). While LLMs show promise in generating structured outputs from natural
language (NL), such as Gherkin Scenarios from NL feature requirements, there's currently no formal method to verify if these outputs are accurate. This paper takes a preliminary step toward addressing this gap by exploring the use of a simple LLM-based autoformalizer to verify LLM-generated outputs against a small set of natural
language requirements. We conducted two distinct experiments. In the first one, the autoformalizer successfully identified that two differently-worded NL requirements were logically equivalent, demonstrating the pipeline's potential for consistency checks. In the second, the autoformalizer was used to identify a logical inconsistency between a given NL requirement and an LLM-generated output, highlighting its utility as a formal verification tool. Our findings, while limited, suggest that autoformalization holds significant potential for ensuring the fidelity and logical consistency of LLM-generated outputs, laying a crucial foundation for future, more extensive studies into this novel application.
Read the full abstract ▾
Computation and Language
Scaling Open-Weight Large Language Models for Hydropower Regulatory Information Extraction: A Systematic Analysis
Hong-Jun Yoon, Faisal Ashraf, Thomas A. Ruggles et al. (+1 more)
Information extraction from regulatory documents using large
language models presents critical trade-offs between performance and computational resources. We evaluated seven open-weight models (0.6B-70B parameters) on hydropower licensing documentation to provide empirical deployment guidance. Our analysis identified a pronounced 14B parameter threshold where validation methods transition from ineffective (F1 $<$ 0.15) to viable (F1 = 0.64). Consumer-deployable models achieve 64% F1 through appropriate validation, while smaller models plateau at 51%. Large-scale models approach 77% F1 but require enterprise infrastructure. We identified systematic
hallucination patterns where perfect recall indicates extraction failure rather than success in smaller models. Our findings establish the first comprehensive resource-performance mapping for open-weight information extraction in regulatory contexts, enabling evidence-based model selection. These results provide immediate value for hydropower compliance while contributing insights into parameter scaling effects that generalize across information extraction tasks.
Read the full abstract ▾
All authors Hong-Jun Yoon, Faisal Ashraf, Thomas A. Ruggles, Debjani Singh
Computation and Language · Cross-listed from cs.AI
Do LLMs Really Struggle at NL-FOL Translation? Revealing their Strengths via a Novel Benchmarking Strategy
Andrea Brunello, Luca Geatti, Michele Mignani et al. (+2 more)
Due to its expressiveness and unambiguous nature, First-Order Logic (FOL) is a powerful formalism for representing concepts expressed in natural
language (NL). This is useful, e.g., for specifying and verifying desired system properties. While translating FOL into human-readable English is relatively straightforward, the inverse problem, converting NL to FOL (NL-FOL translation), has remained a longstanding challenge, for both humans and machines. Although the emergence of Large
Language Models (LLMs) promised a breakthrough, recent literature provides contrasting results on their ability to perform NL-FOL translation. In this work, we provide a threefold contribution. First, we critically examine existing
datasets and protocols for evaluating NL-FOL translation performance, revealing key limitations that may cause a misrepresentation of LLMs' actual capabilities. Second, to overcome these shortcomings, we propose a novel evaluation protocol explicitly designed to distinguish genuine semantic-level logical understanding from superficial pattern recognition, memorization, and
dataset contamination. Third, using this new approach, we show that state-of-the-art, dialogue-oriented LLMs demonstrate strong NL-FOL translation skills and a genuine grasp of sentence-level logic, whereas
embedding-centric models perform markedly worse.
Read the full abstract ▾
All authors Andrea Brunello, Luca Geatti, Michele Mignani, Angelo Montanari, Nicola Saccomanno
Computation and Language
On the Notion that Language Models Reason
Bertram Højer
Language models (LMs) are said to be exhibiting
reasoning, but what does this entail? We assess definitions of
reasoning and how key papers in the field of natural
language processing (NLP) use the notion and argue that the definitions provided are not consistent with how LMs are trained, process information, and generate new
tokens. To illustrate this incommensurability we assume the view that
transformer-based LMs implement an implicit finite-order Markov kernel mapping contexts to conditional
token distributions. In this view,
reasoning-like outputs correspond to statistical regularities and approximate statistical invariances in the learned kernel rather than the implementation of explicit logical mechanisms. This view is illustrative of the claim that LMs are "statistical pattern matchers"" and not genuine reasoners and provides a perspective that clarifies why
reasoning-like outputs arise in LMs without any guarantees of logical consistency. This distinction is fundamental to how epistemic uncertainty is evaluated in LMs. We invite a discussion on the importance of how the computational processes of the systems we build and analyze in NLP research are described.
Read the full abstract ▾
Computation and Language
MiroThinker: Pushing the Performance Boundaries of Open-Source Research Agents via Model, Context, and Interactive Scaling
MiroMind Team, Song Bai, Lidong Bing et al. (+52 more)
We present MiroThinker v1.0, an open-source research
agent designed to advance tool-augmented
reasoning and information-seeking capabilities. Unlike previous
agents that only scale up model size or context length, MiroThinker explores interaction scaling at the model level, systematically training the model to handle deeper and more frequent
agent-environment interactions as a third dimension of performance improvement. Unlike LLM test-time scaling, which operates in isolation and risks degradation with longer
reasoning chains, interactive scaling leverages environment feedback and external information acquisition to correct errors and refine trajectories. Through reinforcement learning, the model achieves efficient interaction scaling: with a 256K context window, it can perform up to 600 tool calls per task, enabling sustained multi-turn
reasoning and complex real-world research workflows. Across four representative
benchmarks-
GAIA, HLE, BrowseComp, and BrowseComp-ZH-the 72B variant achieves up to 81.9%, 37.7%, 47.1%, and 55.6% accuracy respectively, surpassing previous open-source
agents and approaching commercial counterparts such as GPT-5-high. Our analysis reveals that MiroThinker benefits from interactive scaling consistently: research performance improves predictably as the model engages in deeper and more frequent
agent-environment interactions, demonstrating that interaction depth exhibits scaling behaviors analogous to model size and context length. These findings establish interaction scaling as a third critical dimension for building next-generation open research
agents, complementing model capacity and context windows.
Read the full abstract ▾
All authors MiroMind Team, Song Bai, Lidong Bing, Carson Chen, Guanzheng Chen, Yuntao Chen, Zhe Chen, Ziyi Chen, Jifeng Dai, Xuan Dong, Wenhan Dou, Yue Deng, Yunjie Fu, Junqi Ge, Chenxia Han, Tammy Huang, Zhenhang Huang, Jerry Jiao, Shilei Jiang, Tianyu Jiao, Xiaoqi Jian, Lei Lei, Ruilin Li, Gen Luo, Tiantong Li, Xiang Lin, Ziyuan Liu, Zhiqi Li, Jie Ni, Qiang Ren, Pax Sun, Shiqian Su, Chenxin Tao, Bin Wang, Wenhai Wang, Haonan Wang, James Wang, Jin Wang, Jojo Wang, Letian Wang, Shizun Wang, Weizhi Wang, Zixuan Wang, Jinfan Xu, Sen Xing, Chenyu Yang, Hai Ye, Jiaheng Yu, Yue Yu, Muyan Zhong, Tianchen Zhao, Xizhou Zhu, Yanpeng Zhou, Yifan Zhang, Zhi Zhu
Computation and Language · Cross-listed from cs.LG
Optimizing Mixture of Block Attention
Guangxuan Xiao, Junxian Guo, Kasra Mazaheri et al. (+1 more)
Mixture of Block
Attention (MoBA) (Lu et al., 2025) is a promising building block for efficiently processing long contexts in LLMs by enabling queries to sparsely attend to a small subset of key-value blocks, drastically reducing computational cost. However, the design principles governing MoBA's performance are poorly understood, and it lacks an efficient GPU implementation, hindering its practical adoption. In this paper, we first develop a statistical model to analyze MoBA's underlying mechanics. Our model reveals that performance critically depends on the router's ability to accurately distinguish relevant from irrelevant blocks based on query-key affinities. We derive a signal-to-noise ratio that formally connects architectural parameters to this
retrieval accuracy. Guided by our analysis, we identify two key pathways for improvement: using smaller block sizes and applying a short convolution on keys to cluster relevant signals, which enhances routing accuracy. While theoretically better, small block sizes are inefficient on GPUs. To bridge this gap, we introduce FlashMoBA, a hardware-aware CUDA kernel that enables efficient MoBA execution even with the small block sizes our theory recommends. We validate our insights by training LLMs from scratch, showing that our improved MoBA models match the performance of dense
attention baselines. FlashMoBA achieves up to 14.7x speedup over FlashAttention-2 for small blocks, making our theoretically-grounded improvements practical. Code is available at: https://github.com/mit-han-lab/flash-moba.
Read the full abstract ▾
All authors Guangxuan Xiao, Junxian Guo, Kasra Mazaheri, Song Han
Computation and Language
PRBench: Large-Scale Expert Rubrics for Evaluating High-Stakes Professional Reasoning
Afra Feyza Akyürek, Advait Gosai, Chen Bo Calvin Zhang et al. (+21 more)
Frontier model progress is often measured by academic
benchmarks, which offer a limited view of performance in real-world professional contexts. Existing evaluations often fail to assess open-ended, economically consequential tasks in high-stakes domains like Legal and Finance, where practical returns are paramount. To address this, we introduce Professional
Reasoning Bench (PRBench), a realistic, open-ended, and difficult
benchmark of real-world problems in Finance and Law. We open-source its 1,100 expert-authored tasks and 19,356 expert-curated criteria, making it, to our knowledge, the largest public, rubric-based
benchmark for both legal and finance domains. We recruit 182 qualified professionals, holding JDs, CFAs, or 6+ years of experience, who contributed tasks inspired by their actual workflows. This process yields significant diversity, with tasks spanning 114 countries and 47 US jurisdictions. Our expert-curated rubrics are validated through a rigorous quality pipeline, including independent expert validation. Subsequent evaluation of 20 leading models reveals substantial room for improvement, with top scores of only 0.39 (Finance) and 0.37 (Legal) on our Hard subsets. We further catalog associated economic impacts of the
prompts and analyze performance using human-annotated rubric categories. Our analysis shows that models with similar overall scores can diverge significantly on specific capabilities. Common failure modes include inaccurate judgments, a lack of process transparency and incomplete
reasoning, highlighting critical gaps in their reliability for professional adoption.
Read the full abstract ▾
All authors Afra Feyza Akyürek, Advait Gosai, Chen Bo Calvin Zhang, Vipul Gupta, Jaehwan Jeong, Anisha Gunjal, Tahseen Rabbani, Maria Mazzone, David Randolph, Mohammad Mahmoudi Meymand, Gurshaan Chattha, Paula Rodriguez, Diego Mares, Pavit Singh, Michael Liu, Subodh Chawla, Pete Cline, Lucy Ogaz, Ernesto Hernandez, Zihao Wang, Pavi Bhatter, Marcos Ayestaran, Bing Liu, Yunzhong He
Computation and Language · Cross-listed from cs.CV
DocLens : A Tool-Augmented Multi-Agent Framework for Long Visual Document Understanding
Dawei Zhu, Rui Meng, Jiefeng Chen et al. (+3 more)
Comprehending long visual documents, where information is distributed across extensive pages of text and visual elements, is a critical but challenging task for modern
Vision-
Language Models (VLMs). Existing approaches falter on a fundamental challenge: evidence localization. They struggle to retrieve relevant pages and overlook fine-grained details within visual elements, leading to limited performance and model
hallucination. To address this, we propose DocLens, a tool-augmented multi-
agent framework that effectively “zooms in” on evidence like a lens. It first navigates from the full document to specific visual elements on relevant pages, then employs a sampling-adjudication mechanism to generate a single, reliable answer. Paired with Gemini-2.5-Pro, DocLens achieves state-of-the-art performance on MMLongBench-Doc and FinRAGBench-V, surpassing even human experts. The framework's superiority is particularly evident on
vision-centric and unanswerable queries, demonstrating the power of its enhanced localization capabilities.
Read the full abstract ▾
All authors Dawei Zhu, Rui Meng, Jiefeng Chen, Sujian Li, Tomas Pfister, Jinsung Yoon
Computation and Language · Cross-listed from cs.AI
Aligning Machiavellian Agents: Behavior Steering via Test-Time Policy Shaping
Dena Mujtaba, Brian Hu, Anthony Hoogs et al. (+1 more)
The deployment of decision-making AI
agents presents a critical challenge in maintaining
alignment with human values or guidelines while operating in complex, dynamic environments.
Agents trained solely to achieve their objectives may adopt harmful behavior, exposing a key trade-off between maximizing the reward function and maintaining
alignment. For pre-trained
agents, ensuring
alignment is particularly challenging, as retraining can be a costly and slow process. This is further complicated by the diverse and potentially conflicting attributes representing the ethical values for
alignment. To address these challenges, we propose a test-time
alignment technique based on model-guided policy shaping. Our method allows precise control over individual behavioral attributes, generalizes across diverse reinforcement learning (RL) environments, and facilitates a principled trade-off between ethical
alignment and reward maximization without requiring
agent retraining. We evaluate our approach using the MACHIAVELLI
benchmark, which comprises 134 text-based game environments and thousands of annotated scenarios involving ethical decisions. The RL
agents are first trained to maximize the reward in their respective games. At test time, we apply policy shaping via scenario-action attribute classifiers to ensure decision
alignment with ethical attributes. We compare our approach against prior training-time methods and general-purpose
agents, as well as study several types of ethical violations and power-seeking behavior. Our results demonstrate that test-time policy shaping provides an effective and scalable solution for mitigating unethical behavior across diverse environments and
alignment attributes.
Read the full abstract ▾
All authors Dena Mujtaba, Brian Hu, Anthony Hoogs, Arslan Basharat
Computation and Language
W2S-AlignTree: Weak-to-Strong Inference-Time Alignment for Large Language Models via Monte Carlo Tree Search
Zhenyu Ding, Yuhao Wang, Tengyue Xiao et al. (+3 more)
Large
Language Models (LLMs) demonstrate impressive capabilities, yet their outputs often suffer from misalignment with human preferences due to the inadequacy of weak supervision and a lack of fine-grained control. Training-time
alignment methods like Reinforcement Learning from Human Feedback (RLHF) face prohibitive costs in expert supervision and inherent scalability limitations, offering limited dynamic control during inference. Consequently, there is an urgent need for scalable and adaptable
alignment mechanisms. To address this, we propose W2S-AlignTree, a pioneering plug-and-play inference-time
alignment framework that synergistically combines Monte Carlo Tree Search (MCTS) with the Weak-to-Strong Generalization paradigm for the first time. W2S-AlignTree formulates LLM
alignment as an optimal heuristic search problem within a generative search tree. By leveraging weak model's real-time, step-level signals as
alignment proxies and introducing an
Entropy-Aware exploration mechanism, W2S-AlignTree enables fine-grained guidance during strong model's generation without modifying its parameters. The approach dynamically balances exploration and exploitation in high-dimensional generation search trees. Experiments across controlled sentiment generation, summarization, and instruction-following show that W2S-AlignTree consistently outperforms strong baselines. Notably, W2S-AlignTree raises the performance of Llama3-8B from 1.89 to 2.19, a relative improvement of 15.9 on the summarization task.
Read the full abstract ▾
All authors Zhenyu Ding, Yuhao Wang, Tengyue Xiao, Haoying Wang, Caigui Jiang, Ning Ding
Computation and Language
Proactive Hearing Assistants that Isolate Egocentric Conversations
Guilin Hu, Malek Itani, Tuochao Chen et al. (+1 more)
We introduce proactive hearing assistants that automatically identify and separate the wearer's conversation partners, without requiring explicit
prompts. Our system operates on
egocentric binaural audio and uses the wearer's self-
speech as an anchor, leveraging turn-taking behavior and dialogue dynamics to infer conversational partners and suppress others. To enable real-time, on-device operation, we propose a dual-model architecture: a lightweight streaming model runs every 12.5 ms for low-latency extraction of the conversation partners, while a slower model runs less frequently to capture longer-range conversational dynamics. Results on real-world 2- and 3-speaker conversation test sets, collected with binaural
egocentric hardware from 11 participants totaling 6.8 hours, show generalization in identifying and isolating conversational partners in multi-conversation settings. Our work marks a step toward hearing assistants that adapt proactively to conversational dynamics and engagement. More information can be found on our website: https://proactivehearing.cs.washington.edu/
Read the full abstract ▾
All authors Guilin Hu, Malek Itani, Tuochao Chen, Shyamnath Gollakota
Computation and Language · Cross-listed from cs.CV
Synthetic Stimuli, Real Gains: Rethinking VLM Fine-Tuning Through Fully Controlled Data Generation
Massimo Rizzoli, Simone Alghisi, Seyed Mahed Mousavi et al. (+1 more)
Performance gains of
Vision Language Models (VLMs) obtained by fine-tuning are generally based on ad hoc data collection and annotation of real-world scenes. Despite the improvements, this process is often prone to biases, errors, and distribution imbalance, resulting in overfitting and imbalanced performance. Although a few studies have explored synthetic data generation, they typically lack control over data distribution and annotation quality. In this work, we re-evaluate the potential of model fine-tuning by exploring a fully controlled data generation and annotation pipeline, obtaining bias-free data with balanced distribution and clean annotations. Using the spatial
reasoning task of identifying the absolute position of an object as a use case, we fine-tune state-of-the-art VLMs and conduct exhaustive evaluations on both synthetic and real-world
benchmarks, including transferability to real-world scenes. Our experiments reveal two key findings: 1) fine-tuning on balanced data yields uniform performance across the visual scene and mitigates common biases with as few as 130 samples; and 2) fine-tuning on synthetic stimuli improves performance by 13% on real-world data (COCO), outperforming models fine-tuned on the full COCO train set.
Read the full abstract ▾
All authors Massimo Rizzoli, Simone Alghisi, Seyed Mahed Mousavi, Giuseppe Riccardi