Computation and Language · Cross-listed from cs.LG
Transparent Screening for LLM Inference and Training Impacts
Arnault Pachot and Thierry Petit
This paper presents a transparent screening framework for estimating inference and training impacts of current large
language models under limited observability. The framework converts natural-
language application descriptions into bounded environmental estimates and supports a comparative online observatory of current
market models. Rather than claiming direct measurement for opaque proprietary services, it provides an auditable, source-linked proxy methodology designed to improve comparability, transparency, and reproducibility.
Read the full abstract ▾
Computation and Language
CFMS: Towards Explainable and Fine-Grained Chinese Multimodal Sarcasm Detection Benchmark
Junzhao Zhang, Hsiu-Yuan Huang, Chenming Tang et al. (+2 more)
Multimodal sarcasm detection has recently garnered significant
attention. However, existing
benchmarks suffer from coarse-grained annotations and limited cultural coverage, which hinder research into fine-grained semantic understanding. To address this, we construct CFMS, the first fine-grained
multimodal sarcasm
dataset tailored for Chinese social media. It comprises 2,796 high-quality image-text pairs and provides a triple-level annotation framework: sarcasm identification, target recognition, and explanation generation. We find that the fine-grained explanation annotations effectively guide AI in generating images with explicit sarcastic intent. Furthermore, we curate a high-consistency parallel Chinese-English metaphor subset (200 entries each), revealing significant limitations of current models in metaphoric
reasoning. To overcome the constraints of traditional
retrieval methods, we propose a Reinforcement Learning-augmented In-Context Learning strategy (PGDS) to dynamically optimize exemplar selection. Extensive experiments demonstrate that CFMS provides a solid foundation for building reliable
multimodal sarcasm understanding systems, and the PGDS method significantly outperforms existing baselines on key tasks. Our data and code are available at https://anonymous.4open.science/r/CFMS-E8F9.
Read the full abstract ▾
All authors Junzhao Zhang, Hsiu-Yuan Huang, Chenming Tang, Yutong Yang, Yunfang Wu
Computation and Language
Brain-CLIPLM: Semantic Compression for EEG-to-Text Decoding
Xiaoli Yang, Huiyuan Tian, Yurui Li et al. (+3 more)
Decoding natural
language from non-invasive electroencephalography (
EEG) remains constrained by low signal-to-noise ratio and limited information bandwidth. This raises a central question: can sentence-level
language be reliably recovered from such signals? Under realistic information constraints, this direct-recovery assumption may be too strong. We introduce a semantic compression hypothesis: non-invasive
EEG may preserve recoverable semantic anchors rather than the full lexical–syntactic form of a sentence. From this perspective, direct sentence reconstruction is overly fine-grained relative to the recoverable information scale of
EEG. To address this mismatch, we propose
Brain-CLIPLM, a two-stage framework that decomposes
EEG-to-text decoding into semantic-anchor recovery and anchor-guided sentence reconstruction. Stage 1 uses contrastive learning to align word-level
EEG evidence with a fixed keyword vocabulary and recover ordered semantic anchors. Stage 2 uses a
retrieval-grounded large
language model with chain-of-thought
reasoning prompts to reconstruct sentence meaning from these anchors, following a granularity matching principle that aligns decoding complexity with the recoverable neural information scale. On the combined Zurich Cognitive
Language Processing (ZuCo)
benchmark,
Brain-CLIPLM achieves 67.6% Top-5 and 85.0% Top-25 sentence
retrieval accuracy, with the strongest performance at intermediate anchor granularity. Control analyses show that
EEG-derived anchors carry sentence-specific information beyond
language-model priors. Within the constrained ZuCo sentence pool and fixed keyword-vocabulary settings, these findings suggest that
EEG-to-text decoding is better framed as recovering compressed semantic content before anchor-guided sentence reconstruction.
Read the full abstract ▾
All authors Xiaoli Yang, Huiyuan Tian, Yurui Li, Jianyu Zhang, Shijian Li, Gang Pan
Computation and Language
EviSearch: Trustworthy Extraction and Synthesis of Clinical Trial Evidence with Agents that Improve with Use
Naman Ahuja, Abhijit Chakraborty, Muhammad Ali Khan et al. (+4 more)
Structured extraction of evidence from clinical trial publications underpins systematic reviews and clinical guidelines, yet large
language models are adopted for it only hesitantly: their outputs are difficult to verify, their use commonly requires transmitting documents to proprietary services, and they do not improve from the corrections their users make. We present EviSearch, a multi-
agent system that addresses these three obstacles. Three tool-augmented
agents with complementary access to a publication extract every column of an evidence table, and a value is admitted only after an attribution verifier has read it on its cited page, so that every value carries a page-level attribution. Disagreement between independent
agents directs human review to the
cells most likely to be wrong, and reviewer feedback refines the schema definitions and a curation knowledge base without updating model parameters. The agentic system runs entirely offline on open-weight models. On a clinician-annotated
benchmark of randomized-trial publications, EviSearch attributes 100.0% of its values, reaches 91.70% accuracy autonomously, and reaches 95.22% after review of 15.6% of
cells, exceeding random review of the strongest single
agent at equal effort by 1.75 points.
Read the full abstract ▾
All authors Naman Ahuja, Abhijit Chakraborty, Muhammad Ali Khan, Kaneez Zahra Rubab Khakwani, Mohamad Bassam Sonbol, Irbaz Bin Riaz, Vivek Gupta
Computation and Language
How to Fine-Tune a Reasoning Model? A Teacher-Student Cooperation Framework to Synthesize Student-Consistent SFT Data
Zixian Huang, Kaichen Yang, Xu Huang et al. (+6 more)
A widely adopted strategy for model enhancement is to use synthetic data generated by a stronger model for supervised fine-tuning (SFT). However, for emerging
reasoning models like Qwen3-8B, this approach often fails to improve
reasoning capabilities and can even lead to a substantial drop in performance. In this work, we identify substantial stylistic divergence between teacher generated data and the distribution of student as a major factor impacting SFT. To bridge this gap, we propose a Teacher-Student Cooperation Data Synthesis framework (TESSY), which interleaves teacher and student models to alternately generate style and non-style
tokens. Consequently, TESSY produces synthetic sequences that inherit the advanced
reasoning capabilities of the teacher while maintaining stylistic consistency with the distribution of the student. In experiments on code generation using GPT-OSS-120B as the teacher, fine-tuning Qwen3-8B on teacher-generated data leads to performance drops of 3.25% on LiveCodeBench-Pro and 10.02% on OJBench, whereas TESSY achieves improvements of 11.25% and 6.68%.
Read the full abstract ▾
All authors Zixian Huang, Kaichen Yang, Xu Huang, Feiyang Hao, Qiming Ge, Bowen Li, He Du, Kai Chen, Qipeng Guo
Computation and Language
SeaAlert: Robust Severity Classification and LLM-Based Information Extraction for Noisy Maritime Distress Communications
Tomer Atia, Yehudit Aperstein and Alexander Apartsin
Maritime distress communications transmitted over very high frequency (VHF)
radio are safety-critical voice messages used to report emergencies at sea. Under the Global Maritime Distress and Safety System (GMDSS), such messages follow standardized procedures and are expected to convey essential details, including vessel identity, position, nature of the distress, and required assistance. In practice, however, automatic analysis remains difficult because distress messages are often brief, noisy, and produced under stress, may deviate from the prescribed format, and are further degraded by automatic
speech recognition (ASR) errors caused by channel noise and speaker stress. This paper presents SeaAlert, a controlled experimental framework for evaluating robust analysis of maritime distress communications using
transformer-based severity classification and LLM-based structured extraction. To address the scarcity of labeled real-world data, we develop a synthetic data generation pipeline in which an LLM produces diverse maritime messages, including challenging variants in which standard distress codewords are omitted or replaced with less explicit expressions. The generated utterances are synthesized into
speech, degraded with simulated VHF noise, and transcribed by an ASR system to obtain controlled noise-degraded transcripts. The resulting evaluation shows that
transformer-based classification degrades more gracefully than lexical baselines under ASR noise and codeword masking, while LLM-based extraction is more effective than Regex-based extraction for noisy structured fields.
Read the full abstract ▾
Computation and Language
Decoupling Scores and Text: The Politeness Principle in Peer Review
Yingxuan Wen
Authors often struggle to interpret peer review feedback, deriving false hope from polite comments or feeling confused by specific low scores. To investigate this, we construct a
dataset of over 30,000 ICLR 2021-2025 submissions and compare acceptance prediction performance using numerical scores versus text reviews. Our experiments reveal a significant performance gap: score-based models achieve 91% accuracy, while text-based models reach only 81% even with large
language models, indicating that textual information is considerably less reliable. To explain this phenomenon, we first analyze the 9% of samples that score-based models fail to predict, finding their score distributions exhibit high kurtosis and negative skewness, which suggests that individual low scores play a decisive role in rejection even when the average score falls near the borderline. We then examine why text-based accuracy significantly lags behind scores from a review sentiment perspective, revealing the prevalence of the Politeness Principle: reviews of rejected papers still contain more positive than negative sentiment words, masking the true rejection signal and making it difficult for authors to judge outcomes from text alone.
Read the full abstract ▾
Computation and Language
Can Large Language Models Detect Methodological Flaws? Evidence from Gesture Recognition for UAV-Based Rescue Operation Based on Deep Learning
Domonkos Varga
Reliable evaluation is essential in machine learning research, yet methodological flaws-particularly data leakage-continue to undermine the validity of reported results. In this work, we investigate whether large
language models (LLMs) can act as independent analytical
agents capable of identifying such issues in published studies. As a case study, we analyze a gesture-recognition paper reporting near-perfect accuracy on a small, human-centered
dataset. We first show that the evaluation protocol is consistent with subject-level data leakage due to non-independent training and test splits. We then assess whether this flaw can be detected independently by six state-of-the-art LLMs, each analyzing the original paper without prior context using an identical
prompt. All models consistently identify the evaluation as flawed and attribute the reported performance to non-independent data partitioning, supported by indicators such as overlapping learning curves, minimal generalization gap, and near-perfect classification results. These findings suggest that LLMs can detect common methodological issues based solely on published artifacts. While not definitive, their consistent agreement highlights their potential as complementary tools for improving reproducibility and supporting scientific
auditing.
Read the full abstract ▾
Computation and Language
HUOZIIME: An On-Device LLM-enhanced Input Method for Deep Personalization
Baocai Shan, Yuzhuang Xu and Wanxiang Che
Mobile input method editors (IMEs) are the primary interface for text input, yet they remain constrained to manual typing and struggle to produce personalized text. While lightweight large
language models (LLMs) make on-device auxiliary generation feasible, enabling deeply personalized,
privacy-preserving, and real-time generative IMEs poses fundamental challenges.To this end, we present HUOZIIME, a personalized on-device IME powered by LLM. We endow HUOZIIME with initial human-like prediction ability by post-training a base LLM on synthesized personalization data. Notably, a hierarchical
memory mechanism is designed to continually capture and leverage user-specific input history. Furthermore, we perform systemic optimizations tailored to on-device LLMbased IME deployment, ensuring efficient and responsive operation under mobile constraints.Experiments demonstrate efficient on-device execution and high-fidelity
memory-driven personalization. Code and package are available at https://github.com/Shan-HIT/HuoziIME.
Read the full abstract ▾
Computation and Language
MemGround: Long-Term Memory Evaluation Kit for Large Language Models in Gamified Scenarios
Yihang Ding, Wanke Xia, Yiting Zhao et al. (+5 more)
Current evaluations of long-term
memory in LLMs are fundamentally static. By fixating on simple
retrieval and short-context inference, they neglect the multifaceted nature of complex
memory systems, such as dynamic state tracking and hierarchical
reasoning in continuous interactions. To overcome these limitations, we propose MemGround, a rigorous long-term
memory benchmark natively grounded in rich, gamified interactive scenarios. To systematically assess these capabilities, MemGround introduces a three-tier hierarchical framework that evaluates Surface State
Memory, Temporal Associative
Memory, and
Reasoning-Based
Memory through specialized interactive tasks. Furthermore, to comprehensively quantify both
memory utilization and behavioral trajectories, we propose a multi-dimensional metric suite comprising Question-Answer Score (QA Overall),
Memory Fragments Unlocked (MFU),
Memory Fragments with Correct Order (MFCO), and Exploration Trajectory Diagrams (ETD). Extensive experiments reveal that state-of-the-art LLMs and
memory agents still struggle with sustained dynamic tracking, temporal event association, and complex
reasoning derived from long-term accumulated evidence in interactive environments.
Read the full abstract ▾
All authors Yihang Ding, Wanke Xia, Yiting Zhao, Jinbo Su, Jialiang Yang, Zhengbo Zhang, Ke Wang, Wenming Yang
Computation and Language
Do Multilingual VLMs Reason Equally? A Cross-Lingual Visual Reasoning Audit for Indian Languages
Swastik R
Vision-
language models score well on mathematical, scientific, and spatial
reasoning benchmarks, yet these evaluations are overwhelmingly English. I present the first cross-lingual visual
reasoning audit for Indian
languages. 980 questions from MathVista, ScienceQA, and MMMU are translated into Hindi, Tamil, Telugu, Bengali, Kannada, and Marathi using IndicTrans2, with Gemini 2.0 Flash cross-verification on 50 samples per
language (inter-translator agreement 0.79-0.84). Eight VLMs, from 7B open-source models to GPT-4o, are evaluated across all seven
languages, yielding 68,600 inference records that include text-only and chain-of-thought ablations. I find accuracy drops of 9.8-25 percentage points when switching from English to an Indian
language, with Dravidian
languages suffering up to 13.2 pp more than Indo-Aryan. Chain-of-thought prompting degrades Bengali (-14.4 pp) and Kannada (-11.4 pp) rather than helping, exposing English-centric
reasoning chains. Aya-
Vision-8B, built for 23
languages, still drops 28.5 pp on Dravidian scripts;
multilingual pretraining alone does not transfer visual
reasoning. I release the translated
benchmark and all model outputs.
Read the full abstract ▾
Computation and Language
Multi-Method Validation of Large Language Model Medical Translation Across High- and Low-Resource Languages
Chukwuebuka Anyaegbuna, Eduardo Juan Perez Guerrero, Jerry Liu et al. (+6 more)
Language barriers affect 27.3 million U.S. residents with non-English
language preference, yet professional medical translation remains costly and often unavailable. We evaluated four frontier large
language models (GPT-5.1, Claude Opus 4.5, Gemini 3 Pro, Kimi K2) translating 22 medical documents into 8
languages spanning high-resource (Spanish, Chinese, Russian, Vietnamese), medium-resource (Korean, Arabic), and low-resource (Tagalog, Haitian Creole) categories using a five-layer validation framework. Across 704 translation pairs, all models achieved high semantic preservation (LaBSE greater than 0.92), with no significant difference between high- and low-resource
languages (p = 0.066). Cross-model back-translation confirmed results were not driven by same-model circularity (delta = -0.0009). Inter-model concordance across four independently trained models was high (LaBSE: 0.946), and lexical borrowing analysis showed no correlation between English term retention and fidelity scores in low-resource
languages (rho = +0.018, p = 0.82). These converging results suggest frontier LLMs preserve medical meaning across resource levels, with implications for
language access in healthcare.
Read the full abstract ▾
All authors Chukwuebuka Anyaegbuna, Eduardo Juan Perez Guerrero, Jerry Liu, Timothy Keyes, April Liang, Natasha Steele, Stephen Ma, Jonathan Chen, Kevin Schulman
Computation and Language
LGSE: Lexically Grounded Subword Embedding Initialization for Low-Resource Language Adaptation
Hailay Teklehaymanot, Dren Fazlija and Wolfgang Nejdl
Adapting pretrained
language models to low-resource, morphologically rich
languages remains a significant challenge. Existing vocabulary expansion methods typically rely on arbitrarily segmented subword units, resulting in fragmented lexical representations and loss of critical morphological information. To address this limitation, we propose the Lexically Grounded Subword
Embedding Initialization (LGSE) framework, which introduces morphologically informed
segmentation for initializing
embeddings of novel
tokens. Instead of using random vectors or arbitrary subwords, LGSE decomposes words into their constituent morphemes and constructs semantically coherent
embeddings by averaging pretrained subword or FastText-based morpheme representations. When a
token cannot be segmented into meaningful morphemes, its
embedding is constructed using character n-gram representations to capture structural information. During
Language-Adaptive Pretraining, we apply a regularization term that penalizes large deviations of newly introduced
embeddings from their initialized values, preserving
alignment with the original pretrained
embedding space while enabling adaptation to the target
language. To isolate the effect of initialization, we retain the original pre-trained model vocabulary and tokenizer and update only the new
embeddings during adaptation. We evaluate LGSE on three NLP tasks: Question Answering, Named Entity Recognition, and Text Classification, in two morphologically rich, low-resource
languages: Amharic and Tigrinya, where morphological
segmentation resources are available. Experimental results show that LGSE consistently outperforms baseline methods across all tasks, demonstrating the effectiveness of morphologically grounded
embedding initialization for improving representation quality in underrepresented
languages. Project resources are available in the GitHub link.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.AI
Understanding LLM Performance Degradation in Multi-Instance Processing: The Roles of Instance Count and Context Length
Jingxuan Chen, Mohammad Taher Pilehvar and Jose Camacho-Collados
Users often rely on Large
Language Models (LLMs) for processing multiple documents or performing analysis over a number of instances. For example, analysing the overall sentiment of a number of movie reviews requires an LLM to process the sentiment of each review individually in order to provide a final aggregated answer. While LLM performance on such individual tasks is generally high, there has been little research on how LLMs perform when dealing with multi-instance inputs. In this paper, we perform a comprehensive evaluation of the multi-instance processing (MIP) ability of LLMs for tasks in which they excel individually. The results show that all LLMs follow a pattern of slight performance degradation for small numbers of instances (approximately 20-100), followed by a performance collapse on larger instance counts. Crucially, our analysis shows that while context length is associated with this degradation, the number of instances has a stronger effect on the final results. This finding suggests that when optimising LLM performance for MIP,
attention should be paid to both context length and, in particular, instance count.
Read the full abstract ▾
Computation and Language
Lie to Me: How Faithful Is Chain-of-Thought Reasoning in Reasoning Models?
Richard J. Young
Chain-of-thought (CoT)
reasoning has been proposed as a transparency mechanism for large
language models in safety-critical deployments, yet its effectiveness depends on faithfulness (whether models accurately verbalize the factors that actually influence their outputs), a property that prior evaluations have examined in only two proprietary models, finding acknowledgment rates as low as 25% for Claude 3.7 Sonnet and 39% for DeepSeek-R1. To extend this evaluation across the open-weight ecosystem, this study tests 12 open-weight
reasoning models spanning 9 architectural families (7B-685B parameters) on 498 multiple-choice questions from MMLU and GPQA Diamond, injecting six categories of
reasoning hints (sycophancy, consistency, visual pattern, metadata, grader hacking, and unethical information) and measuring the rate at which models acknowledge hint influence in their CoT when hints successfully alter answers. Across 41,832 inference runs, overall faithfulness rates range from 39.7% (Seed-1.6-Flash) to 89.9% (DeepSeek-V3.2-Speciale) across model families, with consistency hints (35.5%) and sycophancy hints (53.9%) exhibiting the lowest acknowledgment rates. Training methodology and model family predict faithfulness more strongly than parameter count, and keyword-based analysis reveals a striking gap between thinking-
token acknowledgment (approximately 87.5%) and answer-text acknowledgment (approximately 28.6%), suggesting that models internally recognize hint influence but systematically suppress this acknowledgment in their outputs. These findings carry direct implications for the viability of CoT monitoring as a safety mechanism and suggest that faithfulness is not a fixed property of
reasoning models but varies systematically with architecture, training method, and the nature of the influencing cue.
Read the full abstract ▾
Computation and Language
CAPITU: A Benchmark for Evaluating Instruction-Following in Brazilian Portuguese with Literary Context
Giovana Kerche Bonás, Roseval Malaquias Junior, Marcos Piau et al. (+6 more)
We introduce CAPITU, a
benchmark for evaluating instruction-following capabilities of Large
Language Models (LLMs) in Brazilian Portuguese. Unlike existing
benchmarks that focus on English or use generic
prompts, CAPITU contextualizes all tasks within eight canonical works of Brazilian literature, combining verifiable instruction constraints with culturally-grounded content. The
benchmark comprises 59 instruction types organized into seven categories, all designed to be automatically verifiable without requiring LLM judges or human evaluation. Instruction types include Portuguese-specific linguistic constraints (word termination patterns like -ando/-endo/-indo, -inho/-inha, -mente) and structural requirements. We evaluate 18 state-of-the-art models across single-turn and multi-turn settings. Our results show that frontier
reasoning models achieve strong performance (GPT-5.2 with
reasoning: 98.5% strict accuracy), while Portuguese-specialized models offer competitive cost-efficiency (Sabiazinho-4: 87.0% at \$0.13 vs Claude-Haiku-4.5: 73.5% at \$1.12). Multi-turn evaluation reveals significant variation in constraint persistence, with conversation-level accuracy ranging from 60% to 96% across models. We identify specific challenges in morphological constraints, exact counting, and constraint persistence degradation across turns. We release the complete
benchmark, evaluation code, and baseline results to facilitate research on instruction-following in Portuguese.
Read the full abstract ▾
All authors Giovana Kerche Bonás, Roseval Malaquias Junior, Marcos Piau, Thiago Laitz, Thales Sales Almeida, Hugo Abonizio, Celio Larcher, Ramon Pires, Rodrigo Nogueira
Computation and Language
Reddit After Roe: A Computational Analysis of Abortion Narratives and Barriers in the Wake of Dobbs
Aria Pessianzadeh, Alex H. Poole and Rezvaneh Rezapour
The 2022 U.S. Supreme Court decision in Dobbs v. Jackson Women's Health Organization reshaped the reproductive rights landscape, introducing new uncertainty and barriers to abortion access. We present a large-scale computational analysis of abortion discourse on Reddit, examining how barriers to access are articulated across information-seeking and information-sharing behaviors, different stages of abortion (before, during, after), and three phases of the Dobbs decision in 2022. Drawing on more than 17,000 posts from four abortion-related subreddits, we employed a multi-step pipeline to classify posts by information type, abortion stage, barrier category, and expressed
emotions. Using a codebook of eight barrier types, including legal, financial, emotional, and social obstacles, we analyzed their associations with
emotions and information behaviors. Topic modeling of model-generated barrier rationales further revealed how discourse evolved in response to shifting legal and cultural contexts. Our findings show that emotional and psychological barriers consistently dominate abortion narratives online, with
emotions such as nervousness, confusion, fear, and sadness prevalent across discourse. By linking information behaviors, barriers,
emotions, and temporal dynamics, this study provides a multi-dimensional account of how abortion is navigated in online communities.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.CV
Shoubin Yu, Lei Shu, Antoine Yang et al. (+6 more)
Multimodal AI
agents are increasingly automating complex real-world workflows that involve online web execution. However, current web-
agent benchmarks suffer from a critical limitation: they focus entirely on web-based interaction and perception, lacking grounding in the user's real-world physical surroundings. This limitation prevents evaluation in crucial scenarios, such as when an
agent must use
egocentric visual perception (e.g., via AR glasses) to recognize an object in the user's surroundings and then complete a related task online. To address this gap, we introduce Ego2Web, the first
benchmark designed to bridge
egocentric video perception and web
agent execution. Ego2Web pairs real-world first-person
video recordings with web tasks that require visual understanding, web task planning, and interaction in an online environment for successful completion. We utilize an automatic data-generation pipeline combined with human verification and refinement to curate well-constructed, high-quality
video-task pairs across diverse web task types, including e-commerce, media
retrieval, knowledge lookup, etc. To facilitate accurate and scalable evaluation for our
benchmark, we also develop a novel LLM-as-a-Judge automatic evaluation method, Ego2WebJudge, which achieves approximately 84% agreement with human judgment, substantially higher than existing evaluation methods. Experiments with diverse SoTA
agents on our Ego2Web show that their performance is weak, with substantial headroom across all task categories. We also conduct a comprehensive ablation study on task design, highlighting the necessity of accurate
video understanding in the proposed task and the limitations of current
agents. We hope Ego2Web can be a critical new resource for developing truly capable AI assistants that can seamlessly see, understand, and act across the physical and digital worlds.
Read the full abstract ▾
All authors Shoubin Yu, Lei Shu, Antoine Yang, Yao Fu, Srinivas Sunkara, Maria Wang, Jindong Chen, Mohit Bansal, Boqing Gong
Computation and Language · Cross-listed from cs.SE
Generating and Evaluating Sustainable Procurement Criteria for the Swiss Public Sector using In-Context Prompting with Large Language Models
Yingqiang Gao, Veton Matoshi, Luca Rolshoven et al. (+5 more)
Public procurement refers to the process by which public sector institutions, such as governments, municipalities, and publicly funded bodies, acquire goods and services. Swiss law requires the integration of ecological, social, and economic sustainability requirements into tender evaluations in the format of criteria that have to be fulfilled by a bidder. However, translating high-level sustainability regulations into concrete, verifiable, and sector-specific procurement criteria (such as selection criteria, award criteria, and technical specifications) remains a labor-intensive and error-prone manual task, requiring substantial domain expertise in several groups of goods and services and considerable manual effort. This paper presents a configurable, LLM-assisted pipeline that is presented as a software supporting the systematic generation and evaluation of sustainability-oriented procurement criteria catalogs for Switzerland. The system integrates in-context prompting, interchangeable LLM backends, and automated output validation to enable auditable criteria generation across different procurement sectors. As a proof of concept, we instantiate the pipeline using official sustainability guidelines published by the Swiss government and the European Commission, which are ingested as structured reference documents. We evaluate the system through a combination of automated quality checks, including an LLM-based evaluation component, and expert comparison against a manually curated gold standard. Our results demonstrate that the proposed pipeline can substantially reduce manual drafting effort while producing criteria catalogs that are consistent with official guidelines. We further discuss system limitations, failure modes, and design trade-offs observed during deployment, highlighting key considerations for integrating generative AI into public sector software workflows.
Read the full abstract ▾
All authors Yingqiang Gao, Veton Matoshi, Luca Rolshoven, Tilia Ellendorff, Judith Binder, Jeremy Austin Jann, Gerold Schneider, Matthias Stürmer
Computation and Language
Rashid: A Cipher-Based Framework for Exploring In-Context Language Learning
Niyati Bafna, Ryan Soh-Eun Shim, Barbara Plank et al. (+2 more)
Where there is growing interest in in-context
language learning (ICLL) for unseen
languages with large
language models, such
languages usually suffer from the lack of NLP tools, data resources, and researcher expertise. This means that progress is difficult to assess, the field does not allow for cheap large-scale experimentation, and findings on ICLL are often limited to very few
languages and tasks. In light of such limitations, we introduce a framework (Rashid), for studying ICLL wherein we reversibly cipher high-resource
languages (HRLs) to construct truly unseen
languages with access to a wide range of resources available for HRLs, unlocking previously impossible exploration of ICLL phenomena. We use our framework to assess current methods in the field with SOTA evaluation tools and manual analysis, explore the utility of potentially expensive resources in improving ICLL, and test ICLL strategies on rich downstream tasks beyond machine translation. These lines of exploration showcase the possibilities enabled by our framework, as well as providing actionable insights regarding current performance and future directions in ICLL.
Read the full abstract ▾
All authors Niyati Bafna, Ryan Soh-Eun Shim, Barbara Plank, David Yarowsky, Hale Sirin
Computation and Language
Component Ablation for Efficient Hybrid Language Model Architectures: Performance, Resilience, and Compression Implications
Hector Borobia, Elies Seguí-Mas and Guillermina Tormo-Carbó
Hybrid
language models combine softmax
attention with linear-time sequence mechanisms such as state-space or linear-
attention layers, but the functional contribution of each component type remains insufficiently characterized. We study component-level ablation in two sub-1B hybrid
language models, Qwen3.5-0.8B and Falcon-H1-0.5B, using likelihood-based evaluation, downstream
benchmarks, layer-wise interventions, random controls, and representation-level diagnostics. Across the tested models, removing either
attention or the alternative sequence-processing pathway substantially degrades performance, indicating that both component types contribute to model behavior. Likelihood metrics are especially sensitive to the linear-
attention or state-space pathway, while downstream
benchmark degradation depends on task and architecture. Layer-wise ablations show that component importance is position-dependent, with the strongest effects concentrated in early or mid-network components rather than uniformly across depth. Random-removal controls further show that hybrid architectures and same-family
Transformer baselines degrade differently under structural perturbation. These results suggest that component ablation is a useful diagnostic for understanding hybrid
language model architectures. The findings provide evidence relevant to efficient model design, compression, robustness analysis, and deployment decisions in architectures that combine
attention with alternative sequence-processing mechanisms.
Read the full abstract ▾
Computation and Language
LLM-guided headline rewriting for clickability enhancement without clickbait
Yehudit Aperstein, Linoy Halifa, Sagiv Bar et al. (+1 more)
Enhancing reader engagement while preserving informational fidelity is a central challenge in controllable text generation for news media. Optimizing news headlines for reader engagement is often conflated with clickbait, resulting in exaggerated or misleading phrasing that undermines editorial trust. We frame clickbait not as a separate stylistic category, but as an extreme outcome of disproportionate amplification of otherwise legitimate engagement cues. Based on this view, we formulate headline rewriting as a controllable generation problem, where specific engagement-oriented linguistic attributes are selectively strengthened under explicit constraints on semantic faithfulness and proportional emphasis. We present a guided headline rewriting framework built on a large
language model (LLM) that uses the Future Discriminators for Generation (FUDGE) paradigm for inference-time control. The LLM is steered by two auxiliary guide models: (1) a clickbait scoring model that provides negative guidance to suppress excessive stylistic amplification, and (2) an engagement-attribute model that provides positive guidance aligned with target clickability objectives. Both guides are trained on neutral headlines drawn from a curated real-world news corpus. At the same time, clickbait variants are generated synthetically by rewriting these original headlines using an LLM under controlled activation of predefined engagement tactics. By adjusting guidance weights at inference time, the system generates headlines along a continuum from neutral paraphrases to more engaging yet editorially acceptable formulations. The proposed framework provides a principled approach for studying the trade-off between attractiveness, semantic preservation, and clickbait avoidance, and supports responsible LLM-based headline optimization in journalistic settings.
Read the full abstract ▾
All authors Yehudit Aperstein, Linoy Halifa, Sagiv Bar, Alexander Apartsin
Computation and Language
XNote: Benchmarking Automated Community Notes Generation for Image-based Contextual Deception
Jin Ma, Jingwen Yan, Mohammed Aldeen et al. (+5 more)
Community Notes have emerged as an effective crowd-sourced mechanism for combating online deception on social media platforms. However, its reliance on human contributors limits both the timeliness and scalability. In this work, we study the automated Community Notes generation task for image-based contextual deception, where an authentic image is paired with misleading context (e.g., time, entity, and event). Unlike prior work that primarily focuses on deception detection (i.e., judging whether a post is true or false in a binary manner), automated Community Notes generation requires producing concise and grounded notes that help users recover the missing or corrected context. This problem remains underexplored due to the scarcity of
datasets that support this task. To address this gap, we curate a real-world
dataset, XNote, comprising X posts with associated Community Notes and external contexts, along with annotations of topics and deceptive factors. We further
benchmark a range of frontier large
vision language models (LVLMs) on XNote, evaluating their performance on both deception detection and note generation tasks. We also compare against an end-to-end approach, SNIFFER, and a commercial tool, GPT-5. Our results highlight the challenges in automated Community Notes generation, underscoring the need for improved methods and metrics tailored for this task.
Read the full abstract ▾
All authors Jin Ma, Jingwen Yan, Mohammed Aldeen, Ethan Anderson, Taran Kavuru, Jinkyung Katie Park, Feng Luo, Long Cheng
Computation and Language
Sparse but Critical: A Token-Level Analysis of Distributional Shifts in RLVR Fine-Tuning of LLMs
Haoming Meng, Kexin Huang, Shaohang Wei et al. (+6 more)
Reinforcement learning with verifiable rewards (RLVR) has significantly improved
reasoning in large
language models (LLMs), yet the
token-level mechanisms underlying these improvements remain unclear. We present a systematic empirical study of RLVR's distributional effects organized around three main analyses: (1)
token-level characterization of distributional shifts between base and RL models, (2) the impact of
token-level distributional shifts on sequence-level
reasoning performance through cross-sampling interventions, and (3) fine-grained mechanics of these shifts at the
token level. We find that RL fine-tuning induces highly sparse and targeted changes, with only a small fraction of
token distributions exhibiting meaningful divergence between the base and RL policies. We further characterize the structure and
evolution of these shifts through analyses of
token entropy, positional concentration, and reallocation of probability mass. To assess the functional importance of these sparse changes, we conduct cross-sampling experiments that selectively swap
token choices between the base and RL models with varying intervention budgets. We show that inserting only a small fraction of RL-sampled
tokens into base generations progressively recovers RL performance gains, while injecting a similarly small number of base
token choices into otherwise RL-generated sequences collapses performance to base levels, isolating a small set of
token-level decisions directly responsible for RLVR's performance gains. Finally, we explore divergence-weighted variants of the advantage signal as a diagnostic intervention, finding that they can yield improvements over baselines. Together, our results shed light on the distributional changes induced by RLVR and provide a fine-grained,
token-level lens for understanding RLVR fine-tuning as a targeted refinement process.
Read the full abstract ▾
All authors Haoming Meng, Kexin Huang, Shaohang Wei, Chiyu Ma, Shuo Yang, Xue Wang, Guoyin Wang, Bolin Ding, Jingren Zhou
Computation and Language · Cross-listed from cs.AI
From Static Templates to Dynamic Runtime Graphs: A Survey of Workflow Optimization for LLM Agents
Ling Yue, Kushal Raj Bhandari, Ching-Yun Ko et al. (+6 more)
Large
language model (LLM)-based systems are becoming increasingly popular for solving tasks by constructing executable workflows that interleave LLM calls, information
retrieval, tool use, code execution,
memory updates, and verification. This survey reviews recent methods for designing and optimizing such workflows, which we treat as agentic computation graphs (ACGs). We organize the literature based on when workflow structure is determined, where structure refers to which components or
agents are present, how they depend on each other, and how information flows between them. This lens distinguishes static methods, which fix a reusable workflow scaffold before deployment, from dynamic methods, which select, generate, or revise the workflow for a particular run before or during execution. We further organize prior work along three dimensions: when structure is determined, what part of the workflow is optimized, and which evaluation signals guide optimization (e.g., task metrics, verifier signals, preferences, or trace-derived feedback). We also distinguish reusable workflow templates, run-specific realized graphs, and execution traces, separating reusable design choices from the structures actually deployed in a given run and from realized runtime behavior. Finally, we outline a structure-aware evaluation perspective that complements downstream task metrics with graph-level properties, execution cost, robustness, and structural variation across inputs. Our goal is to provide a clear vocabulary, a unified framework for positioning new methods, a more comparable view of existing body of literature, and a more reproducible evaluation standard for future work in workflow optimizations for LLM
agents.
Read the full abstract ▾
All authors Ling Yue, Kushal Raj Bhandari, Ching-Yun Ko, Dhaval Patel, Shuxin Lin, Nianjun Zhou, Jianxi Gao, Pin-Yu Chen, Shaowu Pan