Computation and Language
asr_eval: Algorithms and tools for multi-reference and streaming speech recognition evaluation
Oleg Sedukhin and Andrey Kostin
We propose several improvements to the
speech recognition evaluation. First, we propose a string
alignment algorithm that supports both multi-reference labeling, arbitrary-length insertions and better word
alignment. This is especially useful for non-Latin
languages, those with rich word formation, to label cluttered or longform
speech. Secondly, we collect a novel test set DiverseSpeech-Ru of longform in-the-wild Russian
speech with careful multi-reference labeling. We also perform multi-reference relabeling of popular Russian tests set and study fine-tuning dynamics on its corresponding train set. We demonstrate that the model often adopts to
dataset-specific labeling, causing an illusion of metric improvement. Based on the improved word
alignment, we develop tools to evaluate streaming
speech recognition and to align multiple transcriptions to compare them visually. Additionally, we provide uniform wrappers for many offline and streaming
speech recognition models. Our code will be made publicly available.
Read the full abstract ▾
Computation and Language
DeepSearchQA: Bridging the Comprehensiveness Gap for Deep Research Agents
Nikita Gupta, Riju Chatterjee, Lukas Haas et al. (+9 more)
We introduce DeepSearchQA, a 900-
prompt benchmark for evaluating
agents on difficult multi-step information-seeking tasks across 17 different fields. Unlike traditional
benchmarks that target single answer
retrieval or broad-spectrum factuality, DeepSearchQA features a
dataset of challenging, handcrafted tasks designed to evaluate an
agent's ability to execute complex search plans to generate exhaustive answer lists. This shift in design explicitly tests three critical, yet under-evaluated capabilities: 1) systematic collation of fragmented information from disparate sources, 2) de-duplication and entity resolution to ensure precision, and 3) the ability to reason about stopping criteria within an open-ended search space. Each task is structured as a causal chain, where discovering information for one step is dependent on the successful completion of the previous one, stressing long-horizon planning and context retention. All tasks are grounded in the open web with objectively verifiable answer sets. Our comprehensive evaluation of state-of-the-art
agent architectures reveals significant performance limitations: even the most advanced models struggle to balance high recall with precision. We observe distinct failure modes ranging from premature stopping (under-
retrieval) to hedging behaviors, where
agents cast an overly wide net of low-confidence answers to artificially boost recall. These findings highlight critical headroom in current
agent designs and position DeepSearchQA as an essential diagnostic tool for driving future research toward more robust, deep-research capabilities.
Read the full abstract ▾
All authors Nikita Gupta, Riju Chatterjee, Lukas Haas, Connie Tao, Andrew Wang, Chang Liu, Hidekazu Oiwa, Elena Gribovskaya, Jan Ackermann, John Blitzer, Sasha Goldshtein, Dipanjan Das
Computation and Language · Cross-listed from cs.SD
Text-only adaptation in LLM-based ASR through text denoising
Andrés Carofilis, Sergio Burdisso, Esaú Villatoro-Tello et al. (+8 more)
Adapting large
language model (LLM)-based automatic
speech recognition (ASR) systems to new domains using text-only data is a significant yet underexplored challenge. Standard fine-tuning of the LLM on the target domain text often disrupts the critical
alignment between the
speech and text modality learned by the projector, degrading performance. We introduce a novel text-only adaptation method that frames this process as a text denoising task. Our approach trains the LLM to recover clean transcripts from noisy inputs. This process effectively adapts the model to a target domain while preserving cross-modal
alignment. Our solution is lightweight, requiring no architectural changes or additional parameters. Extensive evaluation on two
datasets demonstrates up to 22.1% relative improvement, outperforming recent state-of-the-art text-only adaptation methods.
Read the full abstract ▾
All authors Andrés Carofilis, Sergio Burdisso, Esaú Villatoro-Tello, Shashi Kumar, Kadri Hacioglu, Srikanth Madikeri, Pradeep Rangappa, Manjunath K E, Petr Motlicek, Shankar Venkatesan, Andreas Stolcke
Computation and Language · Cross-listed from eess.AS
Reducing Prompt Sensitivity in LLM-based Speech Recognition Through Learnable Projection
Sergio Burdisso, Esaú Villatoro-Tello, Shashi Kumar et al. (+7 more)
LLM-based automatic
speech recognition (ASR), a well-established approach, connects
speech foundation models to large
language models (LLMs) through a
speech-to-LLM projector, yielding promising results. A common design choice in these architectures is the use of a fixed, manually defined
prompt during both training and inference. This setup not only enables applicability across a range of practical scenarios, but also helps maximize model performance. However, the impact of
prompt design remains underexplored. This paper presents a comprehensive analysis of commonly used
prompts across diverse
datasets, showing that
prompt choice significantly affects ASR performance and introduces instability, with no single
prompt performing best across all cases. Inspired by the
speech-to-LLM projector, we propose a
prompt projector module, a simple, model-agnostic extension that learns to project
prompt embeddings to more effective regions of the LLM input space, without modifying the underlying LLM-based ASR model. Experiments on four
datasets show that the addition of a
prompt projector consistently improves performance, reduces variability, and outperforms the best manually selected
prompts.
Read the full abstract ▾
All authors Sergio Burdisso, Esaú Villatoro-Tello, Shashi Kumar, Srikanth Madikeri, Andrés Carofilis, Pradeep Rangappa, Manjunath K E, Kadri Hacioglu, Petr Motlicek, Andreas Stolcke
Computation and Language · Cross-listed from cs.SD
SW-ASR: A Context-Aware Hybrid ASR Pipeline for Robust Single Word Speech Recognition
Manali Sharma, Riya Naik and Buvaneshwari G
Single-word Automatic
Speech Recognition (ASR) is a challenging task due to the lack of linguistic context and sensitivity to noise, pronunciation variation, and channel artifacts, especially in low-resource, communication-critical domains such as healthcare and emergency response. This paper reviews recent deep learning approaches and proposes a modular framework for robust single-word detection. The system combines denoising and normalization with a hybrid ASR front end (Whisper + Vosk) and a verification layer designed to handle out-of-vocabulary words and degraded audio. The verification layer supports multiple matching strategies, including
embedding similarity, edit distance, and LLM-based matching with optional contextual guidance. We evaluate the framework on the Google
Speech Commands
dataset and a curated real-world
dataset collected from telephony and messaging platforms under bandwidth-limited conditions. Results show that while the hybrid ASR front end performs well on clean audio, the verification layer significantly improves accuracy on noisy and compressed channels. Context-guided and LLM-based matching yield the largest gains, demonstrating that lightweight verification and context mechanisms can substantially improve single-word ASR robustness without sacrificing latency required for real-time telephony applications.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.LG
Evolutionary Strategies lead to Catastrophic Forgetting in LLMs
Immanuel Abdi, Akshat Gupta, Micah Mok et al. (+3 more)
One of the biggest missing capabilities in current AI systems is the ability to learn continuously after deployment. Implementing such continually learning systems have several challenges, one of which is the large
memory requirement of gradient-based
algorithms that are used to train state-of-the-art LLMs. Evolutionary Strategies (ES) have recently re-emerged as a gradient-free alternative to traditional learning
algorithms and have shown encouraging performance on specific tasks in LLMs. In this paper, we perform a comprehensive analysis of ES and specifically evaluate its forgetting curves when training for an increasing number of update steps. We first find that ES is able to reach performance numbers close to GRPO for math and
reasoning tasks with a comparable compute budget. However, and most importantly for continual learning, the performance gains in ES is accompanied by significant forgetting of prior abilities, limiting its applicability for training models online. We also explore the reason behind this behavior and show that the updates made using ES are much less sparse and have orders of magnitude larger $\ell_2$ norm compared to corresponding GRPO updates, explaining the contrasting forgetting curves between the two
algorithms. With this study, we aim to highlight the issue of forgetting in gradient-free
algorithms like ES and hope to inspire future work to mitigate these issues.
Read the full abstract ▾
All authors Immanuel Abdi, Akshat Gupta, Micah Mok, Alexander Lu, Nicholas Lee, Gopala Anumanchipalli
Computation and Language
When Flores Bloomz Wrong: Cross-Direction Contamination in Machine Translation Evaluation
David Tan, Pinzhen Chen, Josef van Genabith et al. (+1 more)
Large
language models (LLMs) can be
benchmark-contaminated, resulting in inflated scores that mask memorization as generalization, and in
multilingual settings, this memorization can even transfer to "uncontaminated"
languages. Using the FLORES-200 translation
benchmark as a diagnostic, we study two 7-8B instruction-tuned
multilingual LLMs: Bloomz, which was trained on FLORES, and Llama as an uncontaminated control. We confirm Bloomz's FLORES contamination and demonstrate that machine translation contamination can be cross-directional, artificially boosting performance in unseen translation directions due to target-side memorization. Further analysis shows that recall of memorized references often persists despite various source-side perturbation efforts like paraphrasing and named entity replacement. However, replacing named entities leads to a consistent decrease in BLEU, suggesting an effective probing method for memorization in contaminated models.
Read the full abstract ▾
All authors David Tan, Pinzhen Chen, Josef van Genabith, Koel Dutta Chowdhury
Computation and Language · Cross-listed from cs.LG
Reward Models Inherit Value Biases from Pretraining
Brian Christian, Jessica A. F. Thompson, Elle Michelle Yang et al. (+4 more)
Reward models (RMs) are central to aligning large
language models (LLMs) with human values but have received less
attention than pretrained and post-trained LLMs themselves. Because RMs are initialized from LLMs, they inherit representations that shape their behavior, but the nature and extent of this influence remain understudied. In a comprehensive study of 10 leading open-weight RMs using validated psycholinguistic corpora, we show that RMs exhibit significant differences along multiple dimensions of human value as a function of their base model. Using the "Big Two" psychological axes, we show a robust preference of Llama RMs for "agency" and a corresponding robust preference of Gemma RMs for "communion." This phenomenon holds even when the preference data and finetuning process are identical, and we trace it back to the logits of the respective instruction-tuned and pretrained models. These log-probability differences themselves can be formulated as an implicit RM; we derive usable implicit reward scores and show that they exhibit the very same agency/communion difference. We run experiments training RMs with ablations for preference data source and quantity, which demonstrate that this effect is not only repeatable but surprisingly durable. Despite RMs being designed to represent human preferences, our evidence shows that their outputs are influenced by the pretrained LLMs on which they are based. This work underscores the importance of safety and
alignment efforts at the pretraining stage, and makes clear that open-source developers' choice of base model is as much a consideration of values as of performance.
Read the full abstract ▾
All authors Brian Christian, Jessica A. F. Thompson, Elle Michelle Yang, Vincent Adam, Hannah Rose Kirk, Christopher Summerfield, Tsvetomira Dumbalska
Computation and Language
Linear representations in language models can change dramatically over a conversation
Andrew Kyle Lampinen, Yuxuan Li, Eghbal Hosseini et al. (+2 more)
Language model representations often contain linear directions that correspond to high-level concepts. Here, we study the dynamics of these representations: how representations evolve along these dimensions within the context of (simulated) conversations. We find that linear representations can change dramatically over a conversation; for example, information that is represented as factual at the beginning of a conversation can be represented as non-factual at the end and vice versa. These changes are content-dependent; while representations of conversation-relevant information may change, generic information is generally preserved. These changes are robust even for dimensions that disentangle factuality from more superficial response patterns, and occur across different model families and layers of the model. These representation changes do not require on-policy conversations; even replaying a conversation script written by an entirely different model can produce similar changes. However, adaptation is much weaker from simply having a sci-fi story in context that is framed more explicitly as such. We also show that steering along a representational direction can have dramatically different effects at different points in a conversation. These results are consistent with the idea that representations may evolve in response to the model playing a particular role that is cued by a conversation. Our findings may pose challenges for interpretability and steering – in particular, they imply that it may be misleading to use static interpretations of features or directions, or probes that assume a particular range of features consistently corresponds to a particular ground-truth value. However, these types of representational dynamics also point to exciting new research directions for understanding how models adapt to context.
Read the full abstract ▾
All authors Andrew Kyle Lampinen, Yuxuan Li, Eghbal Hosseini, Sangnie Bhardwaj, Murray Shanahan
Computation and Language · Cross-listed from cs.LG
Training Reasoning Models on Saturated Problems via Failure-Prefix Conditioning
Minwu Kim, Safal Shrestha, Anubhav Shrestha et al. (+1 more)
As Reinforcement Learning with Verifiable Rewards (RLVR) substantially improves the
reasoning abilities of large
language models (LLMs), a new bottleneck emerges: more training problems become saturated, that is, the LLM answers the questions correctly for nearly every rollout. On such problems, rewards provide little useful learning signal. While collecting harder problems is a natural response, it is costly and increasingly difficult. We propose failure-prefix conditioning, a simple method that unlocks the remaining signal in saturated problems by shifting exploration toward failure-prone
reasoning states. By conditioning on prefixes of rare incorrect trajectories, the method improves the model's ability to recover from misleading early
reasoning. We observe that failure-prefix conditioning consistently improves performance where standard RLVR stalls, and achieves gains comparable to training on newly collected medium-difficulty problems. We further analyze the model's robustness, finding that our method reduces performance degradation under misleading failure prefixes, albeit with a mild trade-off in adherence to correct early
reasoning. Finally, we demonstrate that an iterative approach, which refreshes failure prefixes during training, unlocks additional gains after performance plateaus. Overall, our results show that saturated problems still contain valuable learning signal, and that failure-prefix conditioning provides an effective way to unlock it.
Read the full abstract ▾
All authors Minwu Kim, Safal Shrestha, Anubhav Shrestha, Keith Ross
Computation and Language
Structured Semantic Information Helps Retrieve Better Examples for In-Context Learning Applied to Few-Shot Relation Extraction
Aunabil Chakma, Mihai Surdeanu and Eduardo Blanco
This paper presents several strategies to automatically obtain additional examples for in-context learning, effectively transforming relation extraction from a 1-shot to a few-shot setting. Specifically, we introduce a novel strategy for example selection, in which new examples are selected based on the similarity of their underlying syntactic-semantic structure to the provided 1-shot example. We show that our strategy results in complementary word choices and sentence structures compared to LLM-generated examples. When both strategies are combined, the resulting hybrid system achieves a more holistic picture of the relations of interest than either method alone. Our framework transfers well across
datasets (FS-TACRED and FS-FewRel) and LLM families (Qwen and Gemma). Overall, our hybrid system consistently outperforms alternative strategies achieving state-of-the-art performance on FS-TACRED and strong gains on a customized FewRel subset.
Read the full abstract ▾
Computation and Language
Dissecting Multimodal In-Context Learning: Modality Asymmetries and Circuit Dynamics in modern Transformers
Yiran Huang, Karsten Roth, Quentin Bouniot et al. (+2 more)
Transformer-based
multimodal large
language models often exhibit in-context learning (ICL) abilities. Motivated by this phenomenon, we ask: how do
transformers learn to associate information across modalities from in-context examples? We investigate this question through controlled experiments on small
transformers trained on synthetic classification tasks, enabling precise manipulation of data statistics and model architecture. We begin by revisiting core principles of unimodal ICL in modern
transformers. While several prior findings replicate, we find that Rotary Position
Embeddings (RoPE) increases the data complexity threshold for ICL. Extending to the
multimodal setting reveals a fundamental learning asymmetry: when pretrained on high-diversity data from a primary modality, surprisingly low data complexity in the secondary modality suffices for
multimodal ICL to emerge. Mechanistic analysis shows that both settings rely on an induction-style mechanism that copies labels from matching in-context exemplars;
multimodal training refines and extends these
circuits across modalities. Our findings provide a mechanistic foundation for understanding
multimodal ICL in modern
transformers and introduce a controlled testbed for future investigation. Code is available at: https://github.com/YiranHuangIrene/
multimodal-icl
Read the full abstract ▾
All authors Yiran Huang, Karsten Roth, Quentin Bouniot, Wenjia Xu, Zeynep Akata
Computation and Language · Cross-listed from cs.CY
Jurisdiction as Structural Barrier: How Privacy Policy Organization May Reduce Visibility of Substantive Disclosures
Thomas Brackin
Privacy policies are supposed to provide notice. But what if substantive information appears only where users skip it? We identify a structural pattern we call jurisdiction-siloed disclosure: information about data practices appearing in specific, actionable form only within regional compliance sections labeled "California Residents" or "EU/UK Users," while general sections use vague or qualified
language for the same practices. Our audit of 123 major companies identifies 282 potential instances across 77 companies (62.6% of this purposive sample). A conservative estimate restricted to practice categories validated against OPP-115 human annotations finds 138 instances across 54 companies (44%); post-2018 categories central to our findings await independent validation. If users skip jurisdiction-labeled sections as information foraging theory predicts, users outside regulated jurisdictions would receive less specific information about practices affecting them–a transparency failure operating through document architecture rather than omission. We propose universal substantive disclosure: practices affecting all users should appear in the main policy body, with regional sections containing only procedural rights information. This standard finds support in analogous disclosure regimes (securities, truth-in-lending, nutritional labeling) where material information must reach all affected parties. Regulators could operationalize this through the FTC's "clear and conspicuous" standard and GDPR transparency principles. This work is hypothesis-generating: we establish that the structural pattern exists and ground the transparency concern in behavioral theory, but direct measurement of jurisdiction-specific section skipping remains the critical validation priority. We release our methodology and annotated
dataset to enable replication.
Read the full abstract ▾
Computation and Language
SERA: Soft-Verified Efficient Repository Agents
Ethan Shen, Daniel Tormoen, Saurabh Shah et al. (+2 more)
Open-weight coding
agents should hold a fundamental advantage over closed-source systems because they can specialize to private codebases, encoding repository-specific information directly in their weights. Yet the cost and complexity of training has kept this advantage theoretical until now. We present Soft-Verified Efficient Repository
Agents (SERA), an efficient method for training coding
agents that enables the rapid and cheap creation of
agents specialized to private codebases. Using Soft Verified Generation (SVG), we generate thousands of trajectories from any code repository, without requiring unit tests. Beyond repository specialization, we apply SVG to a larger corpus of codebases, generating 200,000+ synthetic trajectories. Using only supervised finetuning (SFT), SERA achieves leading results among fully open-source (open data, method, code) models while matching the performance of open-weight models like Devstral-Small-2. Creating SERA models is 26x cheaper than reinforcement learning and 57x cheaper than previous synthetic data methods to reach equivalent performance. We use our
dataset to provide detailed analysis of scaling laws, ablations, and confounding factors for training coding
agents. Overall, we believe our work will greatly accelerate research on open coding
agents and showcase the advantage of open-source models that can adapt to private codebases. We release SERA as the first model in Ai2's Open Coding
Agents series, along with all our code, data, and Claude Code integration to support the research community.
Read the full abstract ▾
All authors Ethan Shen, Daniel Tormoen, Saurabh Shah, Ali Farhadi, Tim Dettmers
Computation and Language
Persona Prompting as a Lens on LLM Social Reasoning
Jing Yang, Moritz Hechtbauer, Elisabeth Khalilov et al. (+3 more)
For socially sensitive tasks like hate
speech detection, the quality of explanations from Large
Language Models (LLMs) is crucial for factors like user trust and model
alignment. While Persona prompting (PP) is increasingly used as a way to steer model towards user-specific generation, its effect on model rationales remains underexplored. We investigate how LLM-generated rationales vary when conditioned on different simulated demographic personas. Using
datasets annotated with word-level rationales, we measure agreement with human annotations from different demographic groups, and assess the impact of PP on model bias and human
alignment. Our evaluation across three LLMs results reveals three key findings: (1) PP improving classification on the most subjective task (hate
speech) but degrading rationale quality. (2) Simulated personas fail to align with their real-world demographic counterparts, and high inter-persona agreement shows models are resistant to significant steering. (3) Models exhibit consistent demographic biases and a strong tendency to over-flag content as harmful, regardless of PP. Our findings reveal a critical trade-off: while PP can improve classification in socially-sensitive tasks, it often comes at the cost of rationale quality and fails to mitigate underlying biases, urging caution in its application.
Read the full abstract ▾
All authors Jing Yang, Moritz Hechtbauer, Elisabeth Khalilov, Evelyn Luise Brinkmann, Vera Schmitt, Nils Feldhus
Computation and Language
Like a Therapist, But Not: Reddit Narratives of AI in Mental Health Contexts
Elham Aghakhani and Rezvaneh Rezapour
Large
language models (LLMs) are increasingly used for emotional support and mental health-related interactions outside clinical settings, yet little is known about how people evaluate and relate to these systems in everyday use. We analyze 5,126 Reddit posts from 47 mental health communities describing experiential or exploratory use of AI for emotional support or therapy. Grounded in the Technology Acceptance Model and therapeutic alliance theory, we develop a theory-informed annotation framework and apply a hybrid LLM-human pipeline to analyze evaluative
language, adoption-related attitudes, and relational
alignment at scale. Our results show that engagement is shaped primarily by narrated outcomes, trust, and response quality, rather than emotional bond alone. Positive sentiment is most strongly associated with task and goal
alignment, while companionship-oriented use more often involves misaligned alliances and reported risks such as dependence and symptom escalation. Overall, this work demonstrates how theory-grounded constructs can be operationalized in large-scale discourse analysis and highlights the importance of studying how users interpret
language technologies in sensitive, real-world contexts.
Read the full abstract ▾
Computation and Language
QueerGen: How LLMs Reflect Societal Norms on Gender and Sexuality in Sentence Completion Tasks
Mae Sosto, Delfina Sol Martinez Pandiani and Laura Hollink
This paper examines how Large
Language Models (LLMs) reproduce societal norms, particularly heterocisnormativity, and how these norms translate into measurable biases in their text generations. We investigate whether explicit information about a subject's gender or sexuality influences LLM responses across three subject categories: queer-marked, non-queer-marked, and the normalized "unmarked" category. Representational imbalances are operationalized as measurable differences in English sentence completions across four dimensions: sentiment, regard, toxicity, and prediction diversity. Our findings show that Masked
Language Models (MLMs) produce the least favorable sentiment, higher toxicity, and more negative regard for queer-marked subjects. Autoregressive
Language Models (ARLMs) partially mitigate these patterns, while closed-access ARLMs tend to produce more harmful outputs for unmarked subjects. Results suggest that LLMs reproduce normative social assumptions, though the form and degree of bias depend strongly on specific model characteristics, which may redistribute, but not eliminate, representational harms.
Read the full abstract ▾
Computation and Language
AgentLongBench: A Controllable Long Benchmark For Long-Contexts Agents via Environment Rollouts
Shicheng Fang, Yuxin Wang, Xiaoran Liu et al. (+6 more)
The
evolution of Large
Language Models (LLMs) into autonomous
agents necessitates the management of extensive, dynamic contexts. Current
benchmarks, however, remain largely static, relying on passive
retrieval tasks that fail to simulate the complexities of
agent-environment interaction, such as non-linear
reasoning and iterative feedback. To address this, we introduce AgentLongBench, which evaluates
agents through simulated environment rollouts based on Lateral Thinking Puzzles. This framework generates rigorous interaction trajectories across knowledge-intensive and knowledge-free scenarios. Experiments with state-of-the-art models and
memory systems (32K to 4M
tokens) expose a critical weakness: while adept at static
retrieval,
agents struggle with the dynamic information synthesis essential for workflows. Our analysis indicates that this degradation is driven by the minimum number of
tokens required to resolve a query. This factor explains why the high information density inherent in massive tool responses poses a significantly greater challenge than the
memory fragmentation typical of long-turn dialogues.
Read the full abstract ▾
All authors Shicheng Fang, Yuxin Wang, Xiaoran Liu, Jiahao Lu, Chuanyuan Tan, Xinchi Chen, Yining Zheng, Xuanjing Huang, Xipeng Qiu
Computation and Language · Cross-listed from cs.HC
Polite But Boring? Trade-offs Between Engagement and Psychological Reactance to Chatbot Feedback Styles
Samuel Rhys Cox, Joel Wester and Niels van Berkel
As conversational
agents become increasingly common in behaviour change interventions, understanding optimal feedback delivery mechanisms becomes increasingly important. However, choosing a style that both lessens psychological reactance (perceived threats to freedom) while simultaneously eliciting feelings of surprise and engagement represents a complex design problem. We explored how three different feedback styles: 'Direct', 'Politeness', and 'Verbal Leakage' (slips or disfluencies to reveal a desired behaviour) affect user perceptions and behavioural intentions. Matching expectations from literature, the 'Direct' chatbot led to lower behavioural intentions and higher reactance, while the 'Politeness' chatbot evoked higher behavioural intentions and lower reactance. However, 'Politeness' was also seen as unsurprising and unengaging by participants. In contrast, 'Verbal Leakage' evoked reactance, yet also elicited higher feelings of surprise, engagement, and humour. These findings highlight that effective feedback requires navigating trade-offs between user reactance and engagement, with novel approaches such as 'Verbal Leakage' offering promising alternative design opportunities.
Read the full abstract ▾
Computation and Language
Online Density-Based Clustering for Real-Time Narrative Evolution Monitorin
Ostap Vykhopen, Viktoria Skorik, Maksym Tereshchenko et al. (+1 more)
Automated narrative intelligence systems for social media monitoring face significant scalability challenges when relying on batch clustering methods to process continuous data streams. We investigate replacing offline HDBSCAN with online density-based clustering
algorithms in a production narrative report generation pipeline that processes large volumes of
multilingual social media data. While HDBSCAN effectively discovers hierarchical clusters and handles noise, its batch-only nature requires full retraining for each time window, limiting scalability and real-time adaptability. We evaluate online clustering methods with respect to cluster quality, computational efficiency,
memory footprint, and integration with downstream narrative extraction. Our evaluation combines standard clustering metrics, narrative-specific measures, and human validation of cluster correctness to assess both structural quality and semantic interpretability. Experiments using sliding-window simulations on historical data from the Ukrainian information space reveal trade-offs between temporal stability and narrative coherence, with DenStream achieving the strongest overall performance. These findings bridge the gap between batch-oriented clustering approaches and the streaming requirements of large-scale narrative monitoring systems.
Read the full abstract ▾
All authors Ostap Vykhopen, Viktoria Skorik, Maksym Tereshchenko, Veronika Solopova
Computation and Language
ShieldedCode: Learning Robust Representations for Virtual Machine Protected Code
Mingqiao Mo, Yunlong Tan, Hao Zhang et al. (+2 more)
Large
language models (LLMs) have achieved remarkable progress in code generation, yet their potential for software protection remains largely untapped. Reverse engineering continues to threaten software security, while traditional virtual machine protection (VMP) relies on rigid, rule-based transformations that are costly to design and vulnerable to automated analysis. In this work, we present the first protection-aware framework that learns robust representations of VMP-protected code. Our approach builds large-scale paired
datasets of source code and normalized VM implementations, and introduces hierarchical dependency modeling at intra-, preceding-, and inter-instruction levels. We jointly optimize
language modeling with functionality-aware and protection-aware contrastive objectives to capture both semantic equivalence and protection strength. To further assess resilience, we propose a protection effectiveness optimization task that quantifies and ranks different VM variants derived from the same source. Coupled with a two-stage continual pre-training and fine-tuning pipeline, our method enables models to generate, compare, and reason over protected code. Extensive experiments show that our framework significantly improves robustness across diverse protection levels, opening a new research direction for learning-based software defense. In this work, we present ShieldedCode, the first protection-aware framework that learns robust representations of VMP-protected code. Our method achieves 26.95% Pass@1 on L0 VM code generation compared to 22.58% for GPT-4o., and improves binary similarity detection Recall@1 by 10% over state of art methods like jTrans.
Read the full abstract ▾
All authors Mingqiao Mo, Yunlong Tan, Hao Zhang, Heng Zhang, Yangfan He
Computation and Language
Efficient Multimodal Planning Agent for Visual Question-Answering
Zhuo Chen, Xinyu Geng, Xinyu Wang et al. (+4 more)
Visual Question-Answering (VQA) is a challenging
multimodal task that requires integrating visual and textual information to generate accurate responses. While
multimodal Retrieval-Augmented Generation (mRAG) has shown promise in enhancing VQA systems by providing more evidence on both image and text sides, the default procedure that addresses VQA queries, especially the knowledge-intensive ones, often relies on multi-stage pipelines of mRAG with inherent dependencies. To mitigate the inefficiency limitations while maintaining VQA task performance, this paper proposes a method that trains a
multimodal planning
agent, dynamically decomposing the mRAG pipeline to solve the VQA task. Our method optimizes the trade-off between efficiency and effectiveness by training the
agent to intelligently determine the necessity of each mRAG step. In our experiments, the
agent can help reduce redundant computations, cutting search time by over 60% compared to existing methods and decreasing costly tool calls. Meanwhile, experiments demonstrate that our method outperforms all baselines, including a Deep Research
agent and a carefully designed
prompt-based method, on average over six various
datasets. Code will be released.
Read the full abstract ▾
All authors Zhuo Chen, Xinyu Geng, Xinyu Wang, Yong Jiang, Zhen Zhang, Pengjun Xie, Kewei Tu
Computation and Language
Harnessing Large Language Models for Precision Querying and Retrieval-Augmented Knowledge Extraction in Clinical Data Science
Juan Jose Rubio Jan, Jack Wu and Julia Ive
This study applies Large
Language Models (LLMs) to two foundational Electronic Health Record (EHR) data science tasks: structured data querying (using programmatic
languages, Python/Pandas) and information extraction from unstructured clinical text via a
Retrieval Augmented Generation (RAG) pipeline. We test the ability of LLMs to interact accurately with large structured
datasets for analytics and the reliability of LLMs in extracting semantically correct information from free text health records when supported by RAG. To this end, we presented a flexible evaluation framework that automatically generates synthetic question and answer pairs tailored to the characteristics of each
dataset or task. Experiments were conducted on a curated subset of MIMIC III, (four structured tables and one clinical note type), using a mix of locally hosted and API-based LLMs. Evaluation combined exact-match metrics, semantic similarity, and human judgment. Our findings demonstrate the potential of LLMs to support precise querying and accurate information extraction in clinical workflows.
Read the full abstract ▾
Computation and Language
A Dialectic Pipeline for Improving LLM Robustness
Sara Candussio
Assessing ways in which
Language Models can reduce their
hallucinations and improve the outputs' quality is crucial to ensure their large-scale use. However, methods such as fine-tuning on domain-specific data or the training of a separate ad hoc verifier require demanding computational resources (not feasible for many user applications) and constrain the models to specific fields of knowledge. In this thesis, we propose a dialectic pipeline that preserves LLMs' generalization abilities while improving the quality of its answer via self-dialogue, enabling it to reflect upon and correct tentative wrong answers. We experimented with different pipeline settings, testing our proposed method on different
datasets and on different families of models. All the pipeline stages are enriched with the relevant context (in an oracle-RAG setting) and a study on the impact of its summarization or its filtering is conducted. We find that our proposed dialectic pipeline is able to outperform by significative margins the standard model answers and that it consistently achieves higher performances than Chain-of-Thought only prompting.
Read the full abstract ▾
Computation and Language
P2S: Probabilistic Process Supervision for General-Domain Reasoning Question Answering
Wenlin Zhong, Chengyuan Liu, Yiquan Wu et al. (+5 more)
While reinforcement learning with verifiable rewards (RLVR) has advanced LLM
reasoning in structured domains like mathematics and programming, its application to general-domain
reasoning tasks remains challenging due to the absence of verifiable reward signals. To this end, methods like Reinforcement Learning with Reference Probability Reward (RLPR) have emerged, leveraging the probability of generating the final answer as a reward signal. However, these outcome-focused approaches neglect crucial step-by-step supervision of the
reasoning process itself. To address this gap, we introduce Probabilistic Process Supervision (P2S), a novel self-supervision framework that provides fine-grained process rewards without requiring a separate reward model or human-annotated
reasoning steps. During reinforcement learning, P2S synthesizes and filters a high-quality reference
reasoning chain (gold-CoT). The core of our method is to calculate a Path Faithfulness Reward (PFR) for each
reasoning step, which is derived from the conditional probability of generating the gold-CoT's suffix, given the model's current
reasoning prefix. Crucially, this PFR can be flexibly integrated with any outcome-based reward, directly tackling the reward sparsity problem by providing dense guidance. Extensive experiments on reading comprehension and medical Question Answering
benchmarks show that P2S significantly outperforms strong baselines.
Read the full abstract ▾
All authors Wenlin Zhong, Chengyuan Liu, Yiquan Wu, Bovin Tan, Changlong Sun, Yi Wang, Xiaozhong Liu, Kun Kuang
Computation and Language · Cross-listed from cs.CV
GDCNet: Generative Discrepancy Comparison Network for Multimodal Sarcasm Detection
Shuguang Zhang, Junhong Lian, Guoxin Yu et al. (+2 more)
Multimodal sarcasm detection (MSD) aims to identify sarcasm within image-text pairs by modeling semantic incongruities across modalities. Existing methods often exploit cross-modal
embedding misalignment to detect inconsistency but struggle when visual and textual content are loosely related or semantically indirect. While recent approaches leverage large
language models (LLMs) to generate sarcastic cues, the inherent diversity and subjectivity of these generations often introduce noise. To address these limitations, we propose the Generative Discrepancy Comparison Network (GDCNet). This framework captures cross-modal conflicts by utilizing descriptive, factually grounded image captions generated by
Multimodal LLMs (MLLMs) as stable semantic anchors. Specifically, GDCNet computes semantic and sentiment discrepancies between the generated objective description and the original text, alongside measuring visual-textual fidelity. These discrepancy features are then fused with visual and textual representations via a gated module to adaptively balance modality contributions. Extensive experiments on MSD
benchmarks demonstrate GDCNet's superior accuracy and robustness, establishing a new state-of-the-art on the MMSD2.0
benchmark.
Read the full abstract ▾
All authors Shuguang Zhang, Junhong Lian, Guoxin Yu, Baoxun Xu, Xiang Ao
Computation and Language · Cross-listed from cs.AI
Harder Is Better: Boosting Mathematical Reasoning via Difficulty-Aware GRPO and Multi-Aspect Question Reformulation
Yanqi Dai, Yuxiang Ji, Xiao Zhang et al. (+3 more)
Reinforcement Learning with Verifiable Rewards (RLVR) offers a robust mechanism for enhancing mathematical
reasoning in large models. However, we identify a systematic lack of emphasis on more challenging questions in existing methods from both algorithmic and data perspectives, despite their importance for refining underdeveloped capabilities. Algorithmically, widely used Group Relative Policy Optimization (GRPO) suffers from an implicit imbalance where the magnitude of policy updates is lower for harder questions. Data-wise, augmentation approaches primarily rephrase questions to enhance diversity without systematically increasing intrinsic difficulty. To address these issues, we propose a two-dual MathForge framework to improve mathematical
reasoning by targeting harder questions from both perspectives, which comprises a Difficulty-Aware Group Policy Optimization (DGPO)
algorithm and a Multi-Aspect Question Reformulation (MQR) strategy. Specifically, DGPO first rectifies the implicit imbalance in GRPO via difficulty-balanced group advantage estimation, and further prioritizes harder questions by difficulty-aware question-level weighting. Meanwhile, MQR reformulates questions across multiple aspects to increase difficulty while maintaining the original gold answer. Overall, MathForge forms a synergistic loop: MQR expands the data frontier, and DGPO effectively learns from the augmented data. Extensive experiments show that MathForge significantly outperforms existing methods on various mathematical
reasoning tasks. The code and augmented data are all available at https://github.com/AMAP-ML/MathForge.
Read the full abstract ▾
All authors Yanqi Dai, Yuxiang Ji, Xiao Zhang, Yong Wang, Xiangxiang Chu, Zhiwu Lu
Computation and Language
AgentIF-OneDay: A Task-level Instruction-Following Benchmark for General AI Agents in Daily Scenarios
Kaiyuan Chen, Qimin Wu, Taiyu Hou et al. (+42 more)
The capacity of AI
agents to effectively handle tasks of increasing duration and complexity continues to grow, demonstrating exceptional performance in coding, deep research, and complex problem-solving evaluations. However, in daily scenarios, the perception of these advanced AI capabilities among general users remains limited. We argue that current evaluations prioritize increasing task difficulty without sufficiently addressing the diversity of agentic tasks necessary to cover the daily work, life, and learning activities of a broad demographic. To address this, we propose AgentIF-OneDay, aimed at determining whether general users can utilize natural
language instructions and AI
agents to complete a diverse array of daily tasks. These tasks require not only solving problems through dialogue but also understanding various attachment types and delivering tangible file-based results. The
benchmark is structured around three user-centric categories: Open Workflow Execution, which assesses adherence to explicit and complex workflows; Latent Instruction, which requires
agents to infer implicit instructions from attachments; and Iterative Refinement, which involves modifying or expanding upon ongoing work. We employ instance-level rubrics and a refined evaluation pipeline that aligns LLM-based verification with human judgment, achieving an 80.1% agreement rate using Gemini-3-Pro. AgentIF-OneDay comprises 104 tasks covering 767 scoring points. We benchmarked four leading general AI
agents and found that
agent products built based on APIs and ChatGPT
agents based on
agent RL remain in the first tier simultaneously. Leading LLM APIs and open-source models have internalized agentic capabilities, enabling AI application teams to develop cutting-edge
Agent products.
Read the full abstract ▾
All authors Kaiyuan Chen, Qimin Wu, Taiyu Hou, Tianhao Tang, Xueyu Hu, Yuchen Hou, Bikun Li, Chengming Qian, Guoyin Wang, Haolin Chen, Haotong Tian, Haoye Zhang, Haoyu Bian, Hongbing Pan, Hongkang Zhang, Hongyi Zhou, Jiaqi Cai, Jiewu Rao, Jiyuan Ren, Keduan Huang, Lucia Zhu Huang, Mingyu Yuan, Naixu Guo, Qicheng Tang, Qinyan Zhang, Shuai Chen, Siheng Chen, Ting Ting Li, Xiaoxing Guo, Yaocheng Zuo, Yaoqi Guo, Yinan Wang, Yinzhou Yu, Yize Wang, Yuan Jiang, Yuan Tian, Yuanshuo Zhang, Yuxuan Liu, Yvette Yan Zeng, Zenyu Shan, Zihan Yin, Xiaobo Hu, Yang Liu, Yixin Ren, Yuan Gong
Computation and Language
A Computational Approach to Language Contact – A Case Study of Persian
Ali Basirat, Danial Namazifard and Navid Baradaran Hemmati
We investigate structural traces of
language contact in the intermediate representations of a monolingual
language model. Focusing on Persian (Farsi) as a historically contact-rich
language, we probe the representations of a Persian-trained model when exposed to
languages with varying degrees and types of contact with Persian. Our methodology quantifies the amount of linguistic information encoded in intermediate representations and assesses how this information is distributed across model components for different morphosyntactic features. The results show that universal syntactic information is largely insensitive to historical contact, whereas morphological features such as Case and Gender are strongly shaped by
language-specific structure, suggesting that contact effects in monolingual
language models are selective and structurally constrained.
Read the full abstract ▾
Computation and Language
Single-Nodal Spontaneous Symmetry Breaking in NLP Models
Shalom Rosner, Ronit D. Gross, Ella Koresh et al. (+1 more)
Spontaneous
symmetry breaking in statistical mechanics primarily occurs during phase transitions at the thermodynamic limit where the
Hamiltonian preserves inversion
symmetry, yet the low-temperature free energy exhibits reduced
symmetry. Herein, we demonstrate the emergence of spontaneous
symmetry breaking in natural
language processing (NLP) models during both pre-training and fine-tuning, even under deterministic dynamics and within a finite training architecture. This phenomenon occurs at the level of individual
attention heads and is scaled-down to its small subset of nodes and also valid at a single-nodal level, where nodes acquire the capacity to learn a limited set of
tokens after pre-training or labels after fine-tuning for a specific classification task. As the number of nodes increases, a crossover in learning ability occurs, governed by the tradeoff between a decrease following random-guess among increased possible outputs, and enhancement following nodal cooperation, which exceeds the sum of individual nodal capabilities. In contrast to
spin-glass systems, where a microscopic state of frozen
spins cannot be directly linked to the free-energy minimization goal, each nodal function in this framework contributes explicitly to the global network task and can be upper-bounded using convex hull analysis. Results are demonstrated using BERT-6 architecture pre-trained on Wikipedia
dataset and fine-tuned on the FewRel classification task.
Read the full abstract ▾
All authors Shalom Rosner, Ronit D. Gross, Ella Koresh, Ido Kanter