Computation and Language · Cross-listed from cs.IR
RAG-DIVE: A Dynamic Approach for Multi-Turn Dialogue Evaluation in Retrieval-Augmented Generation
Lorenz Brehme, Benedikt Dornauer, Jan-Henrik Böttcher et al. (+3 more)
Evaluating
Retrieval-Augmented Generation (RAG) systems using static multi-turn
datasets fails to capture the dynamic nature of real-world dialogues. Existing evaluation methods rely on predefined
datasets, which restrict them to static, one-directional queries and limit their ability to capture the adaptive, context-dependent performance of RAG systems in interactive, multi-turn settings. Thus, we introduce the RAG-DIVE, a Dynamic Interactive Validation and Evaluation approach, that simulates user interactions with RAG systems. RAG-DIVE leverages an LLM to generate multi-turn conversations dynamically and is organized into three components. The dialogue generation stage consists of the (1) Conversation Generator, which simulates a user by creating multi-turn queries, and the (2) Conversation Validator, which filters and corrects invalid or low-quality outputs to ensure coherent conversations. The evaluation stage is handled by the (3) Conversation Evaluator, which assesses the RAG system's performance across the entire dialogue and generates both per-turn and multi-turn metrics that provide an aggregated view of system behavior. We validated RAG-DIVE through two experimental setups. First, we tested a sample RAG system, including human evaluation of dialogue quality, repeated trials to assess consistency, and an ablation study showing that RAG-DIVE detects performance changes caused by system modifications. Second, we compared RAG-DIVE with a traditional static
dataset evaluation on an industrial RAG system under different configurations to verify whether both approaches reveal similar performance trends. Our findings demonstrate that RAG-DIVE facilitates dynamic, interaction-driven evaluation for multi-turn conversations, thereby advancing the assessment of RAG systems.
Read the full abstract ▾
All authors Lorenz Brehme, Benedikt Dornauer, Jan-Henrik Böttcher, Klaus Schmid, Mircea-Cristian Racasan, Ruth Breu
Computation and Language
Applied Sociolinguistic AI for Community Development (ASA-CD): A New Scientific Paradigm for Linguistically-Grounded Social Intervention
S M Ruhul Alam and Rifa Ferzana
This paper establishes Applied Sociolinguistic AI for Community Development (ASA-CD) as a novel scientific paradigm for addressing community challenges through linguistically grounded, AI-enabled intervention. ASA-CD introduces three key contributions: (1) linguistic biomarkers as computational indicators of discursive fragmentation; (2) development-aligned natural
language processing (NLP), an AI optimisation paradigm prioritising collective outcomes; and (3) a standardised five-phase protocol for discursive intervention. A proof-of-concept study, incorporating real-world and synthetic corpora, demonstrates systematic associations between exclusionary
language and negative sentiment and simulates intervention-based improvements. ASA-CD provides a unified methodological, ethical and empirical framework for scalable, value-aligned AI in the service of community empowerment.
Read the full abstract ▾
Computation and Language
EQ-5D Classification Using Biomedical Entity-Enriched Pre-trained Language Models and Multiple Instance Learning
Zhyar Rzgar K Rostam and Gábor Kertész
The EQ-5D (EuroQol 5-Dimensions) is a standardized instrument for the evaluation of health-related quality of life. In health economics, systematic literature reviews (SLRs) depend on the correct identification of publications that use the EQ-5D, but manual screening of large volumes of scientific literature is time-consuming, error-prone, and inconsistent. In this study, we investigate fine-tuning of general-purpose (BERT) and domain-specific (SciBERT, BioBERT) pre-trained
language models (PLMs), enriched with biomedical entity information extracted through scispaCy models for each statement, to improve EQ-5D detection from abstracts. We conduct nine experimental setups, including combining three scispaCy models with three PLMs, and evaluate their performance at both the sentence and study levels. Furthermore, we explore a Multiple Instance Learning (MIL) approach with
attention pooling to aggregate sentence-level information into study-level predictions, where each abstract is represented as a bag of enriched sentences (by scispaCy). The findings indicate consistent improvements in F1-scores (reaching 0.82) and nearly perfect recall at the study-level, significantly exceeding classical bag-of-words baselines and recently reported PLM baselines. These results show that entity enrichment significantly improves domain adaptation and model generalization, enabling more accurate automated screening in systematic reviews.
Read the full abstract ▾
Computation and Language
Inference-time Alignment via Sparse Junction Steering
Runyi Hu, Jie Zhang, Shiqian Zhao et al. (+7 more)
Token-level steering has emerged as a pivotal approach for inference-time
alignment, enabling fine grained control over large
language models by modulating their output distributions without parameter updates. While effective, existing methods rely on dense intervention at every decoding step. This persistent manipulation not only incurs substantial computational overhead but also risks compromising generation quality by excessively drifting from the model's intrinsic distribution. In this work, we show that dense intervention is unnecessary and propose Sparse Inference time
Alignment (SIA), which performs sparse junction steering by intervening only at critical decision points along the generation trajectory. Our key insight is that high
entropy junctions mark pivotal decision points in the generation trajectory and are particularly susceptible to misalignment, indicating the need to introduce
alignment related reward signals at these points. Extensive experiments across different model families and
alignment objectives show that steering only 20% to 80% of
tokens achieves superior
alignment-efficiency trade offs. For strong base models such as Qwen3, intervening on as few as 20% of
tokens matches or even surpasses heavily post-trained instruct models. This sparsity enables stronger guidance while better preserving the model's native distribution, integrates seamlessly with search based methods such as Best-of-N, and reduces computational cost by up to 6x.
Read the full abstract ▾
All authors Runyi Hu, Jie Zhang, Shiqian Zhao, Jiale Meng, Jiwei Li, Jason Zeng, Ming Wu, Michael Heinrich, Yonggang Wen, Tianwei Zhang
Computation and Language
Shiyu Wang, Haolin Chen, Liangwei Yang et al. (+8 more)
We propose a
diffusion-based framework for
prompt optimization that leverages
Diffusion Language Models (DLMs) to iteratively refine system
prompts through masked denoising. By conditioning on interaction traces, including user queries, model responses, and optional feedback, our method enables flexible, span-level
prompt updates without requiring gradient access or modifying the downstream
language model. Across diverse
benchmarks (e.g., $\tau$-bench, SST-2, SST-5), DLM-optimized
prompts consistently improve the performance of a frozen target LLM (e.g., GPT-4o-mini). We further show that moderate
diffusion step counts provide the best balance between refinement quality and stability. These results highlight
diffusion-based
prompt optimization as a general, model-agnostic, and scalable approach for enhancing LLM performance through iterative
prompt refinement.
Read the full abstract ▾
All authors Shiyu Wang, Haolin Chen, Liangwei Yang, Jielin Qiu, Rithesh Murthy, Ming Zhu, Zixiang Chen, Silvio Savarese, Caiming Xiong, Shelby Heinecke, Huan Wang
Computation and Language · Cross-listed from cs.AI
Retrieval Augmented (Knowledge Graph), and Large Language Model-Driven Design Structure Matrix (DSM) Generation of Cyber-Physical Systems
H. Sinan Bank and Daniel R. Herber
We explore the potential of Large
Language Models (LLMs),
Retrieval-Augmented Generation (RAG), and Graph-based RAG (GraphRAG) for generating Design Structure
Matrices (DSMs). We test these methods on two distinct use cases–a power screwdriver and a CubeSat with known architectural references–evaluating their performance on two key tasks: determining relationships between predefined components, and the more complex challenge of identifying components and their subsequent relationships. We measure the performance by assessing each element of the DSM and overall architecture. Despite design and computational challenges, we identify opportunities for automated DSM generation, with all code publicly available for reproducibility and further feedback from the domain experts.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.CV
Brazilian Portuguese Image Captioning with Transformers: A Study on Cross-Native-Translated Dataset
Gabriel Bromonschenkel, Alessandro L. Koerich, Thiago M. Paixão et al. (+1 more)
Image captioning (IC) refers to the automatic generation of natural
language descriptions for images, with applications ranging from social media content generation to assisting individuals with visual impairments. While most research has been focused on English-based models, low-resource
languages such as Brazilian Portuguese face significant challenges due to the lack of specialized
datasets and models. Several studies create
datasets by automatically translating existing ones to mitigate resource scarcity. This work addresses this gap by proposing a cross-native-translated evaluation of
Transformer-based
vision and
language models for Brazilian Portuguese IC. We use a version of Flickr30K comprised of captions manually created by native Brazilian Portuguese speakers and compare it to a version with captions automatically translated from English to Portuguese. The experiments include a cross-context approach, where models trained on one
dataset are tested on the other to assess the translation impact. Additionally, we incorporate
attention maps for model inference interpretation and use the CLIP-Score metric to evaluate the image-description
alignment. Our findings show that Swin-DistilBERTimbau consistently outperforms other models, demonstrating strong generalization across
datasets. ViTucano, a Brazilian Portuguese pre-trained VLM, surpasses larger
multilingual models (GPT-4o, LLaMa 3.2
Vision) in traditional text-based evaluation metrics, while GPT-4 models achieve the highest CLIP-Score, highlighting improved image-text
alignment.
Attention analysis reveals systematic biases, including gender misclassification, object enumeration errors, and spatial inconsistencies. The
datasets and the models generated and analyzed during the current study are available in: https://github.com/laicsiifes/
transformer-caption-ptbr.
Read the full abstract ▾
All authors Gabriel Bromonschenkel, Alessandro L. Koerich, Thiago M. Paixão, Hilário Tomaz Alves de Oliveira
Computation and Language
Clause-Internal or Clause-External? Testing Turkish Reflexive Binding in Adapted versus Chain of Thought Large Language Models
Sercan Karakaş
This study evaluates whether state-of-the-art large
language models capture the binding relations of Turkish reflexive pronouns. We construct a balanced evaluation set of 100 Turkish sentences that systematically pit local against non-local antecedents for the reflexives kendi and kendisi. We compare two contrasting systems: an OpenAI chain-of-thought model optimized for multi-step
reasoning and Trendyol-LLM-7B-base-v0.1, a LLaMA 2 derived model extensively fine-tuned on Turkish data. Antecedent choice is assessed using a combined paradigm that integrates sentence-level perplexity with a forced-choice comparison between minimally differing continuations. Overall, Trendyol-LLM favors local bindings in approximately 70 percent of trials, exhibiting a robust locality bias consistent with a preference for structurally proximate antecedents. By contrast, the OpenAI model (o1 Mini) distributes its choices nearly evenly between local and long-distance readings, suggesting weaker or less consistent sensitivity to locality in this binding configuration. Taken together, these results reveal a marked contrast in binding behavior across the two systems and motivate closer analysis of how model architecture, training data, and inference-time
reasoning strategies shape the representation of Turkish anaphoric dependencies.
Read the full abstract ▾
Computation and Language
DecompressionLM: Deterministic, Diagnostic, and Zero-Shot Concept Graph Extraction from Language Models
Zhaochen Hong and Jiaxuan You
Existing knowledge probing methods rely on pre-defined queries, limiting extraction to known concepts. We introduce DecompressionLM, a stateless framework for zero-shot concept graph extraction that discovers what
language models encode without pre-specified queries or shared cross-sequence state. Our method targets three limitations of common decoding-based probing approaches: (i) cross-sequence coupling that concentrates probability mass on high-frequency prefixes, (ii) competitive decoding effects that suppress long-tail concepts, and (iii) scalability constraints arising from sequential exploration. Using Van der Corput low-discrepancy sequences with arithmetic decoding, DecompressionLM enables deterministic, embarrassingly parallel generation without shared state across sequences. Across two model families and five quantization variants, we find that activation-aware quantization (AWQ-4bit) expands concept coverage by 30-170%, while uniform quantization (GPTQ-Int4) induces 71-86% coverage collapse - divergent behaviors not reliably reflected by explanation-level perplexity. Corpus-based verification further reveals a 19.6-point
hallucination gap between top- and bottom-ranked MMLU-Pro Law models. DecompressionLM establishes concept coverage as a complementary evaluation dimension for assessing knowledge breadth and factual grounding in compressed models intended for deployment. Code is available at https://github.com/ulab-uiuc/decompressionlm.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.LG
Post-Training Probability Manifold Correction via Structured SVD Pruning and Self-Referential Distillation
Aaron R. Flouro and Shawn P. Chadwick
Large
language models are expensive to deploy. We introduce Sparse Knowledge
Distillation (SparseKD), a post-training method that compresses
transformer models by combining structured SVD
pruning with self-referential knowledge
distillation. The key insight is simple: instead of using an external teacher, the model teaches itself by matching its own probability distribution from before compression. This self-referential setup enables surprisingly strong quality recovery after aggressive
pruning. Our experiments reveal an unexpected finding: self-referential
distillation alone, applied post-training under an identical objective and fixed calibration
dataset, improves model quality by 39% relative to the original converged checkpoint. When combined with structured
pruning, SparseKD achieves 15-65% parameter reduction with acceptable quality trade-offs. Kernel profiling shows that speedups arise entirely from reduced dense
matrix multiplication in feed-forward layers while
attention remains unchanged, making this approach complementary to
attention optimizations. We validate across two model families (0.6B and 3.8B parameters) with multi-seed experiments confirming high reproducibility. SparseKD requires no external super-teacher, no architectural changes, and no custom inference kernels, making it immediately deployable with existing infrastructure.
Read the full abstract ▾
Computation and Language
Li Siyan, Darshan Deshpande, Anand Kannappan et al. (+1 more)
When recalling information in conversation, people often arrive at the recollection after multiple turns. However, existing
benchmarks for evaluating
agent capabilities in such tip-of-the-tongue search processes are restricted to single-turn settings. To more realistically simulate tip-of-the-tongue search, we introduce Dual-
agent based Evaluation Through Obscure Under-specified
Retrieval (DETOUR), a dual-
agent evaluation
benchmark containing 1,011
prompts. The
benchmark design involves a Primary
Agent, which is the subject of evaluation, tasked with identifying the recollected entity through querying a
Memory Agent that is held consistent across evaluations. Our results indicate that current state-of-the-art models still struggle with our
benchmark, only achieving 36% accuracy when evaluated on all modalities (text, image, audio, and
video), highlighting the importance of enhancing capabilities in underspecified scenarios.
Read the full abstract ▾
All authors Li Siyan, Darshan Deshpande, Anand Kannappan, Rebecca Qian
Computation and Language · Cross-listed from cs.CV
When RAG Hurts: Diagnosing and Mitigating Attention Distraction in Retrieval-Augmented LVLMs
Beidi Zhao, Wenlong Deng, Xinting Liao et al. (+4 more)
While
Retrieval-Augmented Generation (RAG) is one of the dominant paradigms for enhancing Large
Vision-
Language Models (LVLMs) on knowledge-based VQA tasks, recent work attributes RAG failures to insufficient
attention towards the retrieved context, proposing to reduce the
attention allocated to image
tokens. In this work, we identify a distinct failure mode that previous study overlooked:
Attention Distraction (AD). When the retrieved context is sufficient (highly relevant or including the correct answer), the retrieved text suppresses the visual
attention globally, and the
attention on image
tokens shifts away from question-relevant regions. This leads to failures on questions the model could originally answer correctly without the retrieved text. To mitigate this issue, we propose MAD-RAG, a training-free intervention that decouples visual grounding from context integration through a dual-question formulation, combined with
attention mixing to preserve image-conditioned evidence. Extensive experiments on OK-VQA, E-VQA, and InfoSeek demonstrate that MAD-RAG consistently outperforms existing baselines across different model families, yielding absolute gains of up to 4.76%, 9.20%, and 6.18% over the vanilla RAG baseline. Notably, MAD-RAG rectifies up to 74.68% of failure cases with negligible computational overhead.
Read the full abstract ▾
All authors Beidi Zhao, Wenlong Deng, Xinting Liao, Yushu Li, Nazim Shaikh, Yao Nie, Xiaoxiao Li
Computation and Language
Detecting AI-Generated Content in Academic Peer Reviews
Siyuan Shen and Kai Wang
The growing availability of large
language models (LLMs) has raised questions about their role in academic peer review. This study examines the temporal emergence of AI-generated content in peer reviews by applying a detection model trained on historical reviews to later review cycles at International Conference on Learning Representations (ICLR) and Nature Communications (NC). We observe minimal detection of AI-generated content before 2022, followed by a substantial increase through 2025, with approximately 20% of ICLR reviews and 12% of Nature Communications reviews classified as AI-generated in 2025. The most pronounced growth of AI-generated reviews in NC occurs between the third and fourth quarter of 2024. Together, these findings provide suggestive evidence of a rapidly increasing presence of AI-assisted content in peer review and highlight the need for further study of its implications for scholarly evaluation.
Read the full abstract ▾
Computation and Language
MiNER: A Two-Stage Pipeline for Metadata Extraction from Municipal Meeting Minutes
Rodrigo Batista, Luís Filipe Cunha, Purificação Silvano et al. (+4 more)
Municipal meeting minutes are official documents of local governance, exhibiting heterogeneous formats and writing styles. Effective information
retrieval (IR) requires identifying metadata such as meeting number, date, location, participants, and start/end times, elements that are rarely standardized or easy to extract automatically. Existing named entity recognition (NER) models are ill-suited to this task, as they are not adapted to such domain-specific categories. In this paper, we propose a two-stage pipeline for metadata extraction from municipal minutes. First, a question answering (QA) model identifies the opening and closing text segments containing metadata.
Transformer-based models (BERTimbau and XLM-RoBERTa with and without a CRF layer) are then applied for fine-grained entity extraction and enhanced through deslexicalization. To evaluate our proposed pipeline, we
benchmark both open-weight (Phi) and closed-weight (Gemini) LLMs, assessing predictive performance, inference cost, and carbon footprint. Our results demonstrate strong in-domain performance, better than larger general-purpose LLMs. However, cross-municipality evaluation reveals reduced generalization reflecting the variability and linguistic complexity of municipal records. This work establishes the first
benchmark for metadata extraction from municipal meeting minutes, providing a solid foundation for future research in this domain.
Read the full abstract ▾
All authors Rodrigo Batista, Luís Filipe Cunha, Purificação Silvano, Nuno Guimarães, Alípio Jorge, Evelin Amorim, Ricardo Campos
Computation and Language
Faithful-Patchscopes: Understanding and Mitigating Model Bias in Hidden Representations Explanation of Large Language Models
Xilin Gong, Shu Yang, Zehua Cao et al. (+2 more)
Large
Language Models (LLMs) have demonstrated strong capabilities for hidden representation interpretation through Patchscopes, a framework that uses LLMs themselves to generate human-readable explanations by decoding from internal hidden representations. However, our work shows that LLMs tend to rely on inherent linguistic patterns, which can override contextual information encoded in the hidden representations during decoding. For example, even when a hidden representation encodes the contextual attribute "purple" for "broccoli", LLMs still generate "green" in their explanations, reflecting a strong prior association. This behavior reveals a systematic unfaithfulness in Patchscopes. To systematically study this issue, we first designed a
dataset to evaluate the faithfulness of Patchscopes under biased cases, and our results show that there is an 18.84% faithfulness decrease on average. We then propose Bias
Alignment through Logit Recalibration (BALOR), which treats the output logits from an unpatched
prompt as capturing model bias and contrasts them with logits obtained under patched contextual information. By recalibrating the logit distribution through this contrast, BALOR suppresses model bias and amplifies contextual information during generation. Experiments across multiple LLMs demonstrate that BALOR consistently outperforms existing baselines, achieving up to 33% relative performance improvement.
Read the full abstract ▾
All authors Xilin Gong, Shu Yang, Zehua Cao, Lynne Billard, Di Wang
Computation and Language
Semantic Calibration Prevails Where Token Confidence Fails: Benchmarking Long-Form Scientific QA
Philip Müller, Nicholas Popovič, Michael Färber et al. (+1 more)
Reliable uncertainty quantification (UQ) is essential for safe deployment of large
language models (LLMs) in scientific question answering, where long-form outputs exceed practical human verification at scale. We introduce the first large-scale
benchmark for UQ calibration in long-form,
reasoning-demanding scientific QA, evaluating four UQ methods on 685,000 responses across up to 20 LLMs and seven
datasets, supported by an extensible open-source framework whose shared-generation design enables reproducible cross-method comparisons. Instruction tuning is shown to associate with systematic
token probability polarization, collapsing confidence distributions and undermining the reliability of
token-level uncertainty signals.
Reasoning model families diverge: some reproduce this polarization while others actively mitigate it, a pattern that clusters by provider and suggests training pipeline design as a key differentiating factor. Verbalized and
token-aggregation sequence-level methods fail systematically. Only semantic consistency, as measured by consistency of the final answer, yields well-calibrated outputs, providing the first large-scale evidence that semantic calibration persists in multi-step, dependency-rich
reasoning settings.
Read the full abstract ▾
All authors Philip Müller, Nicholas Popovič, Michael Färber, Peter Steinbach
Computation and Language · Cross-listed from cs.LG
TABES: Trajectory-Aware Backward-on-Entropy Steering for Masked Diffusion Models
Shreshth Saini, Avinab Saha, Balu Adsumilli et al. (+3 more)
Masked
Diffusion Models (MDMs) have emerged as a promising non-autoregressive paradigm for generative tasks, offering parallel decoding and bidirectional context utilization. However, current sampling methods rely on simple confidence-based heuristics that ignore the long-term impact of local decisions, leading to trajectory lock-in where early
hallucinations cascade into global incoherence. While search-based methods mitigate this, they incur prohibitive computational costs ($O(K)$ forward passes per step). In this work, we propose Backward-on-
Entropy (BoE) Steering, a gradient-guided inference framework that approximates infinite-horizon lookahead via a single backward pass. We formally derive the
Token Influence Score (TIS) from a first-order expansion of the trajectory cost functional, proving that the gradient of future
entropy with respect to input
embeddings serves as an optimal control signal for minimizing uncertainty. To ensure scalability, we introduce ActiveQueryAttention, a sparse adjoint primitive that exploits the structure of the masking objective to reduce backward pass complexity. BoE achieves a superior Pareto frontier for inference-time scaling compared to existing unmasking methods, demonstrating that gradient-guided steering offers a mathematically principled and efficient path to robust non-autoregressive generation. We will release the code.
Read the full abstract ▾
All authors Shreshth Saini, Avinab Saha, Balu Adsumilli, Neil Birkbeck, Yilin Wang, Alan C. Bovik
Computation and Language
DIVERGE: Diversity-Enhanced RAG for Open-Ended Information Seeking
Tianyi Hu, Niket Tandon and Akhil Arora
Existing
retrieval-augmented generation (RAG) systems often assume that each query has a single correct answer. This assumption overlooks open-ended information-seeking scenarios where multiple plausible answers are valuable, and where diversity is important for creativity,
fairness, and inclusive access to information. We show that standard RAG systems fail to fully use diverse retrieved contexts: simply increasing
retrieval diversity does not necessarily lead to diverse generations. To address this limitation, we propose Diverge, a plug-and-play agentic RAG framework that improves the diversity–quality trade-off through iterative, reflection-guided exploration of diverse viewpoints and diversity-aware
retrieval support. We further introduce evaluation metrics for characterizing the diversity-quality trade-off in open-ended question answering. Experiments across multiple real-world
datasets and backbone LLMs show that Diverge achieves the best trade-off among competitive baselines, increasing diversity by $\sim2\times$ without noticeable quality degradation. These results reveal a systematic limitation of current RAGs and show the value of explicit diversity modeling.
Read the full abstract ▾
Computation and Language · Cross-listed from q-bio.QM
Rank-and-Reason: Multi-Agent Collaboration Accelerates Zero-Shot Protein Mutation Prediction
Yang Tan, Yuanxi Yu, Can Wu et al. (+7 more)
Zero-shot mutation prediction is vital for low-resource
protein engineering, yet existing
protein language models (PLMs) often yield statistically confident results that ignore fundamental biophysical constraints. Currently, selecting candidates for wet-lab validation relies on manual expert
auditing of PLM outputs, a process that is inefficient, subjective, and highly dependent on domain expertise. To address this, we propose Rank-and-Reason (VenusRAR), a two-stage agentic framework to automate this workflow and maximize expected wet-lab fitness. In the Rank-Stage, a Computational Expert and Virtual Biologist aggregate a context-aware multi-modal ensemble, establishing a new Spearman correlation record of 0.551 (vs. 0.518) on ProteinGym. In the Reason-Stage, an agentic Expert Panel employs chain-of-thought
reasoning to audit candidates against
geometric and structural constraints, improving the Top-5 Hit Rate by up to 367% on ProteinGym-DMS99. The wet-lab validation on Cas12i3 nuclease further confirms the framework's efficacy, achieving a 46.7% positive rate and identifying two novel mutants with 4.23-fold and 5.05-fold activity improvements. Code and
datasets are released on GitHub (https://github.com/ai4
protein/VenusRAR/).
Read the full abstract ▾
All authors Yang Tan, Yuanxi Yu, Can Wu, Bozitao Zhong, Mingchen Li, Guisheng Fan, Jiankang Zhu, Yafeng Liang, Nanqing Dong, Liang Hong
Computation and Language · Cross-listed from cs.LG
FOCUS: DLLMs Know How to Tame Their Compute Bound
Kaihua Liang, Xin Tan, An Zhong et al. (+2 more)
Diffusion Large
Language Models (DLLMs) offer a compelling alternative to Auto-Regressive models, but their deployment is constrained by high decoding cost. In this work, we identify a key inefficiency in DLLM decoding: while computation is parallelized over
token blocks, only a small subset of
tokens is decodable at each
diffusion step, causing most compute to be wasted on non-decodable
tokens. We further observe a strong correlation between
attention-derived
token importance and
token-wise decoding probability. Based on this insight, we propose FOCUS, an inference system designed for DLLMs. By dynamically focusing computation on decodable
tokens and evicting non-decodable ones on-the-fly, FOCUS increases the effective batch size, alleviating compute limitations and enabling scalable throughput. Empirical evaluations demonstrate that FOCUS achieves up to 3.52$\times$ throughput improvement over the production-grade engine LMDeploy in large-batch settings, while preserving or improving generation quality across multiple
benchmarks.
Read the full abstract ▾
All authors Kaihua Liang, Xin Tan, An Zhong, Hong Xu, Marco Canini
Computation and Language
UPA: Unsupervised Prompt Agent via Tree-Based Search and Selection
Siran Peng, Weisong Zhao, Tianyu Fu et al. (+6 more)
Prompt agents have recently emerged as a promising paradigm for automated
prompt optimization, framing
prompt discovery as a sequential decision-making problem over a structured
prompt space. While this formulation enables the use of advanced planning
algorithms, these methods typically assume access to supervised reward signals, which are often unavailable in practical scenarios. In this work, we propose UPA, an Unsupervised
Prompt Agent that realizes structured search and selection without relying on ground-truth (GT) rewards. Specifically, during search, UPA iteratively constructs an evolving tree structure to navigate the
prompt space, guided by fine-grained and position-debiased pairwise comparisons from Large
Language Models (LLMs). Crucially, as these local comparisons do not inherently yield a consistent global scale, we decouple systematic
prompt exploration from final selection, introducing a two-stage framework grounded in the Bradley-Terry-Luce (BTL) model. This framework first performs path-wise
Bayesian aggregation of local comparisons to filter candidates under uncertainty, followed by global tournament-style comparisons to infer latent
prompt quality and identify the optimal
prompt. Experiments across multiple tasks demonstrate that UPA consistently outperforms existing
prompt optimization methods, showing that
agent-style optimization can remain highly effective even in unsupervised settings.
Read the full abstract ▾
All authors Siran Peng, Weisong Zhao, Tianyu Fu, Chenxu Zhao, Tianshuo Zhang, Haoyuan Zhang, Xiangyu Zhu, Minghui Wu, Zhen Lei
Computation and Language
PaperBanana: Automating Academic Illustration for AI Scientists
Dawei Zhu, Rui Meng, Yale Song et al. (+4 more)
Despite rapid advances in autonomous AI scientists powered by
language models, generating publication-ready illustrations remains a labor-intensive bottleneck in the research workflow. To lift this burden, we introduce PaperBanana, an agentic framework for automated generation of publication-ready academic illustrations. Powered by state-of-the-art VLMs and image generation models, PaperBanana orchestrates specialized
agents to retrieve references, plan content and style, render images, and iteratively refine via self-critique. To rigorously evaluate our framework, we introduce PaperBananaBench, comprising 292 test cases for methodology diagrams curated from NeurIPS 2025 publications, covering diverse research domains and illustration styles. Comprehensive experiments demonstrate that PaperBanana consistently outperforms leading baselines in faithfulness, conciseness, readability, and aesthetics. We further show that our method effectively extends to the generation of high-quality statistical plots. Collectively, PaperBanana paves the way for the automated generation of publication-ready illustrations.
Read the full abstract ▾
All authors Dawei Zhu, Rui Meng, Yale Song, Xiyu Wei, Sujian Li, Tomas Pfister, Jinsung Yoon
Computation and Language · Cross-listed from cs.LG
Agnostic Language Identification and Generation
Mikael Møller Høgsgaard and Chirag Pabbaraju
Recent works on
language identification and generation have established tight statistical rates at which these tasks can be achieved. These works typically operate under a strong realizability assumption: that the input data is drawn from an unknown distribution necessarily supported on some
language in a given collection. In this work, we relax this assumption of realizability entirely, and impose no restrictions on the distribution of the input data. We propose objectives to study both
language identification and generation in this more general "agnostic" setup. Across both problems, we obtain novel interesting characterizations and nearly tight rates.
Read the full abstract ▾
Computation and Language
Now You Hear Me: Audio Narrative Attacks Against Large Audio-Language Models
Ye Yu, Haibo Jin, Yaoning Yu et al. (+2 more)
Large audio-
language models increasingly operate on raw
speech inputs, enabling more seamless integration across domains such as voice assistants, education, and clinical triage. This transition, however, introduces a distinct class of vulnerabilities that remain largely uncharacterized. We examine the security implications of this modality shift by designing a text-to-audio jailbreak that embeds disallowed directives within a narrative-style audio stream. The attack leverages an advanced instruction-following text-to-
speech (TTS) model to exploit structural and acoustic properties, thereby circumventing safety mechanisms primarily calibrated for text. When delivered through synthetic
speech, the narrative format elicits restricted outputs from state-of-the-art models, including Gemini 2.0 Flash, achieving a 98.26% success rate that substantially exceeds text-only baselines. These results highlight the need for safety frameworks that jointly reason over linguistic and paralinguistic representations, particularly as
speech-based interfaces become more prevalent.
Read the full abstract ▾
All authors Ye Yu, Haibo Jin, Yaoning Yu, Jun Zhuang, Haohan Wang
Computation and Language · Cross-listed from cs.AI
Scaling Multiagent Systems with Process Rewards
Ed Li, Junyu Ren and Cat Yan
While multiagent systems have shown promise for tackling complex tasks via specialization, finetuning multiple
agents simultaneously faces two key challenges: (1) credit assignment across
agents, and (2) sample efficiency of expensive multiagent rollouts. In this work, we propose finetuning multiagent systems with per-action process rewards from AI feedback (MAPPA) to address both. Through assigning credit to individual
agent actions rather than only at task completion, MAPPA enables fine-grained supervision without ground truth labels while extracting maximal training signal from each rollout. We demonstrate our approach on competition math problems and tool-augmented data analysis tasks. On unseen math problems, MAPPA achieves +5.0–17.5pp on AIME and +7.8–17.2pp on AMC. For data analysis tasks, our method improves success rate by +16.7pp while quality metrics improve by up to 47%, validating that per-action supervision can lead to improvements across different multiagent systems on various domains. By addressing these challenges, our work takes a first step toward scaling multiagent systems for complex, long-horizon tasks with minimal human supervision.
Read the full abstract ▾
Computation and Language
Are you going to finish that? A Practical Study of the Partial Token Problem
Hao Xu, Alisa Liu, Jonathan Hayase et al. (+2 more)
Language models (LMs) are trained over sequences of
tokens, whereas users interact with LMs via text. This mismatch gives rise to the partial
token problem, which occurs when a user ends their
prompt in the middle of the expected next-
token, leading to distorted next-
token predictions. Although this issue has been studied using arbitrary character prefixes, its prevalence and severity in realistic
prompts respecting word boundaries remains underexplored. In this work, we identify three domains where
token and "word" boundaries often do not line up:
languages that do not use whitespace, highly compounding
languages, and code. In Chinese, for example, up to 25% of word boundaries do not line up with
token boundaries, making even natural, word-complete
prompts susceptible to this problem. We systematically construct semantically natural
prompts ending with a partial
tokens; in experiments, we find that they comprise a serious failure mode: frontier LMs consistently place three orders of magnitude less probability on the correct continuation compared to when the
prompt is "backed-off" to be
token-aligned. This degradation does not diminish with scale and often worsens for larger models. Finally, we evaluate inference-time mitigations to the partial
token problem and validate the effectiveness of recent exact solutions. Overall, we demonstrate the scale and severity of probability distortion caused by tokenization in realistic use cases, and provide practical recommentions for model inference providers.
Read the full abstract ▾
All authors Hao Xu, Alisa Liu, Jonathan Hayase, Yejin Choi, Noah A. Smith
Computation and Language
Deep Search with Hierarchical Meta-Cognitive Monitoring Inspired by Cognitive Neuroscience
Zhongxiang Sun, Qipeng Wang, Weijie Yu et al. (+3 more)
Deep search
agents powered by large
language models have demonstrated strong capabilities in multi-step
retrieval,
reasoning, and long-horizon task execution. However, their practical failures often stem from the lack of mechanisms to monitor and regulate
reasoning and
retrieval states as tasks evolve under uncertainty. Insights from cognitive neuroscience suggest that human metacognition is hierarchically organized, integrating fast anomaly detection with selectively triggered, experience-driven reflection. In this work, we propose Deep Search with Meta-Cognitive Monitoring (DS-MCM), a deep search framework augmented with an explicit hierarchical metacognitive monitoring mechanism. DS-MCM integrates a Fast Consistency Monitor, which performs lightweight checks on the
alignment between external evidence and internal
reasoning confidence, and a Slow Experience-Driven Monitor, which is selectively activated to guide corrective intervention based on experience
memory from historical
agent trajectories. By
embedding monitoring directly into the
reasoning-
retrieval loop, DS-MCM determines both when intervention is warranted and how corrective actions should be informed by prior experience. Experiments across multiple deep search
benchmarks and backbone models demonstrate that DS-MCM consistently improves performance and robustness.
Read the full abstract ▾
All authors Zhongxiang Sun, Qipeng Wang, Weijie Yu, Jingxuan Yang, Haolang Lu, Jun Xu