Computation and Language
Nicholas Moratelli, Christopher Davis, Leonardo F. R. Ribeiro et al. (+2 more)
Large
Vision-
Language Models (LVLMs) increasingly rely on
retrieval to answer knowledge-intensive
multimodal questions. Existing
benchmarks overlook conflicts between visual and textual evidence and the importance of generating deflections (e.g., Sorry, I cannot answer...) when retrieved knowledge is incomplete. These
benchmarks also suffer from rapid obsolescence, as growing LVLM training sets allow models to answer many questions without
retrieval. We address these gaps with three contributions. First, we propose a dynamic data curation pipeline that preserves
benchmark difficulty over time by filtering for genuinely
retrieval-dependent samples. Second, we introduce VLM-DeflectionBench, a
benchmark of 2,775 samples spanning diverse
multimodal retrieval settings, designed to probe model behaviour under conflicting or insufficient evidence. Third, we define a fine-grained evaluation protocol with four scenarios that disentangle parametric memorization from
retrieval robustness. Experiments across 20 state-of-the-art LVLMs indicate that models usually fail to deflect in the presence of noisy or misleading evidence. Our results highlight the need to evaluate not only what models know, but how they behave when they do not, and serve as a reusable and extensible
benchmark for reliable KB-VQA evaluation. All resources will be publicly available upon publication.
Read the full abstract ▾
All authors Nicholas Moratelli, Christopher Davis, Leonardo F. R. Ribeiro, Bill Byrne, Gonzalo Iglesias
Computation and Language
LLMs Struggle with Abstract Meaning Comprehension More Than Expected
Hamoud Alhazmi and Jiachen Jiang
Understanding abstract meanings is crucial for advanced
language comprehension. Despite extensive research, abstract words remain challenging due to their non-concrete, high-level semantics. SemEval-2021 Task 4 (ReCAM) evaluates models' ability to interpret abstract concepts by presenting passages with questions and five abstract options in a cloze-style format. Key findings include: (1) Most large
language models (LLMs), including GPT-4o, struggle with abstract meaning comprehension under zero-shot, one-shot, and few-shot settings, while fine-tuned models like BERT and RoBERTa perform better. (2) A proposed bidirectional
attention classifier, inspired by human cognitive strategies, enhances fine-tuned models by dynamically attending to passages and options. This approach improves accuracy by 4.06 percent on Task 1 and 3.41 percent on Task 2, demonstrating its potential for abstract meaning comprehension.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.LG
UCS: Estimating Unseen Coverage for Improved In-Context Learning
Jiayi Xin, Xiang Li, Evan Qiang et al. (+4 more)
In-context learning (ICL) performance depends critically on which demonstrations are placed in the
prompt, yet most existing selectors prioritize heuristic notions of relevance or diversity and provide limited insight into the coverage of a demonstration set. We propose Unseen Coverage Selection (UKS), a training-free, subset-level coverage prior motivated by the principle that a good demonstration set should expose the model to latent cluster unrevealed by the currently selected subset. UCS operationalizes this idea by (1) inducing discrete latent clusters from model-consistent
embeddings and (2) estimating the number of unrevealed clusters within a candidate subset via a Smoothed Good–Turing
estimator from its empirical frequency spectrum. Unlike previous selection methods, UCS is coverage-based and training-free, and can be seamlessly combined with both query-dependent and query-independent selection baselines via a simple regularized objective. Experiments on multiple intent-classification and
reasoning benchmarks with frontier Large
Language Models show that augmenting strong baselines with UCS consistently improves ICL accuracy by up to 2-6% under the same selection budget, while also yielding insights into task- and model-level latent cluster distributions. Code is available at https://github.com/Raina-Xin/UCS.
Read the full abstract ▾
All authors Jiayi Xin, Xiang Li, Evan Qiang, Weiqing He, Tianqi Shang, Weijie J. Su, Qi Long
Computation and Language
Self-Distillation Zero: Self-Revision Turns Binary Rewards into Dense Supervision
Yinghui He, Simran Kaur, Adithya Bhaskar et al. (+7 more)
Current post-training methods in verifiable settings fall into two categories. Reinforcement learning (RLVR) relies on binary rewards, which are broadly applicable and powerful, but provide only sparse supervision during training.
Distillation provides dense
token-level supervision, typically obtained from an external teacher or using high-quality demonstrations. Collecting such supervision can be costly or unavailable. We propose Self-
Distillation Zero (SD-Zero), a method that is substantially more training sample-efficient than RL and does not require an external teacher or high-quality demonstrations. SD-Zero trains a single model to play two roles: a Generator, which produces an initial response, and a Reviser, which conditions on that response and its binary reward to produce an improved response. We then perform on-policy self-
distillation to distill the reviser into the generator, using the reviser's
token distributions conditioned on the generator's response and its reward as supervision. In effect, SD-Zero trains the model to transform binary rewards into dense
token-level self-supervision. On math and code
reasoning benchmarks with Qwen3-4B-Instruct and Olmo-3-7B-Instruct, SD-Zero improves performance by at least 10% over the base models and outperforms strong baselines, including Rejection Fine-Tuning (RFT), GRPO, and Self-
Distillation Fine-Tuning (SDFT), under the same question set and training sample budget. Extensive ablation studies show two novel characteristics of our proposed
algorithm: (a)
token-level self-localization, where the reviser can identify the key
tokens that need to be revised in the generator's response based on reward, and (b) iterative self-
evolution, where the improving ability to revise answers can be distilled back into generation performance with regular teacher synchronization. Code: https://github.com/princeton-pli/Self-
Distillation-Zero.
Read the full abstract ▾
All authors Yinghui He, Simran Kaur, Adithya Bhaskar, Yongjin Yang, Jiarui Liu, Narutatsu Ri, Liam Fowl, Abhishek Panigrahi, Danqi Chen, Sanjeev Arora
Computation and Language
Filtered Reasoning Score: Evaluating Reasoning Quality on a Model's Most-Confident Traces
Manas Pathak, Xingyao Chen, Shuozhe Li et al. (+2 more)
Should we trust Large
Language Models (LLMs) with high accuracy? LLMs achieve high accuracy on
reasoning benchmarks, but correctness alone does not reveal the quality of the
reasoning used to produce it. This highlights a fundamental limitation of outcome-based evaluation: models may arrive at correct answers through flawed
reasoning, and models with substantially different
reasoning capabilities can nevertheless exhibit similar
benchmark accuracy, for example due to memorization or over-optimization. In this paper, we ask: given existing
benchmarks, can we move beyond outcome-based evaluation to assess the quality of
reasoning itself? We seek metrics that (1) differentiate models with similar accuracy and (2) are robust to variations in input
prompts and generation configurations. To this end, we propose a
reasoning score that evaluates
reasoning traces along dimensions such as faithfulness, coherence, utility, and factuality. A remaining question is how to aggregate this score across multiple sampled traces. Naively averaging them is undesirable, particularly in long-horizon settings, where the number of possible trajectories grows rapidly, and low-confidence correct traces are more likely to be coincidental. To address this, we introduce the Filtered
Reasoning Score (FRS), which computes
reasoning quality using only the top-K% most confident traces. Evaluating with FRS, models that are indistinguishable under standard accuracy exhibit significant differences in
reasoning quality. Moreover, models with higher FRS on one
benchmark tend to perform better on other
reasoning benchmarks, in both accuracy and
reasoning quality. Together, these findings suggest that FRS complements accuracy by capturing a model's transferable
reasoning capabilities. We open source our evaluation codebase: https://github.com/HumainLab/filtered_
reasoning_score_evaluation.
Read the full abstract ▾
All authors Manas Pathak, Xingyao Chen, Shuozhe Li, Amy Zhang, Liu Leqi
Computation and Language · Cross-listed from cs.CV
INDOTABVQA: A Benchmark for Cross-Lingual Table Understanding in Bahasa Indonesia Documents
Somraj Gautam, Anathapindika Dravichi and Gaurav Harit
We introduce INDOTABVQA, a
benchmark for evaluating cross-lingual Table Visual Question Answering (VQA) on real-world document images in Bahasa Indonesia. The
dataset comprises 1,593 document images across three visual styles (bordered, borderless, and colorful) with one or more than one tables, and 1,593 question-answer sets in four
languages: Bahasa Indonesia, English, Hindi, and Arabic. This enables evaluation of
Vision-
Language Models (VLMs) in both monolingual (Bahasa documents with Bahasa questions) and cross-lingual settings (Bahasa documents with questions in other
languages). We
benchmark leading open-source VLMs (Qwen2.5-VL, Gemma-3, LLaMA-3.2) and GPT-4o and reveal substantial performance gaps, particularly on structurally complex tables and in low-resource
languages. Fine-tuning a compact 3B and LoRA-finetuned 7B model on our
dataset yields 11.6% and 17.8% improvements in accuracy. Providing explicit table region coordinates as additional input further improves performance by 4-7%, demonstrating the value of Spatial priors for table-based
reasoning. Our findings underscore the importance of
language-diverse, domain-specific
datasets and demonstrate that targeted fine-tuning can significantly enhance VLM performance on specialized document understanding tasks. INDOTABVQA provides a valuable resource for advancing research in cross-lingual, structure-aware document understanding, especially in underrepresented regions of the world. Full
dataset can be accessed in huggingface at: https://huggingface.co/
datasets/NusaBharat/INDOTABVQA}
Read the full abstract ▾
Computation and Language · Cross-listed from cs.SE
AnyPoC: Universal Proof-of-Concept Test Generation for Scalable LLM-Based Bug Detection
Zijie Zhao, Chenyuan Yang, Weidong Wang et al. (+3 more)
While recent LLM-based
agents can identify many candidate bugs in source code, their reports remain static hypotheses that require manual validation, limiting the practicality of automated bug detection. We frame this challenge as a test generation task: given a candidate report, synthesizing an executable proof-of-concept (PoC) - such as a script, command sequence, or crafted input - to trigger the suspected defect. Automated PoC generation can act as a scalable validation oracle, enabling end-to-end autonomous bug detection by providing concrete execution evidence. However, naive LLM
agents are unreliable validators: they are biased toward "success" and may reward-hack by producing plausible but non-functional PoCs or even hallucinated traces. To address this, we present ANYPoC, a general multi-
agent framework that (1) analyzes and fact-checks a candidate bug report, (2) iteratively synthesizes and executes a PoC while collecting execution traces, and (3) independently re-executes and scrutinizes the PoC to mitigate
hallucination and reward hacking. In addition, ANYPoC also continuously extracts and evolves a PoC knowledge base to handle heterogeneous tasks. ANYPoC operates on candidate bug reports regardless of their source and can be paired with different bug reporters. To demonstrate practicality and generality, we apply ANYPoC, together with a simple agentic bug reporter, on 12 large-scale, critical software systems, including Firefox, Chromium, LLVM, OpenSSL, SQLite, FFmpeg, and Redis. Compared to the state-of-the-art coding
agents, e.g., Claude Code and Codex, ANYPoC produces 37% more valid PoCs for true-positive bug reports and rejects 9.7x more false-positive bug reports. ANYPoC also enables the discovery of 121 new bugs from over two thousand noisy bug reports, with 108 confirmed by developers and 92 fixed. 46 PoCs have also been adopted as official regression tests.
Read the full abstract ▾
All authors Zijie Zhao, Chenyuan Yang, Weidong Wang, Yihan Yang, Ziqi Zhang, Lingming Zhang
Computation and Language · Cross-listed from cs.AI
GoodPoint: Learning Constructive Scientific Paper Feedback from Author Responses
Jimin Mun, Chani Jung, Xuhui Zhou et al. (+2 more)
While LLMs hold significant potential to transform scientific research, we advocate for their use to augment and empower researchers rather than to automate research without human oversight. To this end, we study constructive feedback generation, the task of producing targeted, actionable feedback that helps authors improve both their research and its presentation. In this work, we operationalize the effectiveness of feedback along two author-centric axes-validity and author action. We first curate GoodPoint-ICLR, a
dataset of 19K ICLR papers with reviewer feedback annotated along both dimensions using author responses. Building on this, we introduce GoodPoint, a training recipe that leverages success signals from author responses through fine-tuning on valid and actionable feedback, together with preference optimization on both real and synthetic preference pairs. Our evaluation on a
benchmark of 1.2K ICLR papers shows that a GoodPoint-trained Qwen3-8B improves the predicted success rate by 83.7% over the base model and sets a new state-of-the-art among LLMs of similar size in feedback matching on a golden human feedback set, even surpassing Gemini-3-flash in precision. We further validate these findings through an expert human study, demonstrating that GoodPoint consistently delivers higher practical value as perceived by authors.
Read the full abstract ▾
All authors Jimin Mun, Chani Jung, Xuhui Zhou, Hyunwoo Kim, Maarten Sap
Computation and Language · Cross-listed from cs.AI
Detecting Safety Violations Across Many Agent Traces
Adam Stein, Davis Brown, Hamed Hassani et al. (+2 more)
To identify safety violations, auditors often search over large sets of
agent traces. This search is difficult because failures are often rare, complex, and sometimes even adversarially hidden and only detectable when multiple traces are analyzed together. These challenges arise in diverse settings such as misuse campaigns, covert sabotage, reward hacking, and
prompt injection. Existing approaches struggle here for several reasons. Per-trace judges miss failures that only become visible across traces, naive agentic
auditing does not scale to large trace collections, and fixed monitors are brittle to unanticipated behaviors. We introduce Meerkat, which combines clustering with agentic search to uncover violations specified in natural
language. Through structured search and adaptive investigation of promising regions, Meerkat finds sparse failures without relying on seed scenarios, fixed workflows, or exhaustive enumeration. Across misuse, misalignment, and task gaming settings, Meerkat significantly improves detection of safety violations over baseline monitors, discovers widespread developer cheating on a top
agent benchmark, and finds nearly 4x more examples of reward hacking on CyBench than previous audits.
Read the full abstract ▾
All authors Adam Stein, Davis Brown, Hamed Hassani, Mayur Naik, Eric Wong
Computation and Language
Saar-Voice: A Multi-Speaker Saarbrücken Dialect Speech Corpus
Lena S. Oberkircher, Jesujoba O. Alabi, Dietrich Klakow et al. (+1 more)
Natural
language processing (NLP) and
speech technologies have made significant progress in recent years; however, they remain largely focused on standardized
language varieties. Dialects, despite their cultural significance and widespread use, are underrepresented in linguistic resources and computational models, resulting in performance disparities. To address this gap, we introduce Saar-Voice, a six-hour
speech corpus for the Saarbrücken dialect of German. The
dataset was created by first collecting text through digitized books and locally sourced materials. A subset of this text was recorded by nine speakers, and we conducted analyses on both the textual and
speech components to assess the
dataset's characteristics and quality. We discuss methodological challenges related to orthographic and speaker variation, and explore grapheme-to-phoneme (G2P) conversion. The resulting corpus provides aligned textual and audio representations. This serves as a foundation for future research on dialect-aware text-to-
speech (TTS), particularly in low-resource scenarios, including zero-shot and few-shot model adaptation.
Read the full abstract ▾
All authors Lena S. Oberkircher, Jesujoba O. Alabi, Dietrich Klakow, Jürgen Trouvain
Computation and Language
Psychological Concept Neurons: Can Neural Control Bias Probing and Shift Generation in LLMs?
Yuto Harada and Hiro Taiyo Hamada
Using psychological constructs such as the Big Five, large
language models (LLMs) can imitate specific personality profiles and predict a user's personality. While LLMs can exhibit behaviors consistent with these constructs, it remains unclear where and how they are represented inside the model and how they relate to behavioral outputs. To address this gap, we focus on questionnaire-operationalized Big Five concepts, analyze the formation and localization of their internal representations, and use interventions to examine how these representations relate to behavioral outputs. In our experiment, we first use probing to examine where Big Five information emerges across model depth. We then identify
neurons that respond selectively to each Big Five concept and test whether enhancing or suppressing their activations can bias latent representations and label generation in intended directions. We find that Big Five information becomes rapidly decodable in early layers and remains detectable through the final layers, while concept-selective
neurons are most prevalent in mid layers and exhibit limited overlap across domains. Interventions on these
neurons consistently shift probe readouts toward targeted concepts, with targeted success rates exceeding 0.8 for some concepts, indicating that the model's internal separation of Big Five personality traits can be causally steered. At the label-generation level, the same interventions often bias generated label distributions in the intended directions, but the effects are weaker, more concept-dependent, and often accompanied by cross-trait spillover, indicating that comparable control over generated labels is difficult even with interventions on a large fraction of concept-selective
neurons. Overall, our findings reveal a gap between representational control and behavioral control in LLMs.
Read the full abstract ▾
Computation and Language
CLSGen: A Dual-Head Fine-Tuning Framework for Joint Probabilistic Classification and Verbalized Explanation
WonJin Yoon, Kangyu Zhu, Ian Bulovic et al. (+5 more)
With the recent progress of Large
Language Models (LLMs), there is a growing interest in applying these models to solve complex and challenging problems. Modern LLMs, capable of processing long contexts and generating verbalized explanations, offer significant potential in addressing real-world applications. However, a critical hurdle in deploying LLMs for practical decision-making is their inability to provide reliable, quantitative probabilities. While task-specific fine-tuning of LLMs using traditional discriminative objectives (similar to encoder-only models) can yield probability estimates, this often leads to catastrophic forgetting and linguistic collapse. Consequently, the model loses its ability to generate explanations, severely undermining its interpretability and usability. To address this challenge, we propose CLSGen, a novel LLM fine-tuning framework designed for binary classification tasks. The CLSGen framework encompasses a new model architecture, training methodology, and data construction strategy to enable robust probability estimation without sacrificing the model's inherent explanation-generation capabilities. Experimental results across multiple
benchmark datasets demonstrate that models fine-tuned with CLSGen outperform existing baselines in classification metrics (AUROC and F1-score). Regarding explanation, the results showed strong
alignment between predicted labels and generated justifications, as well as high readability.
Read the full abstract ▾
All authors WonJin Yoon, Kangyu Zhu, Ian Bulovic, Autumn Sehy, Yanjun Gao, Dmitriy Dligach, Majid Afshar, Timothy A. Miller
Computation and Language
C-ReD: A Comprehensive Chinese Benchmark for AI-Generated Text Detection Derived from Real-World Prompts
Chenxi Qing, Junxi Wu, Zheng Liu et al. (+5 more)
Recently, large
language models (LLMs) are capable of generating highly fluent textual content. While they offer significant convenience to humans, they also introduce various risks, like phishing and academic dishonesty. Numerous research efforts have been dedicated to developing
algorithms for detecting AI-generated text and constructing relevant
datasets. However, in the domain of Chinese corpora, challenges remain, including limited model diversity and data homogeneity. To address these issues, we propose C-ReD: a comprehensive Chinese Real-
prompt AI-generated Detection
benchmark. Experiments demonstrate that C-ReD not only enables reliable in-domain detection but also supports strong generalization to unseen LLMs and external Chinese
datasets-addressing critical gaps in model diversity, domain coverage, and
prompt realism that have limited prior Chinese detection
benchmarks. We release our resources at https://github.com/HeraldofLight/C-ReD.
Read the full abstract ▾
All authors Chenxi Qing, Junxi Wu, Zheng Liu, Yixiang Qiu, Hongyao Yu, Bin Chen, Hao Wu, Shu-Tao Xia
Computation and Language · Cross-listed from cs.LG
ClawGUI: A Unified Framework for Training, Evaluating, and Deploying GUI Agents
Fei Tang, Zhiqiong Lu, Boxuan Zhang et al. (+4 more)
GUI
agents drive applications through their visual interfaces instead of programmatic APIs, interacting with arbitrary software via taps, swipes, and keystrokes, reaching a long tail of applications that CLI-based
agents cannot. Yet progress in this area is bottlenecked less by modeling capacity than by the absence of a coherent full-stack infrastructure: online RL training suffers from environment instability and closed pipelines, evaluation protocols drift silently across works, and trained
agents rarely reach real users on real devices. We present ClawGUI, an open-source framework addressing these three gaps within a single harness. ClawGUI-RL provides the first open-source GUI
agent RL infrastructure with validated support for both parallel virtual environments and real physical devices, integrating GiGPO with a Process Reward Model for dense step-level supervision. ClawGUI-Eval enforces a fully standardized evaluation pipeline across 6
benchmarks and 11+ models, achieving 95.8% reproduction against official baselines. ClawGUI-
Agent brings trained
agents to Android, HarmonyOS, and iOS through 12+ chat platforms with hybrid CLI-GUI control and persistent personalized
memory. Trained end to end within this pipeline, ClawGUI-2B achieves 17.1% Success Rate on MobileWorld GUI-Only, outperforming the same-scale MAI-UI-2B baseline by 6.0%.
Read the full abstract ▾
All authors Fei Tang, Zhiqiong Lu, Boxuan Zhang, Weiming Lu, Jun Xiao, Yueting Zhuang, Yongliang Shen
Computation and Language
General365: Benchmarking General Reasoning in Large Language Models Across Diverse and Challenging Tasks
Junlin Liu, Shengnan An, Shuang Zhou et al. (+10 more)
Contemporary large
language models (LLMs) have demonstrated remarkable
reasoning capabilities, particularly in specialized domains like mathematics and physics. However, their ability to generalize these
reasoning skills to more general and broader contexts–often termed general
reasoning–remains under-explored. Unlike domain-specific
reasoning, general
reasoning relies less on expert knowledge but still presents formidable
reasoning challenges, such as complex constraints, nested logical branches, and semantic interference. To address this gap, we introduce General365, a
benchmark specifically designed to assess general
reasoning in LLMs. By restricting background knowledge to a K-12 level, General365 explicitly decouples
reasoning from specialized expertise. The
benchmark comprises 365 seed problems and 1,095 variant problems across eight categories, ensuring both high difficulty and diversity. Evaluations across 26 leading LLMs reveal that even the top-performing model achieves only 62.8% accuracy, in stark contrast to the near-perfect performances of LLMs in math and physics
benchmarks. These results suggest that the
reasoning abilities of current LLMs are heavily domain-dependent, leaving significant room for improvement in broader applications. We envision General365 as a catalyst for advancing LLM
reasoning beyond domain-specific tasks toward robust, general-purpose real-world scenarios. Code,
Dataset, and Leaderboard: https://general365.github.io
Read the full abstract ▾
All authors Junlin Liu, Shengnan An, Shuang Zhou, Dan Ma, Shixiong Luo, Ying Xie, Yuan Zhang, Wenling Yuan, Yifan Zhou, Xiaoyu Li, Ziwen Wang, Xuezhi Cao, Xunliang Cai
Computation and Language
Agentic Aggregation for Parallel Scaling of Long-Horizon Agentic Tasks
Yoonsang Lee, Howard Yen, Xi Ye et al. (+1 more)
We study parallel test-time scaling for long-horizon agentic tasks such as agentic search and deep research, where multiple rollouts are generated in parallel and aggregated into a final response. While such scaling has proven effective for chain-of-thought
reasoning, agentic tasks pose unique challenges: trajectories are long, multi-turn, and tool-augmented, and outputs are often open-ended. Aggregating only final answers discards rich information from trajectories, while concatenating all trajectories exceeds the model's context window. To address this, we propose AggAgent, an aggregation
agent that treats parallel trajectories as an environment. We equip it with lightweight tools to inspect candidate solutions and search across trajectories, enabling it to navigate and synthesize information on demand. Across six
benchmarks and three model families (GLM-4.7, Qwen3.5, MiniMax-M2.5), AggAgent outperforms all existing aggregation methods-by up to 5.3% absolute on average and 10.3% on two deep research tasks-while adding minimal overhead, as the aggregation cost remains bounded by a single agentic rollout. Our findings establish agentic aggregation as an effective and cost-efficient approach to parallel test-time scaling.
Read the full abstract ▾
All authors Yoonsang Lee, Howard Yen, Xi Ye, Danqi Chen
Computation and Language
HistLens: Mapping Idea Change across Concepts and Corpora
Yi Jing, Weiyun Qiu, Yihang Peng et al. (+1 more)
Language change both reflects and shapes social processes, and the semantic
evolution of foundational concepts provides a measurable trace of historical and social transformation. Despite recent advances in diachronic semantics and discourse analysis, existing computational approaches often (i) concentrate on a single concept or a single corpus, making findings difficult to compare across heterogeneous sources, and (ii) remain confined to surface lexical evidence, offering insufficient computational and interpretive granularity when concepts are expressed implicitly. We propose HistLens, a unified, SAE-based framework for multi-concept, multi-corpus conceptual-history analysis. The framework decomposes concept representations into interpretable features and tracks their activation dynamics over time and across sources, yielding comparable conceptual trajectories within a shared coordinate system. Experiments on long-span press corpora show that HistLens supports cross-concept, cross-corpus computation of patterns of idea
evolution and enables implicit concept computation. By bridging conceptual modeling with interpretive needs, HistLens broadens the analytical perspectives and methodological repertoire available to social science and the humanities for diachronic text analysis.
Read the full abstract ▾
All authors Yi Jing, Weiyun Qiu, Yihang Peng, Zhifang Sui
Computation and Language
LangFlow: Continuous Diffusion Rivals Discrete in Language Modeling
Yuxin Chen, Chumeng Liang, Hangke Sui et al. (+4 more)
Continuous
diffusion has been the foundation of high-fidelity, controllable, and few-step generation of many data modalities such as images. However, in
language modeling, prior continuous
diffusion language models (DLMs) lag behind discrete counterparts due to the sparse data space and the underexplored design space. In this work, we close this gap with LangFlow, the first continuous DLM to rival discrete
diffusion, by connecting
embedding-space DLMs to Flow Matching via Bregman divergence, alongside three key innovations: (1) we derive a novel ODE-based NLL bound for principled evaluation of continuous flow-based
language models; (2) we propose an information-uniform principle for setting the noise schedule, which motivates a learnable noise scheduler based on a Gumbel distribution; and (3) we revise prior training protocols by incorporating self-conditioning, as we find it improves both likelihood and sample quality of
embedding-space DLMs with effects substantially different from discrete
diffusion. Putting everything together, LangFlow rivals top discrete DLMs on both the perplexity (PPL) and the generative perplexity (Gen. PPL), reaching a PPL of 30.0 on LM1B and 24.6 on OpenWebText. It even exceeds autoregressive baselines in zero-shot transfer on 4 out of 7
benchmarks. LangFlow provides the first clear evidence that continuous
diffusion is a promising paradigm for
language modeling. Homepage: https://github.com/nealchen2003/LangFlow
Read the full abstract ▾
All authors Yuxin Chen, Chumeng Liang, Hangke Sui, Ruihan Guo, Chaoran Cheng, Jiaxuan You, Ge Liu
Computation and Language
Discourse Diversity in Multi-Turn Empathic Dialogue
Hongli Zhan, Emma S. Gueorguieva, Javier Hernandez et al. (+3 more)
Large
language models (LLMs) produce responses rated as highly empathic in single-turn settings (Ayers et al., 2023; Lee et al., 2024), yet they are also known to be formulaic generators that reuse the same lexical patterns, syntactic templates, and discourse structures across tasks (Jiang et al., 2025; Shaib et al., 2024; Namuduri et al., 2025). Less
attention has been paid to whether this formulaicity extends to the level of discourse moves, i.e., what a response does for the person it is addressing. This question is especially consequential for empathic dialogue, where effective support demands not just a kind response at one moment but varied strategies as a conversation unfolds (Stiles et al., 1998). Indeed, prior work shows that LLMs reuse the same tactic sequences more than human supporters in single-turn settings (Gueorguieva et al., 2026). We extend this analysis to multi-turn conversations and find that the
rigidity compounds: once a tactic appears in a supporter turn, LLMs reuse it in the next at nearly double the rate of humans (0.50-0.56 vs. 0.27). This pattern holds across LLMs serving as supporters in real emotional support conversations, and is invisible to standard similarity metrics. To address this gap, we introduce MINT (Multi-turn Inter-tactic Novelty Training), the first reinforcement learning framework to optimize discourse move diversity across multi-turn empathic dialogue. The best MINT variant combines an empathy quality reward with a cross-turn tactic novelty signal, improving aggregate empathy by 25.3% over vanilla across 1.7B and 4B models while reducing cross-turn discourse move repetition by 26.3% on the 4B model, surpassing all baselines including quality-only and
token-level diversity methods on both measures. These results suggest that what current models lack is not empathy itself, but the ability to vary their discourse moves across a conversation.
Read the full abstract ▾
All authors Hongli Zhan, Emma S. Gueorguieva, Javier Hernandez, Jina Suh, Desmond C. Ong, Junyi Jessy Li
Computation and Language
Evaluating Cooperation in LLM Social Groups through Elected Leadership
Ryan Faulkner, Anushka Deshpande, David Guzman Piedrahita et al. (+2 more)
Governing common-pool resources requires
agents to develop enduring strategies through cooperation and self-governance to avoid collective failure. While foundation models have shown potential for cooperation in these settings, existing multi-
agent research provides little insight into whether structured leadership and election mechanisms can improve collective decision making. The lack of such a critical organizational feature ubiquitous in human society presents a significant shortcoming of the current methods. In this work we aim to directly address whether leadership and elections can support improved social welfare and cooperation through multi-
agent simulation with LLMs. We present our open-source framework that simulates leadership through elected personas and candidate-driven agendas and carry out an empirical study of LLMs under controlled governance conditions. Our experiments demonstrate that having elected leadership improves social welfare scores by 55.4% and survival time by 128.6% across a range of high performing LLMs. Through the construction of an
agent social graph we compute centrality metrics to assess the social influence of leader personas and also analyze rhetorical and cooperative tendencies revealed through a sentiment analysis on leader utterances. This work lays the foundation for further study of election mechanisms in multi-
agent systems toward navigating complex social dilemmas.
Read the full abstract ▾
All authors Ryan Faulkner, Anushka Deshpande, David Guzman Piedrahita, Joel Z. Leibo, Zhijing Jin
Computation and Language · Cross-listed from cs.AI
SWE-AGILE: A Software Agent Framework for Efficiently Managing Dynamic Reasoning Context
Shuquan Lian, Juncheng Liu, Yazhe Chen et al. (+2 more)
Prior representative ReAct-style approaches in autonomous Software Engineering (SWE) typically lack the explicit System-2
reasoning required for deep analysis and handling complex edge cases. While recent
reasoning models demonstrate the potential of extended Chain-of-Thought (CoT), applying them to the multi-turn SWE task creates a fundamental dilemma: retaining full
reasoning history leads to context explosion and “Lost-in-the-Middle” degradation, while discarding it would force the
agent to redundantly re-reason at every step. To address these challenges, we propose SWE-AGILE, a novel software
agent framework designed to bridge the gap between
reasoning depth, efficiency, and context constraints. SWE-AGILE introduces a Dynamic
Reasoning Context strategy, maintaining a “sliding window” of detailed
reasoning for immediate continuity to prevent redundant re-analyzing, while compressing historical
reasoning content into concise
Reasoning Digests. Empirically, SWE-AGILE sets a new standard for 7B-8B models on SWE-Bench-Verified using only 2.2k trajectories and 896 tasks. Code is available at https://github.com/KDEGroup/SWE-AGILE.
Read the full abstract ▾
All authors Shuquan Lian, Juncheng Liu, Yazhe Chen, Yuhong Chen, Hui Li
Computation and Language · Cross-listed from cs.AI
Agentic Driving Coach: Robustness and Determinism of Agentic AI-Powered Human-in-the-Loop Cyber-Physical Systems
Deeksha Prahlad, Daniel Fan and Hokeun Kim
Foundation models, including large
language models (LLMs), are increasingly used for human-in-the-loop (HITL) cyber-physical systems (CPS) because foundation model-based AI
agents can potentially interact with both the physical environments and human users. However, the unpredictable behavior of human users and AI
agents, in addition to the dynamically changing physical environments, leads to uncontrollable nondeterminism. To address this urgent challenge of enabling agentic AI-powered HITL CPS, we propose a reactor-model-of-computation (MoC)-based approach, realized by the open-source Lingua Franca (LF) framework. We also carry out a concrete case study using the agentic driving coach as an application of HITL CPS. By evaluating the LF-based agentic HITL CPS, we identify practical challenges in reintroducing determinism into such agentic HITL CPS and present pathways to address them.
Read the full abstract ▾
Computation and Language
Legal2LogicICL: Improving Generalization in Transforming Legal Cases to Logical Formulas via Diverse Few-Shot Learning
Jieying Xue, Phuong Minh Nguyen, Ha Thanh Nguyen et al. (+2 more)
This work aims to improve the generalization of logic-based legal
reasoning systems by integrating recent advances in NLP with legal-domain adaptive few-shot learning techniques using LLMs. Existing logic-based legal
reasoning pipelines typically rely on fine-tuned models to map natural-
language legal cases into logical formulas before forwarding them to a symbolic reasoner. However, such approaches are heavily constrained by the scarcity of high-quality annotated training data. To address this limitation, we propose a novel LLM-based legal
reasoning framework that enables effective in-context learning through
retrieval-augmented generation. Specifically, we introduce Legal2LogicICL, a few-shot
retrieval framework that balances diversity and similarity of exemplars at both the latent semantic representation level and the legal text structure level. In addition, our method explicitly accounts for legal structure by mitigating entity-induced
retrieval bias in legal texts, where lengthy and highly specific entity mentions often dominate semantic representations and obscure legally meaningful
reasoning patterns. Our Legal2LogicICL constructs informative and robust few-shot demonstrations, leading to accurate and stable logical rule generation without requiring additional training. In addition, we construct a new
dataset, named Legal2Proleg, which is annotated with
alignments between legal cases and PROLEG logical formulas to support the evaluation of legal semantic parsing. Experimental results on both open-source and proprietary LLMs demonstrate that our approach significantly improves accuracy, stability, and generalization in transforming natural-
language legal case descriptions into logical representations, highlighting its effectiveness for interpretable and reliable legal
reasoning. Our code is available at https://github.com/yingjie7/Legal2LogicICL.
Read the full abstract ▾
All authors Jieying Xue, Phuong Minh Nguyen, Ha Thanh Nguyen, May Myo Zin, Ken Satoh
Computation and Language
Please Make it Sound like Human: Encoder-Decoder vs. Decoder-Only Transformers for AI-to-Human Text Style Transfer
Utsav Paneru
AI-generated text has become common in academic and professional writing, prompting research into detection methods. Less studied is the reverse: systematically rewriting AI-generated prose to read as genuinely human-authored. We build a parallel corpus of 25,140 paired AI-input and human-reference text chunks, identify 11 measurable stylistic markers separating the two registers, and fine-tune three models: BART-base, BART-large, and Mistral-7B-Instruct with QLoRA. BART-large achieves the highest reference similarity – BERTScore F1 of 0.924, ROUGE-L of 0.566, and chrF++ of 55.92 – with 17x fewer parameters than Mistral-7B. We show that Mistral-7B's higher marker shift score reflects overshoot rather than accuracy, and argue that shift accuracy is a meaningful blind spot in current style transfer evaluation.
Read the full abstract ▾
Computation and Language
Playing Along: Learning a Double-Agent Defender for Belief Steering via Theory of Mind
Hanqi Xiao, Vaidehi Patil, Zaid Khan et al. (+3 more)
As large
language models (LLMs) become the engine behind conversational systems, their ability to reason about the intentions and states of their dialogue partners (i.e., form and use a theory-of-mind, or ToM) becomes increasingly critical for safe interaction with potentially
adversarial partners. We propose a novel
privacy-themed ToM challenge, ToM for Steering Beliefs (ToM-SB), in which a defender must act as a Double
Agent to steer the beliefs of an attacker with partial prior knowledge within a shared universe. To succeed on ToM-SB, the defender must engage with and form a ToM of the attacker, with a goal of fooling the attacker into believing they have succeeded in extracting sensitive information. We find that strong frontier models like Gemini3-Pro and GPT-5.4 struggle on ToM-SB, often failing to fool attackers in hard scenarios with partial attacker prior knowledge, even when prompted to reason about the attacker's beliefs (ToM prompting). To close this gap, we train models on ToM-SB to act as AI Double
Agents using reinforcement learning, testing both fooling and ToM rewards. Notably, we find a bidirectionally emergent relationship between ToM and attacker-fooling: rewarding fooling success alone improves ToM, and rewarding ToM alone improves fooling. Across four attackers with different strengths, six defender methods, and both in-distribution and out-of-distribution (OOD) evaluation, we find that gains in ToM and attacker-fooling are well-correlated, highlighting belief modeling as a key driver of success on ToM-SB. AI Double
Agents that combine both ToM and fooling rewards yield the strongest fooling and ToM performance, outperforming Gemini3-Pro and GPT-5.4 with ToM prompting on hard scenarios. We also show that ToM-SB and AI Double
Agents can be extended to stronger attackers, demonstrating generalization to OOD settings and the upgradability of our task.
Read the full abstract ▾
All authors Hanqi Xiao, Vaidehi Patil, Zaid Khan, Hyunji Lee, Elias Stengel-Eskin, Mohit Bansal
Computation and Language
A Robust Evaluation of Probe Robustness: Lessons for Reliable OOD Uncertainty Quantification
Joe Stacey, Hadas Orgad, Kentaro Inui et al. (+2 more)
Recent work has shown that the hidden states of large
language models contain signals useful for uncertainty estimation, motivating a growing interest in efficient probe-based approaches. Yet it remains unclear how robust existing methods are, with prior work reporting conflicting conclusions under substantially different evaluation settings. We address this by introducing ProbeDrift, a systematic evaluation framework for supervised uncertainty probes covering a wide range of OOD settings across models, tasks, and distributional shifts. Using ProbeDrift, we train over 2,000 probes to disentangle the effect of key design choices, showing poor robustness of current methods beyond near-OOD settings. We find that robustness is driven by design decisions that have a largely invisible effect in-distribution, including the choice of feature type, aggregation strategy, and training signal. We argue that robust uncertainty estimation requires robust evaluation. To support this, we release ProbeDrift as a lightweight Python library that contains the train and test splits underpinning our extensive evaluation. We also show how insights from our evaluation can directly lead to more robust methods through a simple Hybrid Back-Off (HBO) strategy.
Read the full abstract ▾
All authors Joe Stacey, Hadas Orgad, Kentaro Inui, Benjamin Heinzerling, Nafise Sadat Moosavi
Computation and Language
RPA-Check: A Multi-Stage Automated Framework for Evaluating Dynamic LLM-based Role-Playing Agents
Riccardo Rosati, Edoardo Colucci, Massimiliano Bolognini et al. (+2 more)
The rapid adoption of Large
Language Models (LLMs) in interactive systems has enabled the creation of dynamic, open-ended Role-Playing
Agents (RPAs). However, evaluating these
agents remains a significant challenge, as standard NLP metrics fail to capture the nuances of role adherence, logical consistency, and long-term narrative stability. This paper introduces RPA-Check, a multi-stage automated evaluation framework designed to objectively assess the performance of LLM-based RPAs in complex, constraints-heavy environments. Our methodology is based on a four-step pipeline: (1) Dimension Definition, establishing high-level qualitative behavioral criteria; (2) Augmentation, where these requirements are expanded into granular boolean checklist indicators; (3) Semantic Filtering, to ensure indicator objectivity, no redundancy and
agent isolation; and (4) LLM-as-a-Judge Evaluation, which employs chain-of-thought verification to score
agent fidelity. We validate this framework by applying it to LLM Court, a serious game for forensic training involving several quantized local models. Experimental results across five distinct legal scenarios demonstrate the framework's ability to identify subtle trade-offs between model size,
reasoning depth, and operational stability. Notably, the findings reveal an inverse relationship between parametric scale and procedural consistency, showing that smaller, adequately instruction-tuned models (8-9B) can outperform larger architectures prone to user-
alignment bias or sycophancy. RPA-Check thus provides a standardized and reproducible metric for future research in generative
agent evaluation within specialized domains.
Read the full abstract ▾
All authors Riccardo Rosati, Edoardo Colucci, Massimiliano Bolognini, Adriano Mancini, Paolo Sernani
Computation and Language
CArtBench: Evaluating Vision-Language Models on Chinese Art Understanding, Interpretation, and Authenticity
Xuefeng Wei, Zhixuan Wang, Xuan Zhou et al. (+5 more)
We introduce CARTBENCH, a museum-grounded
benchmark for evaluating
vision-
language models (VLMs) on Chinese artworks beyond short-form recognition and QA. CARTBENCH comprises four subtasks: CURATORQA for evidence-grounded recognition and
reasoning, CATALOGCAPTION for structured four-section expert-style appreciation, REINTERPRET for defensible reinterpretation with expert ratings, and CONNOISSEURPAIRS for diagnostic authenticity discrimination under visually similar confounds. CARTBENCH is built by aligning image-bearing Palace Museum objects from Wikidata with authoritative catalog pages, spanning five art categories across multiple dynasties. Across nine representative VLMs, we find that high overall CURATORQA accuracy can mask sharp drops on hard evidence linking and style-to-period inference; long-form appreciation remains far from expert references; and authenticity-oriented diagnostic discrimination stays near chance, underscoring the difficulty of connoisseur-level
reasoning for current models.
Read the full abstract ▾
All authors Xuefeng Wei, Zhixuan Wang, Xuan Zhou, Zhi Qu, Hongyao Li, Yusuke Sakai, Hidetaka Kamigaito, Taro Watanabe
Computation and Language
Back to Basics: Let Conversational Agents Remember with Just Retrieval and Generation
Yuqian Wu, Wei Chen, Zhengjun Huang et al. (+5 more)
Existing conversational
memory systems rely on complex hierarchical summarization or reinforcement learning to manage long-term dialogue history, yet remain vulnerable to context dilution as conversations grow. In this work, we offer a different perspective: the primary bottleneck may lie not in
memory architecture, but in the Signal Sparsity Effect within the latent knowledge
manifold. Through controlled experiments, we identify two key phenomena: Decisive Evidence Sparsity, where relevant signals become increasingly isolated with longer sessions, leading to sharp degradation in aggregation-based methods; and Dual-Level Redundancy, where both inter-session interference and intra-session conversational filler introduce large amounts of non-informative content, hindering effective generation. Motivated by these insights, we propose \method, a minimalist framework that brings conversational
memory back to basics, relying solely on
retrieval and generation via Turn Isolation
Retrieval (TIR) and Query-Driven
Pruning (QDP). TIR replaces global aggregation with a max-activation strategy to capture turn-level signals, while QDP removes redundant sessions and conversational filler to construct a compact, high-density evidence set. Extensive experiments on multiple
benchmarks demonstrate that \method achieves robust performance across diverse settings, consistently outperforming strong baselines while maintaining high efficiency in
tokens and latency, establishing a new minimalist baseline for conversational
memory.
Read the full abstract ▾
All authors Yuqian Wu, Wei Chen, Zhengjun Huang, Junle Chen, Qingxiang Liu, Kai Wang, Xiaofang Zhou, Yuxuan Liang
Computation and Language
Utilizing and Calibrating Hindsight Process Rewards via Reinforcement with Mutual Information Self-Evaluation
Jiashu Yao, Heyan Huang, Zeming Liu et al. (+1 more)
To overcome the sparse reward challenge in reinforcement learning (RL) for
agents based on large
language models (LLMs), we propose Mutual Information Self-Evaluation (MISE), an RL paradigm that utilizes hindsight generative self-evaluation as dense reward signals while simultaneously calibrating them against the environmental feedbacks. Empirically, MISE enables an
agent to learn autonomously from dense internal rewards supplementing sparse extrinsic signals. Theoretically, our work provides the first formal foundation for the paradigm of generative self-rewarding. We prove that utilizing hindsight self-evaluation rewards is equivalent to minimizing an objective that combines mutual information with a KL divergence term between the policy and a proxy reward policy. This theoretical insight then informs and justifies our calibration step, which actively aligns these rewards with the optimal policy. Extensive experiments show that MISE outperforms strong baselines, enabling open-source LLMs about 7B parameters to achieve performance comparable to GPT-4o on validation without expert supervision.
Read the full abstract ▾
All authors Jiashu Yao, Heyan Huang, Zeming Liu, Yuhang Guo