Computation and Language
Measuring Inclusion in Interaction: Inclusion Analytics for Human-AI Collaborative Learning
Jaeyoon Choi and Nia Nixon
Inclusion, equity, and access are widely valued in AI and education, yet are often assessed through coarse sample descriptors or post-hoc self-reports that miss how inclusion is shaped moment by moment in collaborative problem solving (CPS). In this proof-of-concept paper, we introduce inclusion analytics, a discourse-based framework for examining inclusion as a dynamic, interactional process in CPS. We conceptualize inclusion along three complementary dimensions – participation equity, affective
climate, and epistemic equity – and demonstrate how these constructs can be made analytically visible using scalable, interaction-level measures. Using both simulated conversations and empirical data from human-AI teaming experiments, we illustrate how inclusion analytics can surface patterns of participation, relational dynamics, and idea uptake that remain invisible to aggregate or post-hoc evaluations. This work represents an initial step toward process-oriented approaches to measuring inclusion in human-AI collaborative learning environments.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.AI
FlyAOC: Evaluating Agentic Ontology Curation of Drosophila Scientific Knowledge Bases
Xingjian Zhang, Sophia Moylan, Ziyang Xiong et al. (+3 more)
Scientific knowledge bases accelerate discovery by curating findings from primary literature into structured, queryable formats for both human researchers and emerging AI systems. Maintaining these resources requires expert curators to search papers, reconcile evidence across documents, and produce ontology-grounded annotations. Existing
benchmarks usually evaluate isolated subtasks, such as named entity recognition or relation extraction, and therefore do not capture this end-to-end workflow. We present FlyAOC to evaluate AI
agents on end-to-end agentic ontology curation from scientific literature. Given a gene symbol, a concise FlyBase gene description, access to a 16,898-paper corpus, and ontology resources,
agents must search for evidence and recover as many curator-relevant structured annotations as possible. Outputs span standardized function terms, expression patterns, and historical synonyms linking decades of nomenclature. The
benchmark includes 7,397 expert-curated annotations across 100 genes drawn from FlyBase, the Drosophila knowledge base. Across four baseline
agent harnesses—memorization, fixed pipeline, single-
agent, and multi-
agent—FlyAOC is sensitive to harness design, model family, and tool-use reliability. These results reveal system-level failure modes that model-only evaluations do not capture. FlyAOC provides a reproducible testbed for
retrieval-augmented scientific curation.
Read the full abstract ▾
All authors Xingjian Zhang, Sophia Moylan, Ziyang Xiong, Qiaozhu Mei, Yichen Luo, Jiaqi W. Ma
Computation and Language
Overview of PAN 2026: Voight-Kampff Generative AI Detection, Text Watermarking, Multi-Author Writing Style Analysis, Generative Plagiarism Detection, and Reasoning Trajectory Detection
Janek Bevendorff, Maik Fröbe, André Greiner-Petter et al. (+9 more)
The goal of the PAN workshop is to advance computational stylometry and text forensics via objective and reproducible evaluation. In 2026, we run the following five tasks: (1) Voight-Kampff Generative AI Detection, particularly in mixed and obfuscated authorship scenarios, (2) Text Watermarking, a new task that aims to find new and
benchmark the robustness of existing text watermarking schemes, (3) Multi-author Writing Style Analysis, a continued task that aims to find positions of authorship change, (4) Generative Plagiarism Detection, a continued task that targets source
retrieval and text
alignment between generated text and source documents, and (5)
Reasoning Trajectory Detection, a new task that deals with source detection and safety detection of LLM-generated or human-written
reasoning trajectories. As in previous years, PAN invites software submissions as easy-to-reproduce Docker containers for most of the tasks. Since PAN 2012, more than 1,100 submissions have been made this way via the TIRA experimentation platform.
Read the full abstract ▾
All authors Janek Bevendorff, Maik Fröbe, André Greiner-Petter, Andreas Jakoby, Maximilian Mayerl, Preslav Nakov, Henry Plutz, Martin Potthast, Benno Stein, Minh Ngoc Ta, Yuxia Wang, Eva Zangerle
Computation and Language · Cross-listed from cs.AI
PABU: Progress-Aware Belief Update for Efficient LLM Agents
Haitao Jiang, Lin Ge, Hengrui Cai et al. (+1 more)
Large
Language Model (LLM)
agents commonly condition actions on full action-observation histories, which introduce task-irrelevant information that easily leads to redundant actions and higher inference cost. We propose Progress-Aware Belief Update (PABU), a belief-state framework that compactly represents an
agent's state by explicitly modeling task progress and selectively retaining past actions and observations. At each step, the
agent predicts its relative progress since the previous round and decides whether the newly encountered interaction should be stored, conditioning future decisions only on the retained subset. Across eight environments in the AgentGym
benchmark, and using identical training trajectories, PABU achieves an 81.0% task completion rate, outperforming previous State of the art (SoTA) models with full-history belief by 23.9%. Additionally, PABU's progress-oriented action selection improves efficiency, reducing the average number of interaction steps to 9.5, corresponding to a 26.9% reduction. Ablation studies show that both explicit progress prediction and selective retention are necessary for robust belief learning and performance gains.
Read the full abstract ▾
All authors Haitao Jiang, Lin Ge, Hengrui Cai, Rui Song
Computation and Language · Cross-listed from cs.LG
Benchmarking the Energy Savings with Speculative Decoding Strategies
Rohit Dutta, Paramita Koley, Soham Poddar et al. (+5 more)
Speculative decoding has emerged as an effective method to reduce latency and inference cost of LLM inferences. However, there has been inadequate
attention towards the energy requirements of these models. To address this gap, this paper presents a comprehensive survey of energy requirements of speculative decoding strategies, with detailed analysis on how various factors – model size and family, speculative decoding strategies, and
dataset characteristics – influence the energy optimizations.
Read the full abstract ▾
All authors Rohit Dutta, Paramita Koley, Soham Poddar, Janardan Misra, Sanjay Podder, Naveen Balani, Saptarshi Ghosh, Niloy Ganguly
Computation and Language · Cross-listed from cs.CV
UI-Venus-1.5 Technical Report
Venus Team, Changlong Gao, Zhangxuan Gu et al. (+24 more)
GUI
agents have emerged as a powerful paradigm for automating interactions in digital environments, yet achieving both broad generality and consistently strong task performance remains challenging. In this report, we present UI-Venus-1.5, a unified, end-to-end GUI
Agent designed for robust real-world applications. The proposed model family comprises two dense variants (2B and 8B) and one mixture-of-experts variant (30B-A3B) to meet various downstream application scenarios. Compared to our previous version, UI-Venus-1.5 introduces three key technical advances: (1) a comprehensive Mid-Training stage leveraging 10 billion
tokens across 30+
datasets to establish foundational GUI semantics; (2) Online Reinforcement Learning with full-trajectory rollouts, aligning training objectives with long-horizon, dynamic
navigation in large-scale environments; and (3) a single unified GUI
Agent constructed via Model Merging, which synthesizes domain-specific models (grounding, web, and mobile) into one cohesive checkpoint. Extensive evaluations demonstrate that UI-Venus-1.5 establishes new state-of-the-art performance on
benchmarks such as ScreenSpot-Pro (69.6%), VenusBench-GD (75.0%), and AndroidWorld (77.6%), significantly outperforming previous strong baselines. In addition, UI-Venus-1.5 demonstrates robust
navigation capabilities across a variety of Chinese mobile apps, effectively executing user instructions in real-world scenarios. Code: https://github.com/inclusionAI/UI-Venus; Model: https://huggingface.co/collections/inclusionAI/ui-venus
Read the full abstract ▾
All authors Venus Team, Changlong Gao, Zhangxuan Gu, Yulin Liu, Xinyu Qiu, Shuheng Shen, Yue Wen, Tianyu Xia, Zhenyu Xu, Zhengwen Zeng, Beitong Zhou, Xingran Zhou, Weizhi Chen, Sunhao Dai, Jingya Dou, Yichen Gong, Yuan Guo, Zhenlin Guo, Feng Li, Qian Li, Jinzhen Lin, Yuqi Zhou, Linchao Zhu, Liang Chen, Zhenyu Guo, Changhua Meng, Weiqiang Wang
Computation and Language · Cross-listed from q-bio.GN
AntigenLM: Structure-Aware DNA Language Modeling for Influenza
Yue Pei, Xuebin Chi and Yu Kang
Language models have advanced sequence analysis, yet DNA foundation models often lag behind task-specific methods for unclear reasons. We present AntigenLM, a generative DNA
language model pretrained on influenza genomes with intact, aligned functional units. This structure-aware pretraining enables AntigenLM to capture evolutionary constraints and generalize across tasks. Fine-tuned on time-series hemagglutinin (HA) and neuraminidase (NA) sequences, AntigenLM accurately forecasts future antigenic variants across regions and subtypes, including those unseen during training, outperforming phylogenetic and
evolution-based models. It also achieves near-perfect subtype classification. Ablation studies show that disrupting genomic structure through fragmentation or shuffling severely degrades performance, revealing the importance of preserving functional-unit integrity in DNA
language modeling. AntigenLM thus provides both a powerful framework for antigen
evolution prediction and a general principle for building biologically grounded DNA foundation models.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.LG
Next-Gen CAPTCHAs: Leveraging the Cognitive Gap for Scalable and Diverse GUI-Agent Defense
Jiacheng Liu, Yaxin Luo, Jiacheng Cui et al. (+3 more)
The rapid
evolution of GUI-enabled
agents has rendered traditional CAPTCHAs obsolete. While previous
benchmarks like OpenCaptchaWorld established a baseline for evaluating
multimodal agents, recent advancements in
reasoning-heavy models, such as Gemini3-Pro-High and GPT-5.2-Xhigh have effectively collapsed this security barrier, achieving pass rates as high as 90% on complex logic puzzles like "Bingo". In response, we introduce Next-Gen CAPTCHAs, a scalable defense framework designed to secure the next-generation web against the advanced
agents. Unlike static
datasets, our
benchmark is built upon a robust data generation pipeline, allowing for large-scale and easily scalable evaluations, notably, for backend-supported types, our system is capable of generating effectively unbounded CAPTCHA instances. We exploit the persistent human-
agent "Cognitive Gap" in interactive perception,
memory, decision-making, and action. By engineering dynamic tasks that require adaptive intuition rather than granular planning, we re-establish a robust distinction between biological users and artificial
agents, offering a scalable and diverse defense mechanism for the agentic era.
Read the full abstract ▾
All authors Jiacheng Liu, Yaxin Luo, Jiacheng Cui, Xinyi Shang, Xiaohan Zhao, Zhiqiang Shen
Computation and Language · Cross-listed from cs.AI
Data Science and Technology Towards AGI Part I: Tiered Data Management
Yudong Wang, Zixuan Fu, Hengyu Zhao et al. (+14 more)
The development of artificial intelligence can be viewed as an
evolution of data-driven learning paradigms, with successive shifts in data organization and utilization continuously driving advances in model capability. Current LLM research is dominated by a paradigm that relies heavily on unidirectional scaling of data size, increasingly encountering bottlenecks in data availability, acquisition cost, and training efficiency. In this work, we argue that the development of AGI is entering a new phase of data-model co-
evolution, in which models actively guide data management while high-quality data, in turn, amplifies model capabilities. To implement this
vision, we propose a tiered data management framework, designed to support the full LLM training lifecycle across heterogeneous learning objectives and cost constraints. Specifically, we introduce an L0-L4 tiered data management framework, ranging from raw uncurated resources to organized and verifiable knowledge. Importantly, LLMs are fully used in data management processes, such as quality scoring and content editing, to refine data across tiers. Each tier is characterized by distinct data properties, management strategies, and training roles, enabling data to be strategically allocated across LLM training stages, including pre-training, mid-training, and
alignment. The framework balances data quality, acquisition cost, and marginal training benefit, providing a systematic approach to scalable and sustainable data management. We validate the effectiveness of the proposed framework through empirical studies, in which tiered
datasets are constructed from raw corpora and used across multiple training phases. Experimental results demonstrate that tier-aware data utilization significantly improves training efficiency and model performance. To facilitate further research, we release our tiered
datasets and processing tools to the community.
Read the full abstract ▾
All authors Yudong Wang, Zixuan Fu, Hengyu Zhao, Chen Zhao, Chuyue Zhou, Xinle Lin, Hongya Lyu, Shuaikang Xue, Yi Yi, Yingjiao Wang, Zhi Zheng, Yuzhou Zhang, Jie Zhou, Chaojun Xiao, Xu Han, Zhiyuan Liu, Maosong Sun
Computation and Language · Cross-listed from cs.CY
Paradox of De-identification: A Critique of HIPAA Safe Harbour in the Age of LLMs
Lavender Y. Jiang, Xujin Chris Liu, Kyunghyun Cho et al. (+1 more)
Privacy is a human right that sustains patient-provider trust. Clinical notes capture a patient's private vulnerability and individuality, which are used for care coordination and research. Under HIPAA Safe Harbor, these notes are de-identified to protect patient
privacy. However, Safe Harbor was designed for an era of categorical tabular data, focusing on the removal of explicit identifiers while ignoring the latent information found in correlations between identity and quasi-identifiers, which can be captured by modern LLMs. We first formalize these correlations using a causal graph, then validate it empirically through individual re-identification of patients from scrubbed notes. The paradox of de-identification is further shown through a diagnosis ablation: even when all other information is removed, the model can predict the patient's neighborhood based on diagnosis alone. This position paper raises the question of how we can act as a community to uphold patient-provider trust when de-identification is inherently imperfect. We aim to raise awareness and discuss actionable recommendations.
Read the full abstract ▾
All authors Lavender Y. Jiang, Xujin Chris Liu, Kyunghyun Cho, Eric K. Oermann
Computation and Language
When Actions Go Off-Task: Detecting and Correcting Misaligned Actions in Computer-Use Agents
Yuting Ning, Jaylen Jones, Zhehao Zhang et al. (+5 more)
Computer-use
agents (CUAs) have made tremendous progress in the past year, yet they still frequently produce misaligned actions that deviate from the user's original intent. Such misaligned actions may arise from external attacks (e.g., indirect
prompt injection) or from internal limitations (e.g., erroneous
reasoning). They not only expose CUAs to safety risks, but also degrade task efficiency and reliability. This work makes the first effort to define and study misaligned action detection in CUAs, with comprehensive coverage of both externally induced and internally arising misaligned actions. We further identify three common categories in real-world CUA deployment and construct MisActBench, a
benchmark of realistic trajectories with human-annotated, action-level
alignment labels. Moreover, we propose DeAction, a practical and universal guardrail that detects misaligned actions before execution and iteratively corrects them through structured feedback. DeAction outperforms all existing baselines across offline and online evaluations with moderate latency overhead: (1) On MisActBench, it outperforms baselines by over 15% absolute in F1 score; (2) In online evaluation, it reduces attack success rate by over 90% under
adversarial settings while preserving or even improving task success rate in benign environments.
Read the full abstract ▾
All authors Yuting Ning, Jaylen Jones, Zhehao Zhang, Chentao Ye, Weitong Ruan, Junyi Li, Rahul Gupta, Huan Sun
Computation and Language
Next Concept Prediction in Discrete Latent Space Leads to Stronger Language Models
Yuliang Liu, Yunchong Song, Yixuan Wang et al. (+6 more)
We propose Next Concept Prediction (NCP), a generative pretraining paradigm built on top of Next
Token Prediction (NTP). NCP predicts discrete concepts that span multiple
tokens, thereby forming a more challenging pretraining objective. Our model, ConceptLM, quantizes hidden states using Vector Quantization and constructs a concept vocabulary. It leverages both NCP and NTP to drive parameter updates and generates a concept to guide the generation of the following
tokens. We train ConceptLM from scratch at scales ranging from 70M to 1.5B parameters with up to 300B training data, including Pythia and GPT-2 backbones. Results on 13
benchmarks show that NCP yields consistent performance gains over traditional
token-level models. Furthermore, continual pretraining experiments on an 8B-parameter Llama model indicate that NCP can further improve an NTP-trained model. Our analysis suggests that NCP leads to more powerful
language models by introducing a harder pretraining task, providing a promising path toward better
language modeling.
Read the full abstract ▾
All authors Yuliang Liu, Yunchong Song, Yixuan Wang, Kewen Ge, Alex Lamb, Qipeng Guo, Kai Chen, Bowen Zhou, Zhouhan Lin
Computation and Language · Cross-listed from cs.SD
Beyond Transcripts: A Renewed Perspective on Audio Chaptering
Fabian Retkowski, Maike Züfle, Thai Binh Nguyen et al. (+2 more)
Audio chaptering, the task of segmenting long-form audio into coherent sections, is increasingly important for navigating podcasts, lectures, and
videos. Despite its relevance, research remains limited and text-based, leaving key questions unresolved about leveraging audio information, handling ASR errors, and transcript-free evaluation. We address these gaps through three contributions: (1) a systematic comparison between text-based models with acoustic features, a novel audio-only architecture (AudioSeg) operating on learned audio representations, and
multimodal LLMs; (2) empirical analysis of factors affecting performance, including transcript quality, acoustic features, duration, and speaker composition; and (3) formalized evaluation protocols contrasting transcript-dependent text-space protocols with transcript-invariant time-space protocols. Our experiments on YTSeg reveal that AudioSeg substantially outperforms text-based approaches, pauses provide the largest acoustic gains, and MLLMs remain limited by context length and weak instruction following, yet MLLMs are promising on shorter audio.
Read the full abstract ▾
All authors Fabian Retkowski, Maike Züfle, Thai Binh Nguyen, Jan Niehues, Alexander Waibel
Computation and Language · Cross-listed from cs.LG
A Behavioural and Representational Evaluation of Goal-Directedness in Language Model Agents
Raghu Arghal, Fade Chen, Niall Dalton et al. (+6 more)
Understanding an
agent's goals helps explain and predict its behaviour, yet there is no established methodology for reliably attributing goals to agentic systems. We propose a framework for evaluating goal-directedness that integrates behavioural evaluation with interpretability-based analyses of models' internal representations. As a case study, we examine an LLM
agent navigating a 2D grid world towards a goal state. Behaviourally, we evaluate the
agent against optimal policies across varying grid sizes, obstacle densities, and goal structures, finding that performance scales with task difficulty while remaining robust to difficulty-preserving transformations and multi-goal structures. We then use probing methods to decode internal representations of the environment and multi-step action plans. We find that the LLM
agent non-linearly encodes a coarse spatial map, preserving approximate task-relevant cues about its position and the goal location; that its actions are broadly consistent with these internal representations; and that
reasoning reorganises them, shifting from spatial cues towards immediate action selection. Our findings support the view that introspective examination is required beyond behavioural evaluations to characterise how
agents represent and pursue their objectives.
Read the full abstract ▾
All authors Raghu Arghal, Fade Chen, Niall Dalton, Evgenii Kortukov, Calum McNamara, Angelos Nalmpantis, Moksh Nirvaan, Gabriele Sarti, Mario Giulianelli
Computation and Language
How Should We Model the Probability of a Language?
Rasul Dent, Pedro Ortiz Suarez, Thibault Clérice et al. (+1 more)
Of the over 7,000
languages spoken in the world, commercial
language identification (LID) systems only reliably identify a few hundred in written form. Research-grade systems extend this coverage under certain circumstances, but for most
languages coverage remains patchy or nonexistent. This position paper argues that this situation is largely self-imposed. In particular, it arises from a persistent framing of LID as decontextualized text classification, which obscures the central role of prior probability estimation and is reinforced by institutional incentives that favor global, fixed-prior models. We argue that improving coverage for tail
languages requires rethinking LID as a routing problem and developing principled ways to incorporate environmental cues that make
languages locally plausible.
Read the full abstract ▾
All authors Rasul Dent, Pedro Ortiz Suarez, Thibault Clérice, Benoît Sagot
Computation and Language · Cross-listed from cs.AI
CoRefine: Confidence-Guided Self-Refinement for Adaptive Test-Time Compute
Chen Jin, Ryutaro Tanno, Tom Diethe et al. (+1 more)
Large
Language Models (LLMs) often rely on test-time scaling via parallel decoding (for example, 512 samples) to boost
reasoning accuracy, but this incurs substantial compute. We introduce CoRefine, a confidence-guided self-refinement method that achieves competitive accuracy using a fraction of the
tokens via a lightweight 211k-parameter Conv1D controller atop a frozen LLM. The controller consumes full-trace confidence to decide whether to halt, re-examine, or try a different approach, enabling targeted self-correction with an average of 2.7 refinement steps per problem and roughly 190-fold
token reduction relative to 512-sample baselines. Across diverse
reasoning benchmarks and three open-source models, the controller achieves 92.6 percent precision when it confidently halts, indicating that confidence dynamics reliably signal correctness without ground-truth verification. We extend this to CoRefine-Tree, a hybrid sequential-parallel variant that adaptively balances exploration and exploitation, with easy serving integration and verifier compatibility. By treating confidence as a control signal rather than a correctness guarantee, CoRefine provides a modular primitive for scalable
reasoning and agentic settings with imperfect verifiers.
Read the full abstract ▾
All authors Chen Jin, Ryutaro Tanno, Tom Diethe, Philip Teare
Computation and Language
GitSearch: Enhancing Community Notes Generation with Gap-Informed Targeted Search
Sahajpreet Singh, Kokil Jaidka, Min-Yen Kan et al. (+1 more)
Community Notes enable scalable, community-based fact-checking, but automated generation faces a coverage–quality trade-off. Note-dependent methods (e.g., summarization-based Supernotes) cannot operate without prior crowd input, while web-search LLMs achieve broad coverage but often retrieve evidence misaligned with what raters need. We introduce GitSearch (Gap-Informed Targeted Search), which treats missing, ambiguous, or disputed information as first-class
retrieval targets. It infers information gaps from a post and optional incomplete notes, searches specifically to resolve them, and synthesizes an evidence-grounded Community Note. To evaluate this, we introduce PolBench, a new
dataset of 78,698 U.S. political tweets with Community Notes. Keeping single-call search inference budget fixed, GitSearch improves human-rated helpfulness by 0.43 over unconditioned web search and wins 59% of pairwise comparisons. It generates notes for 99% of test cases, including the 46% where summarization methods fail. Human evaluators prefer GitSearch to human-written helpful notes in 69% of comparisons, primarily due to better context coverage. These findings show that how search is organized matters more than information access alone.
Read the full abstract ▾
All authors Sahajpreet Singh, Kokil Jaidka, Min-Yen Kan, Preslav Nakov
Computation and Language
Do Reasoning LLMs Refuse What They Infer in Long Contexts?
Yu Fu, Haz Sameen Shahgir, Huanli Gong et al. (+3 more)
Long-context LLMs can infer objectives that are not stated explicitly. This capability is useful for
reasoning over documents, code, retrieved evidence, and tool traces, but it also creates a safety risk: harmful intent can be distributed across a context and become visible only after the model composes the relevant pieces. Existing safety evaluations mostly test explicit harmful requests, and therefore miss this failure mode. We introduce compositional
reasoning attacks, a long-context threat model in which harmful requests are decomposed into semantically incomplete fragments and embedded in long contexts. The final query is neutral; the harmful objective emerges only if the model retrieves the fragments, composes them, and infers the implied goal. We instantiate this setting using AdvBench requests, varying the required
reasoning from Direct
Retrieval to Single-hop Aggregation, Chain
Reasoning, and Multi-hop Deductive
Reasoning, and evaluate 15 frontier LLMs on contexts up to 64k
tokens. Models usually refuse harmful requests when they are directly retrievable. However, refusal rates drop sharply when the same objectives must be reconstructed compositionally, often with larger failures in longer contexts. Benign reconstruction and fragment-position analyses indicate that these failures are not mainly
retrieval errors: models often infer the harmful objective and then comply. Increasing inference-time
reasoning improves refusal but remains incomplete and costly. Our results reveal a long-context safety gap: current models are better at refusing harmful requests they see than harmful objectives they infer.
Read the full abstract ▾
All authors Yu Fu, Haz Sameen Shahgir, Huanli Gong, Zhipeng Wei, N. Benjamin Erichson, Yue Dong
Computation and Language
Large Language Models for Geolocation Extraction in Humanitarian Crisis Response
G. Cafferata, T. Demarco, K. Kalimeri et al. (+2 more)
Humanitarian crises demand timely and accurate geographic information to inform effective response efforts. Yet, automated systems that extract locations from text often reproduce existing geographic and socioeconomic biases, leading to uneven visibility of crisis-affected regions. This paper investigates whether Large
Language Models (LLMs) can address these geographic disparities in extracting location information from humanitarian documents. We introduce a two-step framework that combines few-shot LLM-based named entity recognition with an
agent-based geocoding module that leverages context to resolve ambiguous toponyms. We
benchmark our approach against state-of-the-art pretrained and rule-based systems using both accuracy and
fairness metrics across geographic and socioeconomic dimensions. Our evaluation uses an extended version of the HumSet
dataset with refined literal toponym annotations. Results show that LLM-based methods substantially improve both the precision and
fairness of geolocation extraction from humanitarian texts, particularly for underrepresented regions. By bridging advances in LLM
reasoning with principles of responsible and inclusive AI, this work contributes to more equitable geospatial data systems for humanitarian response, advancing the goal of leaving no place behind in crisis analytics.
Read the full abstract ▾
All authors G. Cafferata, T. Demarco, K. Kalimeri, Y. Mejova, M.G. Beiró
Computation and Language
Understanding Dynamic Compute Allocation in Recurrent Transformers
Ibraheem Muhammad Moosa, Suhas Lohit, Ye Wang et al. (+2 more)
Token-level adaptive computation seeks to reduce inference cost by allocating more computation to harder
tokens and less to easier ones. However, prior work is primarily evaluated on natural-
language benchmarks using task-level metrics, where
token-level difficulty is unobservable and confounded with architectural factors, making it unclear whether compute allocation truly aligns with underlying complexity. We address this gap through three contributions. First, we introduce a complexity-controlled evaluation paradigm using algorithmic and synthetic
language tasks with parameterized difficulty, enabling direct testing of
token-level compute allocation. Second, we propose ANIRA, a unified recurrent
Transformer framework that supports per-
token variable-depth computation while isolating compute allocation decisions from other model factors. Third, we use this framework to conduct a systematic analysis of
token-level adaptive computation across
alignment with complexity, generalization, and decision timing. Our results show that compute allocation aligned with task complexity can emerge without explicit difficulty supervision, but such
alignment does not imply algorithmic generalization: models fail to extrapolate to unseen input sizes despite allocating additional computation. We further find that early compute decisions rely on static structural cues, whereas online halting more closely tracks algorithmic execution state.
Read the full abstract ▾
All authors Ibraheem Muhammad Moosa, Suhas Lohit, Ye Wang, Moitreya Chatterjee, Wenpeng Yin
Computation and Language · Cross-listed from cs.LG
Discovering Interpretable Algorithms by Decompiling Transformers to RASP
Xinting Huang, Aleksandra Bakalova, Satwik Bhattamishra et al. (+2 more)
Recent work has shown that the computations of
Transformers can be simulated in the RASP family of programming
languages. These findings have enabled improved understanding of the expressive capacity and generalization abilities of
Transformers. In particular,
Transformers have been suggested to length-generalize exactly on problems that have simple RASP programs. However, it remains open whether trained models actually implement simple interpretable programs. In this paper, we present a general method to extract such programs from trained
Transformers. The idea is to faithfully re-parameterize a
Transformer as a RASP program and then apply causal interventions to discover a small sufficient sub-program. In experiments on small
Transformers trained on algorithmic and formal
language tasks, we show that our method often recovers simple and interpretable RASP programs from length-generalizing
transformers. Our results provide the most direct evidence so far that
Transformers internally implement simple RASP programs.
Read the full abstract ▾
All authors Xinting Huang, Aleksandra Bakalova, Satwik Bhattamishra, William Merrill, Michael Hahn
Computation and Language
WildReward: Learning Reward Models from In-the-Wild Human Interactions
Hao Peng, Yunjia Qi, Xiaozhi Wang et al. (+3 more)
Reward models (RMs) are crucial for the training of large
language models (LLMs), yet they typically rely on large-scale human-annotated preference pairs. With the widespread deployment of LLMs, in-the-wild interactions have emerged as a rich source of implicit reward signals. This raises the question: Can we develop reward models directly from in-the-wild interactions? In this work, we explore this possibility by adopting WildChat as an interaction source and proposing a pipeline to extract reliable human feedback, yielding 186k high-quality instances for training WildReward via ordinal regression directly on user feedback without preference pairs. Extensive experiments demonstrate that WildReward achieves comparable or even superior performance compared to conventional reward models, with improved calibration and cross-sample consistency. We also observe that WildReward benefits directly from user diversity, where more users yield stronger reward models. Finally, we apply WildReward to online DPO training and observe significant improvements across various tasks. Code and data are released at https://github.com/THU-KEG/WildReward.
Read the full abstract ▾
All authors Hao Peng, Yunjia Qi, Xiaozhi Wang, Zijun Yao, Lei Hou, Juanzi Li
Computation and Language
Affective Flow Language Model for Emotional Support Conversation
Chenghui Zou, Ning Wang, Tiesunlong Shen et al. (+5 more)
Large
language models (LLMs) have advanced emotional support conversation, but existing
alignment methods rely mainly on sparse preferences at the response level or outcomes at the dialogue level, providing limited supervision for sequential strategy decisions in multi-turn interactions. This raises a key question: how can detailed process signals be derived from overall dialogue outcomes to guide the gradual adaptation of support strategies? We propose the Affective Flow
Language Model (AFlow), which models multi-turn emotional support as an affective utility flow evolving along dialogue trajectories. AFlow searches diverse support trajectories and estimates the utility of intermediate dialogue states and candidate strategies. It further introduces Affective Flow Preference Optimization (AFPO), which uses a flow-balance objective defined over dialogue subpaths to propagate downstream preference signals to intermediate states and learn strategy transitions consistent with support outcomes over the full dialogue. AFlow introduces flow-balance learning into multi-turn affective interaction, providing a process-based approach to dynamic affect modeling and continuous strategy optimization. Experiments on ExTES and ESConv show consistent improvements in strategy
alignment, response diversity, and generation quality across different model environments and evaluation settings. Our code is available at https://github.com/chz2025/AffectiveFlow.
Read the full abstract ▾
All authors Chenghui Zou, Ning Wang, Tiesunlong Shen, Luwei Xiao, Chuan Ma, Xiangpeng Li, Rui Mao, Erik Cambria
Computation and Language · Cross-listed from cs.LG
Bayesian Preference Learning for Test-Time Steerable Reward Models
Jiwoo Hong, Shao Tang and Zhipeng Wang
Reward models are central to aligning
language models with human preferences via reinforcement learning (RL). As RL is increasingly applied to settings such as verifiable rewards and multi-objective
alignment, RMs are expected to encode more complex and multifaceted preference distributions. However, classifier RMs remain static once trained, limiting their adaptability at test time. We propose Variational In-Context Reward Modeling (ICRM), a novel
Bayesian reward modeling objective that enables test-time steerability via in-context preference demonstrations. ICRM casts reward modeling as amortized variational inference over a latent preference probability under the Bradley-Terry model using a conjugate Beta prior. We show that ICRM adapts to unseen preference distributions at test time for both single and multi-objective settings. With more demonstrations, ICRM improves RM-Bench accuracy from 60.5 to 70.8, achieves lower calibration error than a generative judge on moral dilemma preferences, and expands the attainable Pareto frontier under conflicting preferences. We further study the practical applicability of ICRM for RL training, showing that it can effectively encode verifiable rewards by outperforming a conventional RM in math
reasoning. Finally, we provide theoretical guarantees that the variational objective admits a global interior optimum with finite confidence, followed by an ablation study on its role for the test-time steerability, and we analyze how KL regularization mitigates reward over-optimization.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.AI
The Use of AI Tools to Develop and Validate Q-Matrices
Kevin Fan, Jacquelyn A. Bialo and Hongli Li
Constructing a Q-
matrix is a critical but labor-intensive step in cognitive diagnostic modeling (CDM). This study investigates whether AI tools (i.e., general
language models) can support Q-
matrix development by comparing AI-generated Q-
matrices with a validated Q-
matrix from Li and Suen (2013) for a reading comprehension test. In May 2025, multiple AI models were provided with the same training materials as human experts. Agreement among AI-generated Q-
matrices, the validated Q-
matrix, and human raters' Q-
matrices was assessed using Cohen's kappa. Results showed substantial variation across AI models, with Google Gemini 2.5 Pro achieving the highest agreement (Kappa = 0.63) with the validated Q-
matrix, exceeding that of all human experts. A follow-up analysis in January 2026 using newer AI versions, however, revealed lower agreement with the validated Q-
matrix. Implications and directions for future research are discussed.
Read the full abstract ▾
Computation and Language
LakeHopper: Knowledge-Aware Adaptation of Column Type Annotators across Data Lakes
Yushi Sun, Xujia Li, Nan Tang et al. (+3 more)
Column Type Annotation (CTA), which assigns a semantic type to a table column, underpins data integration, cleaning, and search over data lakes. State-of-the-art annotators are pre-trained
language models (PLMs) fine-tuned on one particular corpus of tables, i.e., a source data lake, and they degrade sharply once deployed on a new (i.e., target) lake, whose tables and semantic type set both differ. Retraining per lake is prohibitive because it demands large volumes of expert annotations. We recast cross-lake adaptation as a knowledge management problem and make the resulting decomposition explicit: relative to a target annotator, a source annotator holds knowledge that must be discarded (source-specific), realigned and reused (shared), or acquired (target-specific). This decomposition exposes which part of the gap a general-purpose LLM can close and which part only target supervision can. Guided by it, we present LakeHopper, which adapts a source annotator under a fixed annotation budget through three coupled mechanisms: label-set realignment that transplants the output layer for shared types, LLM-verified gap discovery that localizes columns the annotator handles unreliably, and cluster-based propagation plus rehearsal fine-tuning that generalizes each flagged column into a labeling batch without erasing shared knowledge. Casting the LLM as a verifier of the annotator's own predictions rather than an annotator keeps every output inside the target type set, so LakeHopper structurally emits no out-of-domain labels, whereas prompted LLMs hallucinate types on 2.7-47.6% of columns. Across three data lake transfers of differing difficulty, LakeHopperlifts three PLM backbones by up to 71.4% relative macro-F1, reaches near-full-data quality with under 6% of the target labels, and matches fine-tuned table LLMs while training 27-131 times faster.
Read the full abstract ▾
All authors Yushi Sun, Xujia Li, Nan Tang, Quanqing Xu, Chuanhui Yang, Lei Chen
Computation and Language · Cross-listed from cs.AI
Dynamics Within Latent Chain-of-Thought: An Empirical Study of Causal Structure
Zirui Li, Xuefeng Bai, Kehai Chen et al. (+4 more)
Latent or continuous chain-of-thought methods replace explicit textual rationales with a number of internal latent steps, but these intermediate computations are difficult to evaluate beyond correlation-based probes. In this paper, we view latent chain-of-thought as a manipulable causal process in representation space by modeling latent steps as variables in a structural causal model (SCM) and analyzing their effects through step-wise do-interventions. We study two representative paradigms (i.e., Coconut and CODI) on both mathematical and general
reasoning tasks to investigate three key questions: (1) which steps are causally necessary for correctness and when answers become decodable early; (2) how influence propagates across steps and how this structure compares to explicit CoT; and (3) whether intermediate trajectories retain competing answer modes and how output-level commitment differs from representational commitment across steps. We find that latent-step budgets behave less like homogeneous extra depth and more like staged functionality with non-local routing, and we identify a persistent gap between early output bias and late representational commitment. These results motivate mode-conditional and stability-aware analyses, together with corresponding training/decoding objectives, as more reliable tools for interpreting and improving latent
reasoning systems. Code is available at https://github.com/J1mL1/causal-latent-cot.
Read the full abstract ▾
All authors Zirui Li, Xuefeng Bai, Kehai Chen, Yizhi Li, Jian Yang, Chenghua Lin, Min Zhang
Computation and Language
Map of Encoders – Mapping Sentence Encoders using Quantum Relative Entropy
Gaifan Zhang and Danushka Bollegala
We propose a method to compare and visualise sentence encoders at scale by creating a map of encoders where each sentence encoder is represented in relation to the other sentence encoders. Specifically, we first represent a sentence encoder using an
embedding matrix of a sentence set, where each row corresponds to the
embedding of a sentence. Next, we compute the Pairwise Inner Product (PIP)
matrix for a sentence encoder using its
embedding matrix. Finally, we create a feature vector for each sentence encoder reflecting its
Quantum Relative
Entropy (QRE) with respect to a unit base encoder. We construct a map of encoders covering 1101 publicly available sentence encoders, providing a new perspective of the landscape of the pre-trained sentence encoders. Our map accurately reflects various relationships between encoders, where encoders with similar attributes are proximally located on the map. Moreover, our encoder feature vectors can be used to accurately infer downstream task performance of the encoders, such as in
retrieval and clustering tasks, demonstrating the faithfulness of our map.
Read the full abstract ▾
Computation and Language
PERSPECTRA: A Scalable and Configurable Pluralist Benchmark of Perspectives from Arguments
Shangrui Nie, Kian Omoomi, Lucie Flek et al. (+2 more)
Pluralism, the capacity to engage with diverse perspectives without collapsing them into a single viewpoint, is critical for developing large
language models that faithfully reflect human heterogeneity. Yet this characteristic has not been carefully examined in the LLM research community and remains absent from most
alignment studies. Debate-oriented sources provide a natural entry point for pluralism research. Previous work builds on online debate sources but remains constrained by costly human validation. Other debate-rich platforms such as Reddit and Kialo also offer promising material: Reddit provides linguistic diversity and scale but lacks clear argumentative structure, while Kialo supplies explicit pro/con graphs but remains overly concise and detached from natural discourse. We introduce PERSPECTRA, a pluralist
benchmark that integrates the structural clarity of Kialo debate graphs with the linguistic diversity of real Reddit discussions. Using a controlled
retrieval-and-expansion pipeline, we construct 3,810 enriched arguments spanning 762 pro/con stances on 100 controversial topics. Each opinion is expanded to multiple naturalistic variants, enabling robust evaluation of pluralism. We initialise three tasks with PERSPECTRA: opinion counting (identifying distinct viewpoints), opinion matching (aligning supporting stances and discourse to source opinions), and polarity check (inferring aggregate stance in mixed discourse). Experiments with state-of-the-art open-source and proprietary LLMs, highlight systematic failures, such as overestimating the number of viewpoints and misclassifying concessive structures, underscoring the difficulty of pluralism-aware understanding and
reasoning. By combining diversity with structure, PERSPECTRA establishes the first scalable, configurable
benchmark for evaluating how well models represent, distinguish, and reason over multiple perspectives.
Read the full abstract ▾
All authors Shangrui Nie, Kian Omoomi, Lucie Flek, Zhixue Zhao, Charles Welch
Computation and Language
FactSim: Fact-Checking for Opinion Summarization
Leandro Anghinoni and Jorge Sanchez
We explore the need for more comprehensive and precise evaluation techniques for generative artificial intelligence (GenAI) in text summarization tasks, specifically in the area of opinion summarization. Traditional methods, which leverage automated metrics to compare machine-generated summaries from a collection of opinion pieces, e.g. product reviews, have shown limitations due to the paradigm shift introduced by large
language models (LLM). This paper addresses these shortcomings by proposing a novel, fully automated methodology for assessing the factual consistency of such summaries. The method is based on measuring the similarity between the claims in a given summary with those from the original reviews, measuring the coverage and consistency of the generated summary. To do so, we rely on a simple approach to extract factual assessment from texts that we then compare and summarize in a suitable score. We demonstrate that the proposed metric attributes higher scores to similar claims, regardless of whether the claim is negated, paraphrased, or expanded, and that the score has a high correlation to human judgment when compared to state-of-the-art metrics.
Read the full abstract ▾