Computation and Language
How Much Does Corpus Choice Change Dependency-Distance Estimates?
Sirui Chen
Dependency-distance estimates derived from a single corpus are routinely treated as properties of a
language, yet this assumption has not been tested across independently compiled corpora. We compared mean dependency-distance estimates across 38 same-
language treebank pairs from Universal Dependencies v2.18, using concordance correlation, Bland-Altman analysis, and a twelve-specification multiverse design. Cross-treebank agreement was moderate at best: substituting one treebank for another reversed nearly 40 percent of pairwise
language orderings, and treebank choice accounted for roughly 29 percent of between-group variance. This disagreement substantially exceeded within-treebank sampling error and persisted across all twelve preprocessing specifications. Nevertheless, every treebank confirmed dependency-length minimization (normalized ratio below 1). The data are more consistent with MDD as a corpus-conditioned composite of grammatical, register, and annotation factors than as a stable
language-level parameter: the qualitative DLM universal survives corpus substitution, but the ordinal cross-linguistic ranking does not.
Read the full abstract ▾
Computation and Language
RL-ADA: A World-Feedback Framework for Adversarially Robust Enterprise Dialogue Agents
Ram Narayanan, Harshit Rajgarhia and Abhishek Mukherji
Deploying task-oriented dialogue
agents in enterprise customer support faces a persistent annotation bottleneck: robust training requires labelled interaction data at scale, yet enterprise conversational logs are
privacy-sensitive and expensive to annotate, while user behaviour evolves faster than labelling pipelines can keep pace. We present RL-ADA (Reinforcement Learning with
Adversarial Dialogue
Agents), a co-evolutionary training framework that eliminates this bottleneck by replacing human labels with world feedback: consequence-based reward signals derived directly from measurable interaction outcomes. A Customer Support
Agent (DA, 3B parameters) and an
Adversarial Customer
Agent (CA, 7B parameters) co-evolve in an
adversarial arena guided by a fixed automated judge: the DA is rewarded for correctly handling multi-turn customer conversations to successful resolution, while the CA is rewarded for producing realistic, intent-concealing utterances that cause misroutes, creating asymmetric
adversarial pressure through opposing but independently structured rewards. An isolation gym iteratively retrains the weaker
agent on prior-failure transcripts, requiring no human annotation at any stage. In a banking customer support proof of concept, tool-routing errors are eliminated and the strict end-to-end PASS rate doubles over five co-evolutionary cycles, driven solely by automated arena reward with no labelled data. We additionally observe the emergence of Contextual Camouflage, an
adversarial strategy in which the CA learns to embed intent within dense realistic customer detail purely from reward pressure, with direct implications for enterprise red-teaming and robustness evaluation.
Read the full abstract ▾
Computation and Language
Dual-Form ASR: Semantics-Aware Inverse Text Normalization for Chinese Speech Recognition
Fengrun Zhang, Li Fu, Wangjin Zhou et al. (+3 more)
Modern automatic
speech recognition (ASR) scenarios require both spoken-form transcripts for faithful transcription and readable written-form transcripts with inverse text normalization (ITN). However, these forms are typically produced by cascaded modules, where a spoken-form ASR output is rewritten by a separate ITN component, making written-form ASR-ITN vulnerable to recognition errors and decoupling normalization from acoustic-contextual modeling, especially for semantically dependent numeric expressions. In this paper, we propose Dual-Form ASR (DF-ASR), a framework that extends spoken-form ASR capability to semantics-aware written-form ITN through paired spoken-form and written-form supervision while retaining
prompt-level selection between transcript forms. The dual-form supervision is constructed via a large
language model (LLM)-driven generate-and-judge workflow, and training is further enhanced by ITN-MWER, a sequence-level objective that assigns higher cost to errors on normalization-sensitive spans. We also introduce a decision-aware REQUIRE-ITN/\FORBID-ITN protocol to separately measure required normalization and forbidden-span preservation. On manually annotated Chinese subsets from SpeechIO, DF-ASR consistently outperforms open-source ASR-ITN systems, remains competitive with strong closed-source references, and preserves reliable
prompt-level control between spoken-form and written-form outputs.
Read the full abstract ▾
All authors Fengrun Zhang, Li Fu, Wangjin Zhou, Lu Fan, Youzheng Wu, Xiaodong He
Computation and Language
Agentic Scaffolding Amplifies Sycophantic Behavior in Large Language Models
Thantham Jittham
Sycophancy in large
language models, the tendency to prioritize user agreement over truthful responses, has been documented extensively but studied primarily in single-turn settings. This paper investigates a critical question: does subjecting LLMs to greater interaction scaffolding make sycophancy better or worse? Across 4,800 veracity judgments (200 statements $\times$ 6 models $\times$ 4 conditions), we find that the interaction scaffolding characteristic of agentic systems (feedback loops, reconsideration checkpoints, and iterative refinement) systematically amplifies sycophantic behavior. Multi-turn interaction, user pressure, and iterative self-refinement each provide additional opportunities for models to drift toward agreement, and this drift coincides with a mean accuracy drop of $-6.3$ percentage points, establishing the capitulation as harmful rather than corrective. More capable models show larger amplification effects, a troubling inversion of expectations. We introduce the concept of agentic sycophancy amplification (ASA) and two novel metrics: capitulation rate and sycophantic capitulation rate. Our results indicate that as AI systems acquire greater autonomy, sycophancy becomes compounding rather than merely persistent. Systems designed with human oversight loops may inadvertently create the conditions for this drift.
Read the full abstract ▾
Computation and Language
On the Role of Citations in Preference Data
Yu Hou, Hal Daumé III, Rachel Rudinger et al. (+1 more)
Many NLP tasks require systems to provide attribution in their outputs–i.e. citations to grounding sources. Attribution serves as a bulwark against model
hallucination and as a means for users to verify the credibility of model outputs. Yet, it is unclear how humans and LLMs evaluate citations when comparing outputs, a process central to reward modeling and modern LLM post-training. This paper studies the role of citations in the preferences of human judges and four open-source LLMs within the context of scientific question answering, leveraging mixed effects models to investigate the influence of citations on pairwise judgments. Among our key findings are (1) that humans prefer more diverse citations but fewer overall, and (2) that LLMs show some citation-related preferences compared to humans, despite lacking access to the sources, but these preferences depend on the data and specific models. We further discuss the implications of our findings for preference data collection.
Read the full abstract ▾
All authors Yu Hou, Hal Daumé III, Rachel Rudinger, William Walden
Computation and Language · Cross-listed from cs.CY
An Investigation of the NeurIPS and ICML 2025 Position Tracks
Fan Yang, Wenkai Li and Jun Liu
ML venues shape what kinds of research claims become legible to reviewers and what forms of evidence count as rigorous. The NeurIPS and ICML Position Paper Tracks were created for agenda-setting work, making their early composition worth
auditing. This paper argues that the publicly accessible 2025 reviewed pool is dominated by reformist critique, and that the track should explicitly solicit direction-setting work alongside, not in place of, the reformist critiques it already hosts well. We audit every accessible submission to the NeurIPS 2025 and ICML 2025 Position Tracks under a pre-specified rubric, and compare the resulting pattern with a reference class of widely recognized agenda-shifting ML papers. Three-quarters of audited submissions critique an existing
benchmark, evaluation, or methodology; these papers score highly on our artifact-coupling rubric, but evidentiary depth does not predict reviewer rating. The reference class (AlexNet, the
Transformer, Concrete Problems in AI Safety, and others) differs from the accessible reviewed pool in artifact kind: agenda-shifting papers typically gave the field something new to build on, test against, or contest, such as a measurement protocol,
benchmark proposal, toy implementation,
dataset card, audit template, or falsifiable experimental program. We close with four CFP-level interventions aimed at broadening the submission mix without displacing the critiques the track already hosts well.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.CY
Plausible but Not Valid: A Psychometric Audit of LLMs as Synthetic Survey Respondents
Mantas Lukauskas and Viktorija Šarkauskaitė
Large
language models (LLMs) are increasingly used as synthetic survey respondents, but existing evaluations mostly ask whether individual answers look plausible. We ask a stricter, psychometric question: do LLMs preserve the joint distribution, latent structure, reliability, mediation pathways, and demographic effects of real human survey data? We introduce an open Lithuanian organisational-psychology
dataset (n=263 employees; three validated instruments; 68 items, 12 subscales) and condition 37 LLMs on real respondent profiles under a persona-disclosure ladder, presentation,
reasoning-effort and cross-
language ablations, counterfactual demographic swaps, and a memorization probe. The resulting Psychometric Similarity Score (PSS) is anchored against five statistical baselines, a held-out human ceiling, and the real answers of the same respondents. LLMs reproduce the direction of human psychometric relationships, but statistical baselines with no
language model match or beat them. The best LLM (PSS 0.81) is level with a Gaussian copula fit only on other respondents (0.82) and below the real answers of the same 100 people (0.89). A simple classifier separates every LLM sample from humans (median AUC 0.99), and verbatim recall of the items is rare (at most 4.7%) and unrelated to PSS. Counterfactual swaps move answers in the direction of human group differences and by a similar amount (role 0.49 and education 0.33 human SD, vs. 0.46 and 0.44 in humans). Gender swaps move them by 0.06 SD. Tucker's phi on UWES falls inside an item-permutation null for 10 of 37 models. Synthetic respondents compress the response scale (within-respondent SD 0.79 vs. 1.19), lose predictive validity on held-out humans (mean R^2=-0.14 vs. 0.28), and produce significant indirect effects on 4 of 7 mediation paths that are null in humans. LLM samples are therefore not a drop-in replacement for human survey data.
Read the full abstract ▾
Computation and Language
Rachid Arezki
Transformer architectures rely on dense self-
attention to model long-range dependencies, but this mechanism exhibits quadratic complexity with respect to sequence length. We introduce BCMT (Blockwise Causal
Memory Transformer), an architecture for long-context
language modeling that decouples local
token interactions from global context propagation. Dense causal self-
attention is applied independently within local blocks, while each block produces an adaptive summary aggregated through an exponential causal
memory. This
memory is subsequently injected back into the
token representations, enabling efficient propagation of long-range contextual information without relying on explicit global
attention. Unlike standard
Transformers and recurrent
memory architectures, BCMT maintains neither dense interactions between distant
tokens nor learned
memory states. Its
memory mechanism is fully parallelizable and remains compatible with standard implementations of dense self-
attention. Experiments on
language modeling with context lengths of up to 1024
tokens show that BCMT achieves validation performance comparable to that of Dense
Transformers while significantly improving training throughput and reducing
memory consumption. An ablation study further confirms that these improvements arise from the proposed
memory mechanism. These results demonstrate that an exponential causal
memory constructed from block summaries provides an effective alternative to dense global
attention mechanisms for long-context
language modeling.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.IR
SciClaimSeekers at CheckThat! 2026: Retrieving Scientific Sources for Social Media Claims with LLM Reranking
Mohotarema Rashid, Nansu Baniya, Anirban Saha Anik et al. (+2 more)
Scientific claims often spread on social media faster than they can be verified, while posts rarely link to the original scholarly sources. To tackle this problem this paper presents system called SciClaimSeekers, a
retrieval and reranking framework by combining BM25 and zero-shot
multilingual E5
retrieval with Reciprocal Rank Fusion (k=60), followed by Qwen2.5-14B-Instruct pointwise reranking. The pipeline reaches 64.36% MRR@5 on the English development set a 13.67-point jump over BM25 and 10.17 points over the unranked hybrid and 64.39% on the official test set, in the CLEF-2026 CheckThat! Task 1 evaluation. Our experiment suggests that large pre-trained models, when combined into a careful pipeline, can be competitive with fine-tuned approaches on this task.
Read the full abstract ▾
All authors Mohotarema Rashid, Nansu Baniya, Anirban Saha Anik, Xiaoying Song, Lingzi Hong
Computation and Language · Cross-listed from cs.IR
SourceMinds at CheckThat! 2026: NLI-Grounded Citation Auditing in a Multi-Agent Pipeline for Full Fact-Checking Article Generation
Farhan Sharukh Hasan, Anirban Saha Anik, Eric Liu et al. (+3 more)
This paper presents our system for Task 3 of the CLEF 2026 CheckThat! Lab, which focuses on generating full fact-checking articles from claims, veracity labels, and evidence documents. We propose a multi-
agent pipeline that combines evidence
retrieval, structured fact planning, article generation, gated self-critique, and NLI-based citation
auditing. The system retrieves claim-relevant evidence using dense
retrieval, reranking, and source-balanced selection, then generates a citation-supported article from a structured plan. A gated self-critique stage revises weakly grounded drafts, while the NLI citation auditor repairs missing citations and removes unsupported or redundant ones. The approach highlights the importance of combining evidence selection, structured generation, and post-generation citation validation for source-grounded fact-checking article generation.
Read the full abstract ▾
All authors Farhan Sharukh Hasan, Anirban Saha Anik, Eric Liu, Xiaoying Song, Mohotarema Rashid, Lingzi Hong
Computation and Language · Cross-listed from cs.IR
Influence of Prompt Engineering on Small Language Models for Guarded Query Routing
Richard Šléher, William Brach, Kristián Košťál et al. (+1 more)
We study the problem of guarded query routing, where we assume that a user query first meets a router that either determines the ideal endpoint for in-distribution queries or rejects out-of-distribution queries that are potentially unsafe or out of the system's scope. We investigate whether compact open-weight Small
Language Models (SLMs) can jointly handle both tasks under latency constraints. We evaluate 22 models on GQR-Bench and score them with the harmonic mean of in-distribution and out-of-distribution accuracy. We find that mid-scale SLMs come close to frontier model routing quality at much lower latency. Still, many compact models fail because they do not reliably follow the required output format. However, our results show that
prompt optimization techniques enable SLMs to handle such cases gracefully, without changing the models' weights. Moreover, few-shot
prompt optimization raises Mistral 7B from 81.79 to 90.87 GQR-Score and lifts Qwen3.5 9B to 95.74, the best optimized score in our study and within 0.3 points of the strongest unoptimized larger model: Gemma 3 27B at 96.01. The bare DSPy signature, without in-context exemplars, is the most effective strategy for Granite 4 Tiny, raising its score from 54.29 to 83.05. These results show that
prompt optimization is a useful first step for guarded query routing, while weaker models may still need weight-level adaptation or schema-aware training
Read the full abstract ▾
All authors Richard Šléher, William Brach, Kristián Košťál, Lukas Galke Poech
Computation and Language · Cross-listed from cs.IR
When Thinking Before Retrieval Hurts: TraceBound Diagnostics for Adaptive Knowledge-Graph Retrieval
Partha Sarathi Purkayastha
Adaptive
retrieval promises to make knowledge-graph question answering more robust by letting a controller search, inspect neighborhoods, revise actions, and stop when evidence is sufficient. We study this premise by introducing TraceBound, a lightweight profile- and trace-conditioned diagnostic protocol for an ARK-style retriever on text-rich knowledge graphs. TraceBound exposes a compact query profile before
retrieval, issues short trace hints after observable failure symptoms, and logs trajectory counters, while keeping graph data, tools, gold labels, and ranking metrics fixed. Across STaRK validation and held-out subsets, the added conditioning improves inspectability but consistently reduces
retrieval quality under open-weight controllers. Paired trajectory analysis localizes the degradation to repeated calls, zero-result calls, and misallocated exploration budget, while stricter interaction budgets shorten trajectories without repairing the policy. The result diagnoses the common failure mode in that "thinking before
retrieval” must be evaluated as a control problem over action selection, not as a
prompt-format change.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.NE
QDEvo: A Multi-Objective Quality-Diversity Framework for Automated Heuristic Design
Nam Do Khanh, Nhat Nguyen Tran Minh, Dat Pham Vu Tuan et al. (+2 more)
The integration of Large
Language Models (LLMs) with evolutionary computation has emerged as a powerful paradigm for automated heuristic design in combinatorial optimization. However, existing approaches suffer from mode collapse, converging to homogeneous populations that lack semantic diversity and fail to explore the full algorithmic space. We propose Quality-Diversity
Evolution (QDEvo), a multi-objective framework that integrates Quality-Diversity optimization with LLM-driven heuristic search, maintaining an unbounded archive of semantically diverse
algorithms using pre-trained code
embeddings and incorporating hierarchical self-reflection to guide the evolutionary process. Extensive experiments across standard
benchmarks and real-world industrial applications demonstrate that QDEvo significantly outperforms state-of-the-art methods in both Hypervolume and Inverted Generational Distance metrics. Our framework enables the discovery of heuristics that are simultaneously high-performing, computationally efficient, and semantically diverse, providing practitioners with a rich portfolio of solutions for complex optimization problems.
Read the full abstract ▾
All authors Nam Do Khanh, Nhat Nguyen Tran Minh, Dat Pham Vu Tuan, Long Doan, Binh Huynh Thi Thanh
Computation and Language · Cross-listed from cs.MA
Collective Intelligence with Foundation Models
J. de Curtò and I. de Zarzà
As foundation models grow in scale and diversity, coordinating multiple models into cooperative
reasoning systems offers a path toward safer, more reliable AI. This chapter presents a multi-
agent framework where solver models generate independent drafts, each undergoes structured critique and revision by a critic
agent, and an aggregator
agent synthesizes a final consensus solution. A scoring module provides semantic, numerical, and procedural evaluation across all
agents. Through ablation studies on a
benchmark spanning calculus, physics, chemistry, biology, economics, optimization, statistics, and mathematics, we isolate the contributions of framework architecture versus model diversity. We compare four configurations: (1) Individual Baseline, (2) Homogeneous Framework using one shared model, (3) Redundant Homogeneous Solvers using multiple instances of the same model, and (4) Heterogeneous Framework with diverse specialized models. Results show that while framework structure and redundant sampling yield modest gains, model heterogeneity is the critical factor driving substantial performance improvements. The heterogeneous configuration achieves superior step-wise accuracy (0.64 vs. 0.54 for individual models; 2.3x improvement over homogeneous configurations) with reduced variance across categories and difficulty levels. Step-wise
reasoning quality (correctness of intermediate steps, not just final answers) improves dramatically only with model diversity, showing that heterogeneous
agents provide complementary error detection and
reasoning refinement essential for explainability and auditability. We discuss architectural principles, evaluation methodology, and implications for Global Applied AI, showing how heterogeneous multi-
agent coordination supports transparent, auditable, high-confidence decision-making across scientific and industrial domains.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.PL
SPL: Orchestrating Workflows with Declarative Deterministic-Probabilistic Composition
Wen G. Gong
We present SPL (Structured
Prompt Language), a declarative
language that composes deterministic and probabilistic computation modes in a single specification. While existing frameworks separate these – orchestration systems (AutoGen, CrewAI, LangGraph) for LLM calls, symbolic tools (SymPy, SageMath, Lean) for computation – SPL unifies them. It provides GENERATE/EVALUATE for probabilistic computation and SOLVE/ASSERT for deterministic computation, sharing syntax, variable bindings, and runtime routing. A .spl specification runs unchanged across local nodes (Ollama), cloud APIs (OpenRouter, Anthropic), and distributed grids (Momagrid), with model and verifier selection deferred to invocation time. We validate SPL through an extensive 78-recipe cookbook and a controlled 1,200-run experiment (10 models x 20 problems x 2 arms x 3 repetitions; the 20 problems span 6 difficulty tiers). The solver arm achieves 82-93% machine-verified correctness (sonnet-4-6: 85%, gemma4:e2b: 93%) while the LLM-only arm measures output production without mathematical verification, making the comparison one of verified correctness against unverified fluency. A backend difficulty gradient emerges (SymPy 78%, Sage 54%), and the dominant failure mode is solver_error (kernel-rejected expressions), not format non-compliance.
Read the full abstract ▾
Computation and Language · Cross-listed from eess.IV
Reconfigurable Radiology Labels Without Relabeling
Jean-Benoit Delbrouck, Dave Van Veen, Akash Pattnaik et al. (+4 more)
Public chest-radiograph (CXR)
datasets are typically released with small, fixed label schemas such as CheXpert-14. However, the underlying free-text reports describe far more findings – and which findings matter depends on the task, site, and reader. We release a pipeline that converts free-text reports into multi-label
matrices and then reconfigures the label schema through dictionary edits rather than new inference passes, i.e., without relabeling the corpus. After this one-time pass, reconfiguring MIMIC-CXR (223K reports) from cached annotations takes 196 seconds with no API cost, compared to \$6.6K for an equivalent relabeling pass with Claude Opus 4.7. Using a 58-label taxonomy, we show that 43% of CXR studies contain at least one finding outside CheXpert-14. Image probes trained on these labels match CheXpert-14 probes on shared targets while also reaching 0.78 AUROC on expert-reviewed long-tail labels that CheXpert-14 cannot represent. These results suggest a different unit of work for radiology labeling: once reports are structured, the label schema becomes a configuration to edit, not a corpus to relabel.
Read the full abstract ▾
All authors Jean-Benoit Delbrouck, Dave Van Veen, Akash Pattnaik, Kalina Slavkova, Javid Abderezaei, Harris Bergman, Khan Siddiqui
Computation and Language
Where to cut, how deep: BPE and Unigram-LM on chemistry SMILES
Hunter Heidenreich
Every chemical
language model reading SMILES begins with a tokenizer, yet the field has inherited byte-pair encoding (BPE) from natural
language with little scrutiny. In natural
language, BPE's principal alternative, Unigram-LM, is known to build structurally different vocabularies. Whether that contrast survives in chemistry was open. We report a controlled comparison of BPE and Unigram-LM over a fixed 165-
token chemistry base, at the small vocabulary sizes where
token embeddings are learnable, across three corpus typologies (diverse, drug-like, natural-products) and both pre-tokenization boundary policies. The two do not converge. In all 22 matched conditions they build near-disjoint subword vocabularies: cross-
algorithm Jaccard overlap on the learned pieces never exceeds 0.161, and at most 0.05 once weighted toward the high-frequency pieces a model updates most. Unigram-LM also segments held-out molecules into 29-41% more
tokens; the arms largely agree on where to cut but not how deeply, so BPE's
segmentation is a strict coarsening of Unigram-LM's on 80-99% of molecules. The separation holds across corpus, boundary, and vocabulary size, persisting even at eight times that scale. The subword
algorithm is therefore a modeling decision, not a free default. The study trains no
language models.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.AI
Yusuf Khan and Carlo Lipizzi
Language agents run a loop - observe, reason, act - but the
memory they reason over sits outside it: a store queried at most once per turn. We study the regime where
memory moves inside the loop, read and written on every step. The obstacle has always been latency: networked stores answer in tens to hundreds of milliseconds, and in-loop
retrieval can inflate end-to-end latency by up to 83x when
retrieval is expensive. Prior work manages that cost rather than questioning it: serving-layer scheduling hides it, "
memory-first" designs ration
retrieval to once per turn. We argue latency is a property of where the store lives, not the in-loop pattern: an in-process store answers in ~100us, three orders of magnitude below the network regime, and at that speed the per-step tax collapses. By the extended-mind thesis's parity principle, a store fast enough to be constantly and directly available becomes extended working
memory, not a tool the
agent merely consults. The premise is causal: holding a fixed per-turn
memory-latency budget and varying only the store's answer speed, redundant actions rise monotonically with latency - 0.0 of 12 at in-process speed, 7.2 of 12 at a 110ms cloud round trip (gpt-5-nano, gpt-5-mini; exact permutation p=0.0079). We demonstrate the regime end-to-end: across four GPT-5-class models under a bounded window, recall improves from 0/5 to 3.6-4.8/5 with in-loop
memory, store ops at p50 80-165us - though an instructed restate-every-reply baseline also solves it perfectly, at a
token cost that grows with the working set. The store never lost a fact in any run (244 of 244 writes kept); every miss traces to the
agent's read policy, not the store. Our measurements also relocate the bottleneck: the dominant per-step cost is
embedding (~200-400ms over the network); pairing the in-process store with a small local embedder returns the complete operation to a measured ~40us.
Read the full abstract ▾
Computation and Language
UCSC NLP at SemEval-2026 Task 10: Boundary-Aware Span Extraction and RoBERTa Classification for Conspiracy Detection
Dom Marhoefer, Milos Suvakovic, Glenn Grant-Richards et al. (+2 more)
We present our systems for SemEval-2026 Task 10 (PsyCoMark), addressing conspiracy marker extraction (Subtask 1) and document-level conspiracy detection (Subtask 2). For marker extraction, we formulate the task as multi-label span classification over enumerated candidate spans, using IoU >= 0.95 positive labeling, hard-negative sampling, and containment-based non-maximum suppression (NMS) with boundary-aware span representations. Document classification is modeled independently using a sequence classifier with label smoothing and a stratified train-validation split. Analysis shows that entity-like roles (Actor, Victim) are detected robustly, while abstract roles (Action, Effect, Evidence) remain sensitive to boundary criteria. On the official test set, our systems rank 7th in Subtask 1 (0.2251 macro F1) and 11th in Subtask 2 (0.7694 weighted F1).
Read the full abstract ▾
All authors Dom Marhoefer, Milos Suvakovic, Glenn Grant-Richards, Aidan Pinero, Ryan King
Computation and Language
RPAM: A Principled Metric for Evaluating Associations in Language Models with High Predictive Validity in Downstream Outputs
Damian Hodel, Jevin West and Aylin Caliskan
Language models (LMs) exhibit problematic biases, such as stereotypes. Effectively analyzing and mitigating such biases requires accurate and generalizable evaluation methods of the underlying associations. Some existing approaches focus on downstream metrics that analyze associations in generated text. Since generated text content can vary drastically across LMs, such metrics often require specialized evaluation
datasets, which limits the generalization of such downstream metrics. In contrast, upstream metrics examine LMs at the fundamental level of
embeddings or continuation probabilities, enabling principled association analyses across LMs. Yet, to date, no upstream metric for generative LMs has uncovered a strong relationship with real-world associations, including those measured in generated text. To address this gap, we introduce the Relative Probability Association Metric (RPAM), an association evaluation metric for generative LMs. For three LMs of different quality of
language generation and purpose (Mistral-7B-Instruct, Mistral-7B, and GPT-2) and well-studied evaluation
datasets (WEAT-WS, Bellezza, WS-353, and SST2), we find a strong relationship between upstream RPAM measurements and corresponding implicit and explicit associations observed in humans, as well as biases measured downstream with LM-specific tasks, outperforming prior record values where applicable.
Read the full abstract ▾
Computation and Language
Do It Right! A Methodology for Successful NLP System Development
Olga V. Patterson, Brett South, T. Elizabeth Workman et al. (+1 more)
Natural
language processing (NLP) is a common method for supplying data to clinical research and decision making by extracting information from electronic medical records. Numerous textbooks and tutorials describe specific
algorithms and applications for text processing, yet algorithmic knowledge is only one ingredient of a successful NLP project. Drawing on the available literature, this paper presents a stepwise approach that applies the Systems Development Life Cycle (SDLC) to projects that rely on data extraction through
language processing.
Read the full abstract ▾
All authors Olga V. Patterson, Brett South, T. Elizabeth Workman, Scott L DuVall
Computation and Language
Population-Level Profiling of DSM-5 Depressive Symptoms Among Self-Reported ADHD and ASD Users on Twitter: An Exploratory Study Using Advanced NLP and Statistical Analysis
Muhammad Rizwan, David Nabergoj and Jure Demšar
Background: Depression frequently co-occurs with ADHD and autism spectrum disorder (ASD), but population-level differences in symptom expression between these groups remain underexplored. Objective: We examined whether social media users with ADHD and ASD differ in how they express DSM-5 depressive symptoms in their tweets, and whether differences persist across varying levels of depressive-content filtering. Methods: We analysed 1,282,437 tweets from 792 users (622 ADHD; 170 ASD) with self-reported diagnoses on Twitter. Tweets were pre-filtered for depressive relevance using zero-shot NLI, then classified into nine DSM-5 symptoms using MentalRoBERTa fine-tuned on ReDSM5. Profiles were mean-centered per user. We applied L1-penalised logistic regression with cross-validation to distinguish ADHD from ASD users, complemented by Pearson correlations for symptom co-occurrence, and tested robustness across five filtering thresholds using bootstrapping. Results: MentalRoBERTa achieved macro-F1 of 0.901 on a held-out set, outperforming the original ReDSM5
benchmark. ADHD vs ASD classification yielded stable but modest performance (cross-validated ROC-AUC 0.645-0.653). Cognitive issues, sleep issues, appetite change, and fatigue leaned toward ADHD, while suicidal ideation and anhedonia leaned toward ASD. A largely shared symptom co-occurrence structure emerged between groups; no pair met our criterion for a robust disorder-specific difference. Conclusions: Population-level differences in depression-related
language between ADHD and ASD social media users were consistently observed across thresholds, reflecting reproducibility rather than clinical validity. Findings are exploratory and do not establish differing phenomenology at the individual level.
Read the full abstract ▾
Computation and Language
NAVER LABS System Re-implementation for the IWSLT 2026 Instruction-Following Task
Anand Kamble and Aniket Tathe
We re-implement the NAVER LABS IWSLT 2025 instruction-following pipeline for the IWSLT 2026 Shared Task (constrained condition, short audio track), adapting it to the mandated components: SeamlessM4T-v2-large as the
speech encoder and Qwen3-4B-Instruct as the LLM backbone. The three-stage approach projector
alignment, text-only LoRA pre-training, and
multimodal merging is preserved from the original design. We additionally construct 100k synthetic instruction-following examples across ten
speech-centric task types (10k per task) from the provided corpora, suitable for further Stage 3 fine-tuning. Our primary model achieves COMET 0.781 on EN-ZH
speech translation and BERTScore-F1 0.346 on English SQA on the MCIF
benchmark.
Read the full abstract ▾
Computation and Language
BaFCo: A Document Understanding Benchmark for Complex Bangla Form Comprehension
Abu Tyeb Azad, Ishita Sur Apan, Fahim Ahmed et al. (+8 more)
Document comprehension is a challenging yet impactful task for
Multimodal Large
Language Models, especially as these systems see growing adoption in real-world, human-centric applications. However, this adoption is limited for low-resource
languages such as Bangla due to the scarcity of high-quality annotated data. To address this gap, we introduce BaFCo, a
benchmark dataset for Bangla form comprehension with a focus on Document Layout Analysis (DLA) and Key Information Extraction (KIE). BaFCo curates 200 multi-page complex Bangladeshi government forms, sourced from across diverse sectors including agriculture, education, banking, and land management. To accurately capture the structural and contextual complexity of these forms, we define a fine-grained annotation schema comprising 26 types of form entities, along with a separate coarse form entity set consisting of 5 types. We evaluate the latest MLLMs from the ChatGPT, Gemini, Claude, Qwen, and Kimi series using zero-shot and chain-of-thought
prompts under both low and high
reasoning setups. Our results reveal limitations in current MLLMs' ability in comprehending Bangla forms, particularly in accurately localizing highly granular form entities. Our
dataset and code is available at: https://huggingface.co/
datasets/Mausul/bafco
Read the full abstract ▾
All authors Abu Tyeb Azad, Ishita Sur Apan, Fahim Ahmed, Sumaiya Karim Katha, Ezharuddin Jubaer, Armun Alam, Pranjal Kumar Nandi, Amin Ahsan Ali, Aman Chadha, Md Mofijul Islam, AKM Mahbubur Rahman
Computation and Language
Revisiting the Relation Between Language Model Perplexity and ASR Word Error Rate for Modern End-to-End Speech Recognition
Mohammad Zeineldeen, Albert Zeyer, Haoran Zhang et al. (+3 more)
Language model (LM) perplexity (PPL) has historically been used as a proxy for automatic
speech recognition (ASR) word error rate (WER), with prior work reporting an approximately linear relation in log-log space. Modern end-to-end ASR systems challenge this assumption because they already contain internal
language modeling capacity, are often evaluated without external
language models, and can now be combined with neural LMs and large
language models (LLMs) through different recognition strategies. This paper revisits the relation between PPL and WER for modern ASR systems. We study whether external LMs still improve current end-to-end ASR systems, whether the PPL-WER relation remains linear in log-log space, how encoder context length affects this relation, and how LLM perplexities fit into the trend observed for standard neural LMs. We further investigate internal
language modeling (ILM) in
attention-based encoder-decoder systems and show that ILM subtraction changes the observed PPL-WER relation, indicating that the decoder's internal LM must be considered when interpreting the effect of external LM quality.
Read the full abstract ▾
All authors Mohammad Zeineldeen, Albert Zeyer, Haoran Zhang, Robin Schmitt, Ralf Schlüter, Hermann Ney
Computation and Language
ResonatorLM: Causal Resonant Field Mixing for Efficient Long-Context Language Modeling
Archie Chaudhury
Contemporary
language models are dominated by the
transformer architecture, which leverages self-
attention mechanisms to enable more efficient, parallelized training across a wide set of documents and corpora. This has allowed
transformers to effectively model data across a wide range of modalities and contexts. However,
transformers, along with their conventional counterparts such as recurrent neural networks (RNNs) and convolutional neural networks (CNNs), often struggle to maintain efficiency when processing long contexts. We introduce ResonatorLM, a new mechanism that replaces
attention with a physics-derived alternative. ResonatorLM treats
token sequences as a single, driven one-dimensional latent field and replaces
attention dot products with causal functions of damped resonators. We implement ResonatorLM on a traditional network architecture and test it on standard long-context modeling tasks. We find that in a small, 6M matched setting, training and prefill speedups increase with sequence length, decode speed reaches 6.47x compared to that of a standard, optimized
transformer at 32K
tokens, and accuracy reaches 61.31 percent (compared to 55.32 percent) on WikiText.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.AI
Narrative World Model: Narratology-Grounded Writer Memory for Long-Form Fiction
Mohammad Saifullah, Thomas Kornmaier, Taaha Kazi et al. (+3 more)
Long-form fiction writers need
memory that answers multi-hop questions about evolving story state: who knows a secret and when they learned it, whether an event preceded the narration that revealed it, whether a setup paid off, and how a relationship shifted. General-purpose
retrieval and
agent-
memory systems represent entities and facts but not the narratological structure these questions turn on, so they surface the wrong evidence or none at all. We introduce the Narrative World Model (NWM), a writer-
memory system that pairs a narratology-grounded typed temporal-state graph with query-conditioned hybrid
retrieval. To measure
memory rather than the answerer, we read every system through a single held-constant Opus 4.8 reader over only that system's chapter-safe evidence, on a reproducible public corpus and a validated multi-hop
benchmark, and we compare against the strongest existing temporal-knowledge-graph
agent-
memory framework, Graphiti/Zep (Rasmussen et al., 2025). NWM substantially and significantly outperforms this baseline on multi-hop narratological QA across both corpora, and far exceeds GraphRAG and flat
retrieval. The advantage is representational rather than an artifact of extraction: it survives rebuilding the baseline with NWM's own extractor, and traces to its narratology-grounded structure and query-conditioned
retrieval, not to graph size or extractor quality.
Read the full abstract ▾
All authors Mohammad Saifullah, Thomas Kornmaier, Taaha Kazi, Vasu Sharma, Aditya Sanjiv Kanade, Aanand Kumar Yadav
Computation and Language
Prompt Robustness Is Task-Dependent: Comparing Objective and Belief-Style Questions in LLM Evaluation
Sadia Kamal, Arefa Patwary, Anthony Marchiafava et al. (+2 more)
Survey-style evaluations of large
language models often treat a prompted response as a measure of a model's values or beliefs. This assumption is particularly fragile when responses are read as evidence of political values, social attitudes, or beliefs. We ask whether
prompt robustness differs between objective questions with fixed answers and subjective questions that ask for opinions or values. We evaluate four instruction-tuned model families on three objective
datasets (MMLU, ARC, and CulturalBench) and three subjective
datasets (Political Compass Test, ValueBench, and World Values Survey). For each question/statement, we apply multiple types of
prompt changes, such as variations in wording, framing, and format, and measure whether the model gives the same answer across variants. Using a binomial generalized estimating equation, we find significant effects of model,
dataset,
prompt category, and their interactions. The
dataset type effect is also significant, and the interaction between
dataset type and
prompt category is large. These results show that
prompt robustness depends on the question type, the
prompt change, and the model.
Read the full abstract ▾
All authors Sadia Kamal, Arefa Patwary, Anthony Marchiafava, Sagnik Ray Choudhury, Atriya Sen