Computation and Language · Cross-listed from cs.AI
Large AI Models in Dental Healthcare: From General-Purpose Systems to Domain-Specific Foundation Models
Sema Helali, Lina Abu Nada, Sausan Al Kawas et al. (+3 more)
Background: Oral diseases affect nearly 3.5 billion people worldwide, yet the comparative clinical potential of large-scale AI models in dentistry remains poorly understood. Three distinct model categories have emerged:
language-generative models, discriminative
vision foundation models, and dental-specific foundation models, with no unified review examining their relationships and collective limitations. Methods: Following PRISMA-ScR guidelines, we systematically searched four databases (PubMed, Google Scholar, Scopus, arXiv), screened independently by two reviewers. After applying inclusion/exclusion criteria, 97 studies (2020-2026) were included. We propose a two-dimensional classification framework organizing models by architectural paradigm and dental specialization degree. Results:
Language-generative models excel at text-based tasks (clinical
reasoning, licensing exams, patient communication) but show inconsistent performance on image-dependent diagnostics. Adapted SAM and CLIP variants achieve strong tooth
segmentation and lesion detection results. Dental-specific models (DentVFM, DentVLM, OralGPT) demonstrate strongest performance on complex
multimodal tasks. Integrated pipelines consistently outperform single-model approaches. A data asymmetry is observed: dental-specific pretraining concentrates almost entirely in the
vision domain, reflecting scarce large-scale dental text corpora. Conclusions: General-purpose and dental-specific models play complementary roles; the most effective systems combine both within structured pipelines. Safe autonomous deployment requires resolving three persistent barriers:
hallucination in generative models, limited annotated dental
datasets, and absent standardized clinical evaluation
benchmarks.
Read the full abstract ▾
All authors Sema Helali, Lina Abu Nada, Sausan Al Kawas, Alaa Abd-Alrazaq, Faleh Tamimi, Rafat Damseh
Computation and Language
The Ghost Annotator: a Framework to Explore Human Label Variation in Content Moderation through Conformal Prediction
Mirko Lai, Alessandra Urbinati, Simona Frenda et al. (+2 more)
Current research primarily focuses on model performance, while comparatively less
attention has been devoted to uncertainty estimation, particularly in settings where LLMs are increasingly used to generate annotated data. We introduce a framework combining conformal prediction with Collaborative Filtering-style annotators' representation to model LLM behavior in relation to human annotators and to analyze patterns of agreement and disagreement. Using Non-Conformity Scores, we introduce the Ghost Prediction metric and the Ghost Annotator representation to quantify cases in which model predictions diverge from all available human annotations. We compute cosine similarity measures to explore differences in model behavior across sociodemographic axes. We evaluated four LLMs of different size and families across four content moderation
datasets. Our finding shows that while we find that all models uncertainty increases with annotator disagreement, larger models tend to be more confident in the classification of texts that are not aligned with any human annotation. Finally, the Ghost Annotator framework reveals a consistent and robust pattern of demographic misalignment, suggesting a structural bias likely rooted in pretraining corpora.
Read the full abstract ▾
All authors Mirko Lai, Alessandra Urbinati, Simona Frenda, Fabiana Vernero, Marco Antonio Stranisci
Computation and Language
WRIT: Write-Read Intensive Trajectory Synthesis for Multi-Turn User-Facing Agents
Hengrui Gu, Xiaotian Han and Kaixiong Zhou
Multi-turn user-facing
agents must infer user intent from incomplete requests, collect missing information through dialogue and tools, and execute valid actions. A training trajectory records this process as an interleaved sequence of user messages,
agent responses, tool calls, etc. Synthesizing sufficiently complex trajectory has become a central route to train
agents: existing pipelines often increase difficulty by composing multiple user requests into longer tasks, producing write-intensive trajectories that train sequential execution. We argue that a single write decision can itself be difficult when the
agent must gather and compare substantial read-tool evidence before its arguments become identifiable, a challenge that write-intensive data alone cannot address. Guided by this insight, we propose WRIT (\uline{W}rite-\uline{R}ead \uline{I}ntensive \uline{T}rajectory Synthesis), a pipeline for synthesizing multi-turn
agent training trajectories along two complexity axes: the number of write decisions in a task and the evidence burden of each individual decision. WRIT first generates write-intensive and read-heavy tasks. It then diversifies user behavior instructions to reflect realistic conversational variation, and finally simulates
agent-user interactions in an executable environment to produce complete training trajectories. The resulting data trains
agents not only for longer task execution, but also for robust, evidence-grounded decision making under high information load. With only 2K synthesized trajectories, a 4B model trained on WRIT outperforms GPT-5.1 no-think on $\tau^2$-bench and substantially reduces inference-time
token usage, showing that compact SFT data can convert part of expensive test-time
reasoning into efficient
agent behavior.
Read the full abstract ▾
Computation and Language
Linear Probes Detect Task Format, Not Reasoning Mode in Language Model Hidden States
Subramanyam Sahoo, Vinija Jain, Aman Chadha et al. (+1 more)
Linear probing of large
language model (LLM) hidden states is widely used to claim that models learn distinct representations for different
reasoning types. We test this by probing Qwen3-14B on three
benchmarks spanning the classical trichotomy: LogiQA 2.0 (deductive), ARC-Challenge (inductive), and $\alpha$NLI (abductive). At layer 32 of 40, linear probes achieve 100% cross-validated accuracy with well-separated geometry (intrinsic dimensionalities: 20.6, 28.5, 33.6; convex hull contamination $\leq$1.5%). However, this separation is entirely driven by format confounds. Residualizing source identity, option count, and response length reduces accuracy to chance. Trace-anchor similarity indicates largely shared
reasoning across tasks (42.5% agreement vs.\ 33.3% chance), and causal steering with random controls ($n=20$) shows no functional link between geometry and
reasoning mode ($p=0.286$). Thus, high probe accuracy reflects task format rather than computational structure, motivating routine format deconfounding in mechanistic interpretability.
Read the full abstract ▾
All authors Subramanyam Sahoo, Vinija Jain, Aman Chadha, Divya Chaudhary
Computation and Language
Adaptive Latent Agentic Reasoning
Dongwon Jung, Peng Shi, Yi Zhang et al. (+2 more)
Large
reasoning models improve performance by generating extended chain-of-thought (CoT)
reasoning, but this behavior becomes inefficient when applied to LLM
agents. Current LLM
agents often generate verbose textual
reasoning at every decision step and allocate
reasoning effort nearly uniformly across turns, leading to substantial inefficiency in multi-turn agentic trajectories. We propose Adaptive Latent Agentic
Reasoning (ALAR), a dual-mode framework that uses compact latent
reasoning for routine turns and selectively escalates to explicit chain-of-thought when deeper deliberation is needed. ALAR learns latent
reasoning by using the
agent's actions as supervision anchors and is further optimized to use latent
reasoning when it is sufficient for task success and reserve explicit CoT for harder decisions. Experiments on agentic search and tool-use
benchmarks show that ALAR maintains comparable or better task accuracy while substantially reducing generated
tokens by up to 43.6% in search and 84.6% in tool use. These results demonstrate that ALAR improves the accuracy-efficiency trade-off of LLM
agents by reducing unnecessary textual
reasoning while preserving explicit deliberation for harder decision steps.
Read the full abstract ▾
All authors Dongwon Jung, Peng Shi, Yi Zhang, Junshan Zhang, Muhao Chen
Computation and Language · Cross-listed from cs.AI
When Helping Hurts and How to Fix It: Multi-Agent Debate for Data Cleaning
Chirag Parmar, Akshat Mehta, Henglin Wu et al. (+2 more)
When does multi-
agent debate help data cleaning, and when does it hurt? Across three
benchmarks, four model families, and over 6,000 task-condition pairs, we find debate's effect reverses sign: it degrades generation across all four models (-1.6 to -15.5pp) through critique-induced confusion (CIC), hallucinated Critic feedback that the Generator accepts uncritically, yet improves error detection (+27.4pp F1, d=1.0). We derive a debate benefit condition: debate helps when the probability of rescuing a wrong output (Critic verification odds weighted by fixability) exceeds the probability of destroying a correct one. A factorial experiment proves
adversarial separation is essential: self-verification with identical tools fails, while a separate Critic with code-execution grounding and evidence-gated generation produces the first debate configuration to significantly exceed single-
agent on a generative task (+5.3pp, p<0.05). The condition correctly predicts all nine task types and generalizes with zero false positives across 19 published comparisons in seven domains.
Read the full abstract ▾
All authors Chirag Parmar, Akshat Mehta, Henglin Wu, Jagadish Ramamurthy, Shweta Medhekar
Computation and Language
Economy of Minds: Emerging Multi-Agent Intelligence with Economic Interactions
Zhenting Qi, Huangyuan Su, Ao Qu et al. (+13 more)
How can a population of
agents self-orchestrate and self-adapt into stronger collective intelligence without centralized control? Inspired by Friedrich Hayek's economic theory of decentralized coordination in
markets, we study this question through an
agent economy in which
agents compete via auctions for the right to act, exchange payments, and accumulate wealth from environmental rewards. These simple local incentives induce decentralized credit assignment, driving global planning without explicit orchestration or communication protocols. The population evolves through economic selection: effective
agents accumulate wealth and are mutated via exploitation, while ineffective ones go bankrupt and are replaced via exploration. Using modern Large
Language Model (LLM) based
agents, we show that, initialized with weak
agents, the
agent economy produces emergent multi-step
reasoning strategies and outperforms stronger monolithic baselines across five agentic tasks, including mathematical
reasoning, financial research, scientific research, accelerator design, and distributed-system optimization. Our results suggest a new path to multi-
agent intelligence: rather than engineering coordination, we can design decentralized incentive structures under which it automatically emerges.
Read the full abstract ▾
All authors Zhenting Qi, Huangyuan Su, Ao Qu, Chenyu Wang, Yu Yao, Han Zheng, Kushal Chattopadhyay, Guowei Xu, Zihan Wang, Weirui Ye, Vijay Janapa Reddi, Ju Li, Paul Pu Liang, Himabindu Lakkaraju, Sham Kakade, Yilun Du
Computation and Language
Fixing FOLIO and MALLS: Verified Annotations and an LLM-assisted Framework to Focus Human Relabeling
Andrea Brunello, Cristian Curaba, Luca Geatti et al. (+3 more)
Accurate translation from Natural
Language to First-Order Logic (NL-to-FOL) underpins neurosymbolic AI systems and Natural
Language Inference (NLI), making the quality of NL-to-FOL
benchmarks essential—yet these
datasets have never been rigorously audited. Our first contribution is to present a systematic human inspection of the validation split of FOLIO and a subset of MALLS test instances, finding that approximately 42.5% and 42% of entries, respectively, contain incorrect FOL formalizations (i.e., ground truth labels), with additional rates of ambiguous NL sentences (17.8% and 51%) and incorrect NLI labels in FOLIO (8.4%). Our second contribution is to develop and release corrected ground truths for such
datasets, showing that annotation errors distort model evaluation on a reference
benchmark task: testing three state-of-the-art LLMs (Gemma~4 31B-it, Qwen3-30B-A3B, and GPT-4o-mini) with the corrected ground truths yields accuracy gains from +11 to +23 percentage points. Motivated by these findings, we propose an LLM-based framework to support humans in manual reviewing NL-to-FOL
datasets. By directing reviewers toward the most error-prone instances, we empirically show that it is possible to achieve 90%
dataset accuracy after reviewing fewer than 20% of instances, compared to over 76% required by unguided review. We release all human-verified annotations and the code for our framework.
Read the full abstract ▾
All authors Andrea Brunello, Cristian Curaba, Luca Geatti, Michele Mignani, Angelo Montanari, Nicola Saccomanno
Computation and Language · Cross-listed from cs.IR
Do Neural Retrievers Prefer Certain Documents? Evidence of Learned Relevance Priors
Francisco Valentini, Edgar Altszyler and Martin Fajcik
Neural retrievers are trained to estimate query-document relevance from annotated query-document pairs. Yet annotation protocols may not purely reflect relevance: they select only a subset of documents for labeling, and this selection can favor certain document types over others. We investigate whether supervised bi-encoder retrievers implicitly learn a document-level relevance prior: a query-independent signal encoded in their representation space as a side effect of training on annotated data. We estimate this prior by training simple classifiers on frozen document
embeddings and evaluate three state-of-the-art retrievers across multiple IR
benchmarks. We find that supervised neural retrievers encode relevance priors that generalize to unseen documents and are consistent across models. These priors create a findability gap: documents with lower prior are systematically harder to retrieve, even when genuinely relevant. This effect appears in supervised dense retrievers but is weaker and less consistent in BM25, and it persists under controlled matched-document comparisons. Using LLM-based explanations, we find that judged-relevant documents tend to be comprehensive, self-contained summaries of mainstream topics, while niche, fragmentary, or highly technical content is often left unjudged. Retrievers internalize this bias, ranking documents with these favored features higher than documents that lack them, independently of their actual relevance. Our findings expose a structural limitation of supervised
retrieval: models trained on annotated data do not just learn relevance, but also the implicit document preferences in their training data.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.AI
Traj-Evolve: A Self-Evolving Multi-Agent System for Patient Trajectory Modeling in Lung Cancer Early Detection
Sihang Zeng, Matthew Thompson, Ruth Etzioni et al. (+1 more)
Modeling patient trajectories from longitudinal electronic health records (EHRs) requires
reasoning over sparse, noisy, and long-context
multimodal sequences. Existing LLM-based multi-
agent systems address context length but process patients in isolation, failing to mirror how clinicians leverage accumulated experience from similar prior cases. We present Traj-Evolve, a self-evolving multi-
agent system with two complementary evolving mechanisms. First, an Experience Pool (ExPool) acts as a non-parametric
memory, indexing rejection-sampled
reasoning traces to retrieve similar patients as few-shot contexts. Second, multi-
agent reinforcement learning (MARL) via reward-ranked fine-tuning parametrically optimizes inter-
agent and
agent-
memory collaboration. A leave-one-out cross-
retrieval strategy unifies the two, aligning training- and inference-time behavior under
retrieval augmentation. On a lung cancer prediction task utilizing up to five years of
multimodal EHRs, Traj-Evolve outperforms 9 strong baselines on the overall population and a challenging never-smoker population. Analysis of the evolving dynamics highlights three key findings: (1) expanding the ExPool shifts optimal
retrieval from diverse to specific samples; (2) under MARL, the manager
agent's prediction loss converges quickly while the worker
agents' temporal
reasoning continues to benefit from more verified patients; and (3) the two mechanisms are complementary on the predicted risk, where ExPool improves specificity while MARL improves sensitivity.
Read the full abstract ▾
All authors Sihang Zeng, Matthew Thompson, Ruth Etzioni, Meliha Yetisgen
Computation and Language
Translating Classical Poetry into Modern Prose
Chalamalasetti Kranti and Sowmya Vajjala
We built a
dataset for the task of poem-to-prose translation from 13th-17th Century Telugu classical poetry to contemporary Telugu and English prose, which we call Padyam2Gadyam. The
dataset consists of 600 poems and their human-verified Telugu and English prose translations. We evaluated 2 machine translation systems and 5 contemporary Large
Language Models (LLMs) on their ability to do zero-shot poem-to-prose translation into Telugu and English using this
dataset. Our results indicate that while the general purpose LLMs are better than the machine translation systems, there are systematic issues with the generation and evaluation of prose translation in both
languages across systems.
Read the full abstract ▾
Computation and Language
Do Value Vectors in Deep Layers Need Context from the Residual Stream?
Muyu He, Yuchen Liu, Qingya Huang et al. (+1 more)
The success of the
transformer architecture is in large part due to its use of
attention layers. An
attention layer follows the standard neural network paradigm: it takes the residual stream as input and thereby produces context-dependent query, key, and value vectors. However, we find that model performance meaningfully improves when deeper layers learn only a context-free value vector to preserve the original
token information, without drawing on any context from the residual stream. When the model has access to this context-free value vector, adding back the context-dependent component provides little additional benefit for aggregate
benchmark performance. Such context-free value vectors can be stored as sparse model parameters, eliminating the need to recompute or persistently cache these values. Through systematic ablations on the key design choices for such context-free value vectors, we propose Bank of Values (BoV), a new way of computing value vectors in
attention by learning a lookup table of
token-specific value vectors for each of the last third of layers. Across 135M and 780M models, BoV improves validation loss over standard
attention and, at 780M, the average score across 21
benchmarks, matching the previous best method that adds
token information to the value vector with less compute and
memory.
Read the full abstract ▾
All authors Muyu He, Yuchen Liu, Qingya Huang, Li Zhang
Computation and Language
Topics as Proxies for Sociodemographics: How Conversational Context Affects LLM Answers
Vera Neplenbroek, Gabriele Sarti, Arianna Bisazza et al. (+1 more)
When large
language models (LLMs) are used in high-stakes scenarios, such as legal, medical and financial advice, even a single conversation history is enough to drive differences in outcomes between users. Prior work has demonstrated that this results in outcome disparities between sociodemographic groups, with some groups receiving more advantageous outcomes than others. In this work, we demonstrate that LLMs actually struggle to infer user sociodemographics from a single conversation history and that although there are disparities between sociodemographic groups, they are minimal in magnitude. To investigate what is the main driver of disparities between users, we compare user sociodemographics to a range of (psycho)linguistic features of conversations, including conversation topic,
emotions, and readability. We find that conversation topics are most predictive of LLM-generated advice within a conversational context, which, to some extent, function as proxies for sociodemographic groups and often affect advice in unpredictable ways. This is cause for concern and highlights the need for future research to better understand the effect of conversational context on LLM outputs in high-stakes scenarios.
Read the full abstract ▾
All authors Vera Neplenbroek, Gabriele Sarti, Arianna Bisazza, Raquel Fernández
Computation and Language
On the Persistent Effects of Lexicality in Large Language Models
Hammad Rizwan, Muhammad Umair Haider, Nishant Subramani et al. (+3 more)
Representations extracted from large
language models (LLMs) play an important role in many downstream applications. However, the structure of these representations is often influenced by lexical overlap rather than semantic content. Our understanding of the relationship between this lexical influence and semantic content, and its implications for downstream tasks, remains limited. In this work, we investigate representations to quantify the effect of lexical overlap relative to semantic content. We consider several
adversarial semantic stress tests and further connect our findings to the information theory perspective. We find that lexical influence extends across the depth of models, consistently across architectures, training regimes, and objective functions, including the models trained for semantic similarity. Moreover, we observe a mid-depth region in which both lexical and semantic signals degrade simultaneously, indicating a transitional regime where representations are poor for both surface form and meaning. We further demonstrate the effect of lexical influence on downstream uses of LLMs using summarization and model editing as a case study.
Read the full abstract ▾
All authors Hammad Rizwan, Muhammad Umair Haider, Nishant Subramani, Mona T. Diab, A.B. Siddique, Hassan Sajjad
Computation and Language
Greener Than Humans? Environmental Attitudes in Large Language Models
Stefanie Kunkel, Tilman Hartwig, Marcus Voss et al. (+2 more)
Large
language models (LLMs) are increasingly used in sustainability-related decision support, reporting, and public communication, yet little systematic evidence exists on the environmental attitudes embedded in their outputs. This paper develops a
benchmark for evaluating environmental cognition, affect, and behavioural recommendations in LLMs and applies it to 31 widely used proprietary and open-weight models. Drawing on questions from established environmental awareness surveys and additional sustainability-related behavioural measures, we compare LLM responses 1) among models and 2) between models and human survey
benchmarks from Germany. We assess their robustness across prompting conditions. We find that many LLMs align more closely with environmentally progressive attitudes than the average survey respondent, exhibiting higher levels of environmental affect and cognition and recommending behaviours associated with substantial potential CO2 reductions. At the same time, we observe no systematic relationship between sustainability-oriented responses and model origin, size, or release context. However, models exhibit contextual sensitivity, controlled by persona-based prompting and show sycophantic shifts mirroring user-specified ideological positions, which raises concerns about steerability and normative reliability in real-world deployments. Our findings provide a reusable evaluation framework for assessing sustainability-related value
alignment in LLMs and highlight the importance of governance, transparency, and critical oversight as AI systems become increasingly embedded in sustainability transformations and public decision-making.
Read the full abstract ▾
All authors Stefanie Kunkel, Tilman Hartwig, Marcus Voss, Emma K. Schütt, Angelika Gellrich
Computation and Language · Cross-listed from cs.IR
Attention Calibration for Position-Fair Dense Retrieval
Andrianos Michail, Elias Schuhmacher, Juri Opitz et al. (+2 more)
Dense
retrieval compresses a passage into a single vector, but this compression is positionally skewed: early content dominates the
embedding, and
retrieval degrades when the relevant span appears later. Prior work proposed an inference-time method that counteracts this skew by equalizing the pooling
token's
attention across passage segments. However, (i) it redistributes
attention at a fixed strength, (ii) it forces the pooling
token's
attention to itself to a fixed basket-level mass despite substantial variation across layers and architectures, and (iii) its effect on
retrieval has not been evaluated. We introduce a strength coefficient that interpolates between uncalibrated and fully equalized
attention, together with an efficient implementation that reduces peak calibration
memory overhead from 5-7 GiB to under 1 MiB. Across three
embedding models and two pooling schemes, moderate calibration provides a better
retrieval trade-off than full equalization. We introduce a variant that preserves the pooling
token's self-
attention mass and redistributes only the remaining mass. On a position-aware
retrieval benchmark spanning 10
languages and 31 domains, a configuration selected on English FineWeb-PosQ and transferred without tuning reduces position sensitivity in all 16 evaluated length-quartile, model, and
retrieval-setting combinations, by up to 43% relative, while improving nDCG@10 by up to 4.8% relative and leaving general
retrieval effectiveness on NanoBEIR essentially unchanged. Calibration runs at indexing time, adding no query-time latency. We release our code at github.com/impresso/fair-sentence-
transformers
Read the full abstract ▾
All authors Andrianos Michail, Elias Schuhmacher, Juri Opitz, Simon Clematide, Rico Sennrich
Computation and Language · Cross-listed from cs.LG
Filter, Then Reweight: Rethinking Optimization Granularity in On-Policy Distillation
Yuying Li, Leqi Zheng, Yongzi Yu et al. (+6 more)
On-Policy
distillation (OPD) in large
language models is shifting from full-trace KL supervision toward more selective training paradigms. Recent OPD methods increasingly focus on selecting which trajectories to learn from, which
tokens are most informative, and which supervision signals are most reliable. Motivated by this trend, we rethink optimization granularity of OPD and propose \fireicon\ FiRe-OPD (Filter, then Reweight), which jointly adjusts supervision signals at both trajectory and
token levels. In details, FiRe-OPD first filters trajectories to remove low-quality rollout samples, and then applies soft reweighting within the retained trajectories to emphasize informative
tokens. Compared with hard
token selection, FiRe-OPD leverages a soft-weighting mechanism to effectively mitigate information loss and enhance optimization stability, thereby achieving finer-grained OPD optimization. We validate the effectiveness of FiRe-OPD across strong-to-weak, single-teacher, and multi-teacher settings, and demonstrate its superiority over recent
token-level OPD methods ( (e.g., +6.25 on AIME 2024 in strong-to-weak, +18.81 on Miner in multi-teacher). Our code is available at https://github.com/YuYingLi0/FiRe-OPD.
Read the full abstract ▾
All authors Yuying Li, Leqi Zheng, Yongzi Yu, Wenrui Zhou, Xuchang Zhong, Xing Hu, Jing Jin, Hangjie Yuan, Tao Feng
Computation and Language · Cross-listed from cs.CV
AdaCodec: A Predictive Visual Code for Video MLLMs
Haowen Hou, Zhen Huang, Zheming Liang et al. (+8 more)
Video is temporally redundant: adjacent frames usually share most objects, background, and layout. Yet existing
video multimodal large
language models (
video MLLMs) usually encode each sampled frame as an independent RGB image, causing visual
tokens to repeat content already present in earlier frames. This suggests a more direct
video interface: send a full reference frame only when the scene cannot be predicted well from prior context, and otherwise transmit a compact description of inter-frame changes. We call this interface a predictive visual code, and instantiate it for
video MLLMs as AdaCodec. AdaCodec spends full visual
tokens on a reference frame only when its conditional predictive cost is high; otherwise, it encodes inter-frame changes, including motion and prediction residuals, as compact P-
tokens. Across all eleven
benchmarks, AdaCodec improves over the Qwen3-VL-8B per-frame RGB baseline at a matched visual-
token budget. Even at $1/7$ the budget, AdaCodec with 32k
tokens surpasses the 224k baseline on all long-
video benchmarks; on five general-
video benchmarks, it raises the average score while substantially cutting time-to-first-
token from 9.26s to 1.62s.
Read the full abstract ▾
All authors Haowen Hou, Zhen Huang, Zheming Liang, Qingyi Si, Chenglin Li, Shuai Dong, Kele Shao, Ruilin Li, Dianyi Wang, Nan Duan, Jiaqi Wang
Computation and Language · Cross-listed from cs.AI
ClinEnv: An Interactive Multi-Stage Long Horizon EHR Environment for Agents
Yuxing Lu, Yushuhong Lin, Wenqi Shi et al. (+4 more)
Clinical practice is not the selection of an answer from enumerated options: a physician gathers heterogeneous information incrementally and commits to sequential, irreversible decisions under uncertainty. Static
benchmarks cannot probe and existing interactive medical
benchmarks each compromise on at least one of them. We present ClinEnv, an interactive
benchmark that evaluates LLMs as attending physicians over real inpatient admissions under a paradigm we term Longitudinal Inpatient Simulation. Each case is automatically constructed into an ordered sequence of decision stages; at every stage the model must actively query four specialized
agents before committing to medications, procedures, and diagnoses. ClinEnv scores both what the model decides, through deterministic ontology-grounded matching, and how it gathers information. Across seven models, the strongest reaches only 0.31 decision F1, and outcome quality is sharply decoupled from process quality. Difficulty concentrates in management decisions and later stages, where models recover discharge diagnoses far more reliably than management actions (0.51 vs. 0.17 F1) and continue to issue redundant queries as cases progress. ClinEnv makes this information-acquisition gap, invisible to outcome-only evaluation, directly measurable.
Read the full abstract ▾
All authors Yuxing Lu, Yushuhong Lin, Wenqi Shi, J. Ben Tamo, Xukai Zhao, Jinzhuo Wang, May Dongmei Wang
Computation and Language
From Layers to Submodules: Rethinking Granularity in Replacement-Based LLM Compression
Elia Cunegatti, Marcus Vukojevic, Erik Nielsen et al. (+1 more)
Post-training compression of Large
Language Models (LLMs) removes entire architectural components, either deleting them or replacing them with fitted modules. Existing replacement-based methods share two design constraints: full-layer granularity and contiguous selection. We argue that this is overly restrictive: in fact, redundancy in pretrained
transformers is not confined to contiguous regions, nor does it evenly distribute between
Attention and FeedForward outputs, implying that different strategies best approximate different submodule types and that removable components need not cluster within contiguous depth ranges. Based on this intuition, we introduce SubFit (Submodule-level Fitted residual replacement), which compresses LLMs at the submodule level:
Attention and FeedForward submodules are selected non-contiguously, and each receives its own lightweight fitted residual bypass. SubFit operates post-training and requires only calibration data. Across ten LLMs (five base, five instruction-tuned), five sparsity levels from 12.5% to 37.5%, and four replacement-based baselines, SubFit achieves the best aggregate perplexity-accuracy trade-off across the evaluated sparsity levels, with larger gains under aggressive compression. At 25% sparsity, it retains 84.6% of dense downstream accuracy and incurs 2.42x perplexity degradation, against 81.6% and 4.34x for the strongest baselines, while delivering measurable inference speedup and KV-cache savings. Code is available at https://github.com/eliacunegatti/SubFit.
Read the full abstract ▾
All authors Elia Cunegatti, Marcus Vukojevic, Erik Nielsen, Giovanni Iacca
Computation and Language
HERO'S JOURNEY: Testing Complex Rule Induction with Text Games
Anshun Asher Zheng, Kanishka Misra, David I. Beaver et al. (+1 more)
We introduce HERO'S JOURNEY, a
benchmark for rule induction in goal-directed episodic tasks, where
agents must infer hidden rules from demonstrations and act on them through multi-step execution. HERO'S JOURNEY covers eight tasks across attribute and procedural induction families, each with four structural rule forms, controllable lexical grounding, and identifiability conditions. Evaluating state-of-the-art LLMs, we find that models show evidence of rule induction, but the ability is limited and uneven across tasks. Meanwhile, process execution adds an execution bottleneck for models, whereas surface semantics has minimal effect. Induction-specific steering methods improve performance on attribute tasks but show no reliable gains on procedural tasks, suggesting the gap in procedural induction remains an open challenge.
Read the full abstract ▾
All authors Anshun Asher Zheng, Kanishka Misra, David I. Beaver, Junyi Jessy Li
Computation and Language
SN-WER: Script-Normalized WER for Multi-Script Indic ASR Evaluation
Priyaranjan Pattnayak
Word Error Rate (WER) is the dominant metric for automatic
speech recognition (ASR), but it can overestimate errors when references and hypotheses encode the same words in different scripts. This issue is common in
multilingual settings where ASR models may emit romanized text. We propose Script-Normalized WER (SN-WER), a training-free, evaluation-only scoring method that transliterates both reference and hypothesis text into a
language-specific canonical script before computing WER. We evaluate SN-WER on 5 Indic
languages, 2
datasets, and 3 ASR models. On curated FLEURS data, SN-WER reduces inflated model gaps by up to 12%, while on noisier Common Voice data the reductions are smaller or inconsistent, indicating genuine recognition weaknesses rather than only script mismatch. Controlled stress tests show a 67% attenuation of artificial romanization-induced WER inflation, while lexical-substitution controls show near-identical sensitivity to semantic errors, with Delta SN-WER / Delta WER approximately 1.09. SN-WER is robust to transliterator choice, normalization changes, and shows low
token-collision rates below 0.1% in the evaluated Indic setting. We argue that SN-WER should be reported alongside WER and CER as a companion metric for script-insensitive ASR evaluation, especially when transcripts feed downstream search, indexing, or
multilingual LLM pipelines.
Read the full abstract ▾
Computation and Language
Transferable Self-Harm Surveillance from Emergency Department Triage Notes Using an Evidence-Augmented Machine Learning Approach
Liuliu Chen, Gowri Rajaram, Eleanor Bailey et al. (+5 more)
Self-harm is a major public health concern, but current surveillance relying on hospital presentations is inadequate due to the low sensitivity of diagnostic codes. Emergency Department (ED) triage notes, recorded at the initial point of contact, provide a succinct summary of presentations and an opportunity to identify self-harm. We developed a three-stage approach, augmenting traditional machine learning with large
language model-based screening and evidence extraction to detect self-harm in ED triage notes. We assessed model transferability across three Australian hospitals. Our approach showed AUPRCs of 0.887 +/- 0.016 and 0.884 +/- 0.012 during internal and external validation. Prospectively, it achieved AUPRC of 0.881 +/- 0.008 at the development site, and 0.879 +/- 0.012 and 0.816 +/- 0.015 at two external sites without site-specific retraining. A key advantage of the approach is that it enables identification of the primary self-harm method with an accuracy of 95%, supporting more granular surveillance beyond binary classification.
Read the full abstract ▾
All authors Liuliu Chen, Gowri Rajaram, Eleanor Bailey, Katrina Witt, Michelle Lamblin, Jo Robinson, Mike Conway, Vlada Rozova
Computation and Language
SimSD: Simple Speculative Decoding in Diffusion Language Models
Junxia Cui, Haotian Ye, Runchu Tian et al. (+9 more)
Diffusion large
language models (dLLMs) have recently emerged as a promising alternative to autoregressive (AR) LLMs, offering faster inference through parallel or blockwise decoding. However, their masked
language modeling formulation remains incompatible with standard
token-level speculative decoding, one of the most effective acceleration techniques for AR models. In AR decoding, the causal mask preserves temporally valid
token-level contexts, enabling a target model to verify multiple drafted
tokens in a single forward pass. In contrast, dLLMs rely on mask
tokens and bidirectional
attention, causing the effective context to change across denoising steps and preventing direct
token-level speculative verification. To bridge this gap, we propose a simple but effective speculative decoding
algorithm for
diffusion language models, named SimSD, which mainly adopts a plug-and-play masking strategy that equips dLLMs with temporally valid
token-level contexts for speculative decoding. Our method explicitly introduces reference
tokens from draft-model predictions and designs an
attention mask that regulates their interaction with current-step
tokens, allowing dLLMs to compute valid logits for drafted
tokens in a single forward pass. This restores the key verification ability provided by causal masking in AR models while preserving the parallel decoding advantages of dLLMs. The proposed method is training-free and can be flexibly integrated with other acceleration techniques such as KV cache and blockwise decoding. Experiments on SDAR-family dLLMs across four
benchmarks show that our method achieves up to 7.46x higher decoding throughput while maintaining and even improving average generation quality.
Read the full abstract ▾
All authors Junxia Cui, Haotian Ye, Runchu Tian, Hongcan Guo, Jinya Jiang, Haoru Li, Chaojie Ren, Yiming Huang, Kaijie Zhu, Zhongkai Yu, Kun Zhou, Jingbo Shang
Computation and Language
SkillHarm: Lifecycle-Aware Skill-Based Attacks via Automated Construction
Yuting Ning, Zhehao Zhang, Yash Kumar Lal et al. (+8 more)
Agent skills occupy a privileged position in the
agent workflow, as
agents are expected to implicitly follow and execute them, rendering third-party skills a vulnerable attack surface. Existing studies have revealed unsafe
agent behaviors induced by skill-based attacks, but they primarily evaluate poisoned skills within a single task execution and enumerate harms through ad hoc risk lists. To bridge these gaps, we introduce SkillHarm, a
benchmark of skill-based attacks across the skill-use lifecycle, paired with a systematic taxonomy of skill-relevant risks. SkillHarm evaluates two attack scenarios: Fixed-Payload Poisoning (FPP), where a fixed poisoned skill package directly compromises any task session that invokes it, and Self-Mutating Poisoning (SMP), where an initially benign execution silently mutates persistent skill content, deferring harm until a subsequent reuse. It further defines 12 risk types based on the
agent workflow component targeted by the harm: data pipelines, system environments, and
agent autonomy. To instantiate these attacks at scale, we build AutoSkillHarm, an automated construction pipeline with coding
agents driven by natural-
language harnesses. The resulting
benchmark contains 879 attack samples across 71 skills. Experiments show that current
agents remain vulnerable with attack success rates up to 86.3% in FPP and 69.3% in SMP. Our analysis further reveals a latent risk: many apparent attack failures stem from the
agent failing to engage with the poisoned file rather than genuine resistance, and current defenses still fail to reliably mitigate the threat.
Read the full abstract ▾
All authors Yuting Ning, Zhehao Zhang, Yash Kumar Lal, Boyu Gou, Junyi Li, Weitong Ruan, Chentao Ye, Rahul Gupta, Diyi Yang, Yu Su, Huan Sun
Computation and Language · Cross-listed from cs.AI
SafeSteer: Localized On-Policy Distillation for Efficient Safety Alignment
Hao Li, Jingkun An, Zijun Song et al. (+8 more)
Aligning Large
Language Models (LLMs) with human values often degrades their general capabilities, termed the
alignment tax. Existing methods mitigate this by balancing dual objectives, which heavily rely on massive general-purpose data or auxiliary reward models. In this paper, we argue that, because safety features are inherently sparse within the output distribution,
alignment requires localized modifications rather than global trade-offs. To this end, we propose SafeSteer, which performs on-policy
distillation confined to safety
tokens. First, we construct a safety teacher via activation steering. Based on this teacher, we develop a safety
token selection
algorithm. Consequently, SafeSteer restricts the reverse KL penalty to these
tokens during training to preserve general capabilities. Experimental results across diverse models show that our SafeSteer achieves a superior trade-off between safety and general capability compared with existing methods, attaining strong safety performance on seven safety
benchmarks with only minimal degradation on five general capability
benchmarks. Notably, SafeSteer requires only 100 harmful samples without using any general-purpose data, less than 1% of what previous baselines used, considerably reducing
alignment cost. More details are on our project page at https://anjingkun.github.io/SafeSteer.
Read the full abstract ▾
All authors Hao Li, Jingkun An, Zijun Song, Pengyu Zhu, Rui Li, Hao Wang, Wendi Feng, Yesheng Liu, Lijun Li, Jin-Ge Yao, Lei Sha
Computation and Language
FigSIM: A Dataset for Fine-grained Suicide Severity and Figurative Language in Suicide Memes
Liuliu Chen, Elise R. Carrotte, Brian E. Chapman et al. (+2 more)
Suicide memes are memes used to express suicide-related thoughts or comment on suicide-related issues. Suicide memes are increasingly common on social media, yet remain poorly understood and potentially harmful. There is an urgent need to better understand their characteristics and to develop appropriate content moderation strategies that limits users' exposure to potentially harmful content. Currently, the absence of annotated
datasets of suicide memes remains a key barrier to developing and evaluating automated moderation approaches. In this paper, we introduce FigSIM, the first
dataset designed for fine-grained analysis of suicide memes. The
dataset consists of 1049 memes, each annotated for (1) fine-grained suicide severity levels, (2) figurative phenomena (e.g., metaphors), and (3) suicide-related content (e.g., suicide method depiction). We
benchmark 16 unimodal and
multimodal models across three tasks: figurative
language, suicide severity, and suicide-related content detection. Overall, FigSIM demonstrates that suicide memes pose unique challenges for both modeling and content moderation. Analysis revealed biases, such as underprediction of higher suicide severity levels, especially for figurative memes. The
dataset (including splits used for analyses) is publicly available. Content Warning: This paper contains suicide-related content that may be triggering.
Read the full abstract ▾
All authors Liuliu Chen, Elise R. Carrotte, Brian E. Chapman, Jo Robinson, Mike Conway
Computation and Language
When Rating Scales Fall Short: LLM-Assisted Discovery of ADHD Signals in Turkish Teacher Narratives
Baris Karacan, Irem Aktar Songur, Ahmet Ozaslan et al. (+1 more)
Attention Deficit Hyperactivity Disorder (ADHD) is one of the most common neurodevelopmental disorders in childhood, and its diagnosis relies on assessments combining clinician judgment with standardized rating scales and reports from parents and teachers. While structured instruments such as the Conners' Teacher Rating Scale-Revised Short Form (CTRS-R:S) quantify ADHD-related behaviors, teachers also provide open-ended narratives that may contain complementary signals not captured by structured assessments. However, it remains unclear to what extent teacher narratives encode signals overlooked by rating scales. In this study, we analyze de-identified Turkish teacher evaluation forms collected during clinical ADHD assessments, including both CTRS-R:S scores and open-ended teacher narratives. We compare predictive signals from structured scores and narrative text and identify cases where structured assessments fail to clearly distinguish ADHD from non-ADHD students while narrative-based models capture distinct behavioral patterns. Notably, these cases show minimal overlap with those missed by the narrative model, suggesting that structured and narrative information encode complementary signals. To interpret these differences, we apply a large
language model (LLM)-assisted theme discovery pipeline that reveals distinct
attention, behavioral, and family-related patterns, highlighting the potential of natural
language processing (NLP) to uncover clinically relevant signals from teacher narratives and to complement traditional ADHD screening tools.
Read the full abstract ▾
All authors Baris Karacan, Irem Aktar Songur, Ahmet Ozaslan, Elvan Iseri
Computation and Language
CRAM: Centroid-Routing and Adaptive MoE for Multimodal Continual Instruction Tuning
Jun-Tao Tang, Zhen-Hao Xie, Yu-Cheng Shi et al. (+1 more)
Multimodal Large
Language Models (MLLMs) unify heterogeneous
vision-
language tasks under a shared generative framework via instruction tuning, yet real-world deployment demands continuous capability expansion, making
Multimodal Continual Instruction Tuning (MCIT) essential. Existing methods either update all tasks with a shared parameter set or allocate dedicated modules for each new task. Shared updates force heterogeneous tasks to compete, causing forgetting of learned capabilities. Conversely, isolated expansion prevents interference but severely limits parameter efficiency over long task streams. To address this dilemma, we propose CRAM (Centroid-Routing and Adaptive MoE). Specifically, by isolating task-specific patterns into independent modules, CRAM mitigates catastrophic forgetting across tasks. To further boost parameter efficiency, we utilize adaptive-rank instantiation to identify the capability gap between existing expert capability and new task demands, and dynamically allocate only the necessary parameters. To ensure stable reuse among tasks, centroid-guided routing recognizes and activates existing experts' capabilities, while an orthogonality penalty confines new updates to task-specific directions, preventing re-learning general capability. Extensive experiments across diverse
benchmarks demonstrate its superiority over existing methods. Code is available at https://github.com/LAMDA-CL/EMNLP2026-CRAM.
Read the full abstract ▾
All authors Jun-Tao Tang, Zhen-Hao Xie, Yu-Cheng Shi, Da-Wei Zhou
Computation and Language
Not What, But How: A Framework for Auditing LLM Responses across Positioning, Generalization, Anthropomorphism, and Maxims
Siddhesh Milind Pawar, Sarah Masud, Haneul Yoo et al. (+2 more)
Large
language models (LLMs) are being increasingly used to answer subjective, information-seeking questions, where users are sensitive to how responses are communicated, not just whether the answers are correct. Existing LLM evaluations for subjective cultural queries largely focus on factual correctness, ignoring how the response is framed. To this end, we introduce FRANZ, an automated FRAmework for respoNse characteriZation to conduct communicative audit of LLM responses along four dimensions: cultural positioning, use of generalizing
language, anthropomorphic cues, and adherence to conversational maxims. To enable this evaluation, we contribute SQUARE - a corpus of 376k subjective questions sourced from 57 subreddits, and mapped to 7 countries and 19 question categories. We demonstrate FRANZ's applicability by scoring responses from three open-weight LLMs. We observe that LLMs show statistically significant differences in the frequency with which they employ each response characteristic. Unlike single-dimensional audits, FRANZ reveals that insider positioning and anthropomorphism are positively coupled, with the degree of coupling varying by country, providing a diagnostic lens for identifying framing divergences.
Read the full abstract ▾
All authors Siddhesh Milind Pawar, Sarah Masud, Haneul Yoo, Alice Oh, Isabelle Augenstein