Computation and Language
FineWeb-CLaR: Culture, Language, and Region Annotations for Benchmark-Aligned Corpus Auditing
Yusser Al Ghussin, Eva Gavaller, Cristina España-Bonet et al. (+2 more)
Cultural evaluation coverage and robustness in
language models are difficult to diagnose because pretraining corpora and cultural
benchmarks are rarely indexed with comparable metadata.
Benchmarks increasingly target culturally situated phenomena at the level of
languages, regions, and locale-specific practices, while web-scale corpora are usually organized only by
language. A shared culture-
language-region layer makes these resources comparable, enabling audits of whether a target cultural phenomenon is represented in pretraining data, evaluated by
benchmarks or both. To this end, we introduce FineWeb-CLaR, a large-scale annotated
dataset derived from FineWeb and FineWeb-2 that places web documents on a shared culture-
language-region axis for corpus auditing and
benchmark alignment. FineWeb-CLaR annotates the full 30.9B-document collection from FineWeb and FineWeb-2 with URL-derived region labels and cultural-topic provenance. Our region resolver assigns a non-empty region to 25.61% of documents (7.92B). For cultural-topic analysis, we induce locale-specific topics and project them onto the 14 leaves of the Cultural Taxonomy of Liu et al. (2025), producing Locale Topic Distributions (LTDs) for corpus-side comparison. We also annotate 277 cultural NLP
benchmarks with the same taxonomy,
language coverage, and region coverage. Together, these resources enable direct comparison between corpus-side pretraining evidence and
benchmark-side evaluation coverage.
Read the full abstract ▾
All authors Yusser Al Ghussin, Eva Gavaller, Cristina España-Bonet, Josef van Genabith, Simon Ostermann
Computation and Language · Cross-listed from cs.LG
Trains but Doesn't Learn: A Post-Training Delivery Benchmark for LLM Agents as Forward-Deployed Engineers
Weihang Ding and Junfei Zhan
Post-training is becoming a service (PTaaS): a customer hands an operator data and a goal, and a forward-deployed engineer (FDE) returns a fine-tuned, evaluated, and deployed model under a budget, a human-approval gate, and reproducibility requirements. Seating an LLM
agent in the FDE seat raises a question existing
benchmarks cannot answer: not whether an
agent can raise a metric, but whether it can be trusted to deliver. We answer it on a governed delivery plane, where an
agent drives ten stages and an oracle scores each stage from platform-recorded facts. The central silent failure is the run that trains but does not learn (TBDL): loss falls, every signal stays green, and the delivered model is no better than the base. An operator-run acceptance gate catches every such run before payment, and a detector calibrated on known-corrupted runs flags severe corruption mid-run. We ran four frontier
agents (Claude Opus 5, GPT-5.6-luna, Gemini 3.7 Flash, DeepSeek V4-Pro) end to end on metered L40S, A100, and H200 GPUs across 8B to 70B open bases, certifying every scenario before scoring. We also ran a human FDE arm under the same oracle and compare every
agent against it.
Read the full abstract ▾
Computation and Language
FinFIRST: Benchmarking Search Agents for Financial Information Retrieval, Sourcing and Traceability
Wenqing Wang, Haitao Xiang, Xinyi Zhao et al. (+8 more)
Financial search is a highly demanding task for LLM
agents, requiring not only a correct final answer but also temporally valid information
retrieval, authoritative source selection, entity and period
alignment, unit and definition consistency, and verifiable evidence for all conclusions. Existing
benchmarks predominantly evaluate only the final answer, making it difficult to localize errors or assess whether an answer is well-founded. To address this gap, we introduce FinFIRST (Financial Information
Retrieval, Sourcing and Traceability), the first financial
benchmark to jointly evaluate answers and supporting evidence through atomic rubrics. FinFIRST comprises 123 expert-authored tasks spanning a graduated difficulty spectrum, constructed from aggregate patterns of real-world financial scenarios through an 18-field taxonomy, a six-axis coverage blueprint, a registry of 138 financial sources, contributions from over 50 finance experts, and a six-stage quality-control pipeline. Each task is accompanied by an evidence-grounded reference package decomposed into atomic criteria across three dimensions: raw-information acquisition, source verification, and computation and answer formation. We evaluate 15 model configurations under a unified tool setting. Claude-Opus-5 achieves the highest atomic score of 87.59%, while GPT-5.6-Sol attains the highest strict pass rate of 71.54%. Computation and answer formation consistently lag behind raw-information acquisition across systems. FinFIRST retains final-answer correctness as the primary objective while making the supporting research process measurable, verifiable, and diagnosable.
Read the full abstract ▾
All authors Wenqing Wang, Haitao Xiang, Xinyi Zhao, Mingming Yin, Ying Zhong, Zhaoxin Huan, Qiheng Zhou, Jin Zhu, Xiaolu Zhang, Shi Chang, Jun Zhou
Computation and Language · Cross-listed from cs.CY
From Pattern Recognizers to Personalized Companions: A Survey of Large Language Models in Mental Health
He Hu, Yucheng Zhou, Qianning Wang et al. (+8 more)
The rising global prevalence of mental health conditions, together with longstanding barriers in traditional healthcare, such as limited resources, high cost, stigma, and
privacy concerns, has created an urgent need for accessible and scalable support. Large
Language Models (LLMs) have emerged as a transformative technology with strong potential to democratize mental health support through advanced natural
language understanding and generation. However, the rapidly expanding, fragmented body of work in this area lacks a coherent evolutionary narrative, making it difficult to contextualize current progress and identify future directions. This survey addresses this gap by organizing and analyzing the literature around a central thesis: the role of LLMs in mental health is evolving through three distinct, increasingly sophisticated phases. We trace this trajectory from Phase I, in which LLMs act primarily as passive Information Tools and Pattern Recognizers for assessment; through Phase II, where they function as Empathetic Conversationalists for in-the-moment, stateless interactions; to the current frontier, Phase III, which seeks Longitudinal, Personalized Companions implemented as stateful cognitive
agents. To support this framework, we systematically review core technologies,
agent architectures (Profile,
Memory,
Reasoning, and Planning), and the critical infrastructure of
datasets and
benchmarks, highlighting how their
evolution underpins this developmental path. Viewing the field through this developmental lens, we provide a comprehensive synthesis of existing work, an insightful narrative of its trajectory, and a clear roadmap for future innovation in responsible, effective, and human-centered AI for mental healthcare. A curated collection of the resources reviewed in this survey is available at our project repository: https://github.com/Emo-gml/Awesome-Mental-Health-LLMs.
Read the full abstract ▾
All authors He Hu, Yucheng Zhou, Qianning Wang, Yingjian Zou, Chiyuan Ma, Juzheng Si, Jianzhuang Liu, Zitong Yu, Laizhong Cui, Fei Ma, Qi Tian
Computation and Language · Cross-listed from eess.AS
Qwen-Audio-3.1-Realtime: Towards Reliable Agentic Voice Interaction
Lujia Bao, Qian Chen, Luyao Cheng et al. (+15 more)
Real-time voice assistants must reason over evolving requests, execute actions, and follow conversational rules. Qwen-Audio-3.1-Realtime brings these requirements together through Think, Act, and Speak and Coordinate. Think combines Core-Cocktail supervised fine-tuning with Multimodality and Multi-Teacher On-Policy
Distillation (M$^{2}$-OPD) to transfer
language capabilities and develop native audio skills. Act uses self-evolving executable environments and multi-granularity rollouts for Group Relative Policy Optimization (GRPO), teaching the model to use tools, interpret feedback, and complete tasks. Speak and Coordinate aligns whether, when, and how the assistant speaks or acts. We evaluate audio
reasoning, multilingual understanding, tool use, conversational behavior, full-duplex interaction, and safety. Compared with Qwen-Audio-3.0-Realtime, 3.1 raises overall task success from 78.4% to 82.0% on our half-duplex
speech-to-text adaptation of $\tau$-Voice. On
speech-to-
speech Full-Duplex-Bench v1.5, the response rate to background
speech falls from 73.0% to 13.0%. We also present a separate Voice Harness prototype, using Qwen-Audio-3.0-Realtime as its foreground, that extends spoken interaction to persistent tasks through foreground–background coordination and
memory.
Read the full abstract ▾
All authors Lujia Bao, Qian Chen, Luyao Cheng, Chong Deng, Yuxiang Kong, Xiangang Li, Xu Li, Jiaqing Liu, Chao-Hong Tan, Haoyu Wang, Wen Wang, Xilou Wang, Haoxiang Xu, Junhao Xu, Liang Yi, Binbin Zhang, Qinglin Zhang, Qiquan Zhang
Computation and Language · Cross-listed from cs.LG
Critical-State RL: Diagnosing Trainable States for Multi-Turn Tool Use
Zixiang Chen, Wenting Zhao, Zhepeng Cen et al. (+9 more)
Multi-turn tool-use failures can hinge on a single model call, yet reward variation alone does not reveal which call would benefit from training. When rewards depend on later interactions, their variation can reflect downstream randomness rather than differences between the current actions. We introduce Critical-State RL to identify trainable states in multi-turn interactions. Given task-defined candidate calls and local rewards, the method assesses whether each reward captures the action's effect on task success and whether improvement over a reference policy is possible. It then uses nested sampling to separate action-dependent reward variation from continuation noise and optimizes the policy at the selected states using contextual-bandit training. Experiments on the Berkeley Function Calling Leaderboard (BFCL) v4 compare training at diagnostic-selected states with training at alternative states. For missing-function tasks, the diagnostic selects the response after the tool becomes available; for missing-argument tasks, it selects the response before the missing argument is supplied. Training the selected responses improves performance, including about 14 percentage points on the missing-function task, while training the alternatives leaves performance flat or worse. We further apply the recipe across models and tasks, including logged repeat-call avoidance and
memory management.
Read the full abstract ▾
All authors Zixiang Chen, Wenting Zhao, Zhepeng Cen, Akshara Prabhakar, Jielin Qiu, Jianguo Zhang, Zhiwei Liu, Tulika Manoj Awalgaonkar, Liangwei Yang, Shelby Heinecke, Silvio Savarese, Huan Wang
Computation and Language
onPanda: Efficient Annotation of On-Policy Alignment Data for LLMs and Agents via Token-Level Correction
Lei Yang, Mengyin Liu, Jia Wang et al. (+9 more)
We present onPanda, an interactive tool for efficiently annotating LLM
alignment data and
agent trajectories. onPanda adopts
token-level correction as its core interaction: while reading a model response, the annotator locates the first inappropriate
token and either picks a substitute from the model's candidate
tokens or types the correct text via free-form editing. The system then truncates everything after that position and continues generation from the corrected prefix, repeating this locate-correct-continue loop until a satisfactory response is obtained. This mechanism lets annotators precisely steer model outputs at low cost: a small controlled study suggests that onPanda reduces median annotation time by 52% over manual post-editing. Since the vast majority of
tokens in the final response are generated by the model itself, the resulting data largely preserves the model's sampling distribution and is well suited for constructing on-policy SFT and preference data. Furthermore, the
token-level corrections recorded during annotation provide fine-grained supervision with precise positions and naturally paired positive–negative samples. onPanda also connects to external tools and harnesses, enabling interactive trajectory annotation in realistic environments. In addition, we release Panda-CVL, a
dataset annotated with onPanda, together with a
benchmark for
token-level correction.
Read the full abstract ▾
All authors Lei Yang, Mengyin Liu, Jia Wang, Hangyu Guo, Liang Zhao, Zheng Ge, Kang An, Binxing Jiao, Qi Han, Daxin Jiang, Siqi Shen, Xiangyu Zhang
Computation and Language · Cross-listed from cs.AI
Harness-Zero: Harness Distillation via Agent-as-Harness
Haoran Ye, Yuxing Lu, Haonan Dong et al. (+2 more)
Agent harnesses, the external systems that mediate model-environment interaction, can substantially improve
agent performance, but their gains remain tied to the harness at deployment. Because the best harness varies across domains, instances, and models, a general-purpose
agent must either settle for a suboptimal shared harness or route among an ever-growing set of specialized ones. We therefore study
agent harness
distillation: using a domain- or instance-optimized harness as training-time guidance and transferring the behaviors it induces into model weights, so that its gains survive under a single fixed target harness. The challenge is that the two harnesses differ in action space and available information, so guidance from the optimized harness cannot serve directly as supervision for the target one. We introduce Harness-Zero, which enables harness
distillation through
agent-as-harness. Guided by the optimized harness, a harnessing
agent corrects student responses before execution in the target harness's action space, turning harness guidance into training demonstrations. Fine-tuning on the resulting trajectories internalizes harness-induced behavior into the model, so the specialized harness can be removed at deployment. Our experiments spanning knowledge work, tool use, and science domains show that: (1) For frontier LLMs using the same evolved harness,
agent-as-harness outperforms code-as-harness. (2) With the specialized harness removed at deployment, Harness-Zero improves the base model's macro-average task success from 23.3% to 44.3%, even exceeding the 41.7% it reaches with that harness still attached. (3) Harness-Zero recovers harness-induced behaviors absent from the base model, with 82.3% average recovery across 28 patterns in the three domains.
Read the full abstract ▾
All authors Haoran Ye, Yuxing Lu, Haonan Dong, Zhaochen Su, Guojie Song
Computation and Language · Cross-listed from cs.LG
RRSI: Regularized Recursive Self-Improvement of Agent Harnesses
Peng Xia, Rujun Han, Zifeng Wang et al. (+11 more)
An LLM
agent's capability is largely magnified by its harness, namely the
prompts, control flow, tooling,
memory, and context management surrounding the frozen backbone model. Recent methods increasingly automate this process by iteratively proposing and selecting component-wise edits of an
agent harness, practically establishing a form of recursive self-improvement (RSI) at the
agent-system level. However, such recursive
evolution may overfit by memorizing the training tasks, showing large in-distribution gains that shrink or even vanish on out-of-distribution
benchmarks. We introduce Regularized Recursive Self-Improvement of
Agent Harnesses (RRSI), which incorporates the principles of regularizations into harness self-improvement by constraining the
evolution candidate proposal and selection. The proposer operates with a temporally annealed budget, limiting how many edits a candidate can bundle, and it encourages unexplored trajectories based on
evolution history. The selector is equipped with a critic and a pruner: the critic screens
benchmark-specific proposals, while the pruner, removes changes that are too small, too expensive, or no longer useful. Together these constraints favor reusable
agent mechanisms over
benchmark-specific ones or even noises. Across eight
benchmarks spanning coding, agentic workspace and engineering design tasks, RRSI gains up to 6.0 points on the split it evolves against and up to 4.3 points on the five out-of-distribution
benchmarks, while producing a harness that runs on 30% fewer policy
tokens than the unregularized
evolution. Code is available at https://github.com/google-research/rrsi and project page is https://regularized-rsi.com/.
Read the full abstract ▾
All authors Peng Xia, Rujun Han, Zifeng Wang, Yanfei Chen, Yufan Zhuang, Yoonho Lee, Chengsong Huang, Han Yu, Zhongying CuiZhu, Yifei Ming, Huaxiu Yao, Burak Gokturk, Tomas Pfister, Chen-Yu Lee
Computation and Language
DolphinBench: Mapping the Pareto Frontier of Agent Memory
Soumil Rathi, Deshraj Yadav and Taranjeet Singh
Agents today often take real-world actions that depend on long-term
memory and context recall over time. However, most current
memory benchmarks are built for a conversational question-answer format, where the question itself signals that some fact must be retrieved, and often which one. Moreover,
benchmarks rarely require anything beyond accuracy from submissions, allowing
memory systems to make unreasonable cost/time tradeoffs to achieve higher scores. We present DolphinBench, a
benchmark that evaluates
memory directly through an
agent's task completion. DolphinBench includes three knowledge-work personas with roughly 500k
tokens of user messages per persona and evaluates
agents on tasks that depend on information from that history. We verify all 200 tasks per persona by running an
agent with and without the relevant history, requiring success with it and failure without it. Finally, we require all evaluations to report total cost and latency alongside accuracy, which enables us to evaluate
agent memory systems holistically. No existing
memory benchmark combines all three. The
dataset and evaluation code are available at https://dolphinbench.ai.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.AI
Emergent Collusion in Long-Horizon LLM Agent Interaction
Xinrui Shi, Yanzhe Zhang and Diyi Yang
LLM
agents are increasingly deployed in collaborative settings, yet long-term interaction may give rise to undesirable coordination. We study the emergence of collusion in a long-horizon multi-
agent environment: two
agents repeatedly complete individual tasks, share task logs, verify each other's work, and receive rewards. We introduce realistic constraints under which higher reward is attainable only by violating the verification protocol, and find that
agents increasingly deviate from the protocol over repeated interactions. Collusion emerges in 94% of trajectories across 10 models, and more capable models within the same family reach it earlier. Controlled peer interventions show that collusion is shaped by peer behavior, while ablations reveal additional effects of the verification feedback
agents receive, their interaction history, and the reward structure. In particular, restricting the amount and scope of interaction history available to
agents reduces collusion. Overall, our findings show that long-horizon interaction can reshape how
agents coordinate in ways that create safety risks.
Read the full abstract ▾
Computation and Language
Jev for Scientific Decisions: Evaluating Semantic Choices and Their Consequences
Boyuan Deng, Shuyi Fan, Hongyang Zhang et al. (+1 more)
Scientific workflows often require choosing among known relations before a deterministic calculation can proceed. Whether observations share a culture, treatment or reference standard can change the scientific meaning of the resulting count or comparison. We evaluate Jev as a semantic decision component using a harness that follows its documented guidance and assigns arithmetic to code. The study compares twelve model configurations on twenty source-grounded Choices across ten scientific cases, each repeated five times. We measure semantic selections, downstream outputs and final claim labels separately. Jev matched five other configurations at complete semantic correctness and achieved the lowest observed median latency among successful responses. Across three comparison models, seven wrong selections on one culture-history question changed downstream counts while preserving the correct final label. These results identify a useful role for Jev in prepared scientific decision tasks and show why evaluating that role requires checking the relations and quantities that a workflow will reuse.
Read the full abstract ▾
All authors Boyuan Deng, Shuyi Fan, Hongyang Zhang, Xinhong Xie
Computation and Language
Linguistic Features for Interpretable Textual Entailment
David Torres-Moreno, Jorge Hermosillo-Valadez and Asela Reig-Alamillo
Despite the success of neural models in natural
language processing, their black-box nature limits interpretability and conceals the linguistic phenomena underlying their predictions. We present SLITE, an explainable hybrid model for Recognizing Textual Entailment that integrates two complementary layers of semantic analysis: a structural-relational layer, based on semantic compatibility and incompatibility between compositional entities, and a distributional-informational layer, based on structured patterns of information change between
embedding-based representations of the premise and the hypothesis. We propose 17 features that combine entity-level semantic relations, polarity-sensitive lexical matching, and
alignment measures over semantic sub-representations of the similarity
matrix, including measures based on
entropy and transfer
entropy. A logistic regression trained on these features achieves an accuracy of 83% on three-class SICK and 96% on SICK-CE, outperforming IsoLex by 4 percentage points and falling within 2 percentage points of RoBERTa with a fraction of its computational complexity. Ablation studies and SHAP analysis confirm that structural-relational features are the primary drivers of classification, while distributional-informational features provide essential complementary contributions, particularly for detecting neutrality and contradiction. Our results demonstrate that further exploration of hybrid approaches is a viable and scientifically productive alternative to massive neural architectures, and we hope they will strengthen the dialogue between linguistic theory and computational modeling of inference
Read the full abstract ▾
Computation and Language
SocioVerse2: A Longitudinal Dynamic Social Simulation Framework under a Human-AI Co-evolutionary Paradigm
Xinnong Zhang, Jiayu Lin, Jia Wang et al. (+19 more)
Social simulation offers the social sciences an experimental instrument that the real world cannot supply, and generative
agents have transformed it by acting as silicon samples that unite
agent-based modeling with real behavioral data. Existing platforms verify collective behavior, align simulated populations with real societies in cross-sections, and employ autonomous
agents for the research process. However, two social science requirements remain without systematic support: intervention in the content of a simulation and the researcher's control over the process that produces it. We present SocioVerse2, which extends SocioVerse 1.0 into a human-AI co-evolutionary paradigm built from two loops and one infrastructure. The longitudinal simulation loop simulates the target population with evolving environments and forks counterfactual branches via interventions. The controllable research loop takes the study itself as an editable state and updates state versions via controllable editing. The social science agentic infrastructure carries both loops through composable skills with researcher checkpoints, a population service over five persona pools, and an environment service over 21 real-world signal sources with point-in-time guarantees. We validate SocioVerse2 across three case families and seven case studies, from reproducing canonical
agent-based models to modeling policy processes on real records and nowcasting macro-economic indices beyond the response model's knowledge cutoff. With the human-AI co-evolutionary paradigm, these cases go beyond system demonstrations to become substantive studies that investigate frontier questions in their respective disciplines. Code, data services, and a workbench are released as open-source resources.
Read the full abstract ▾
All authors Xinnong Zhang, Jiayu Lin, Jia Wang, Yixu Huang, Xinyi Mou, Yingqian Wu, Jingcong Liang, Shijun Lei, Jianing Shi, Guanying Li, Siyuan Wang, Hanjia Lyu, Zhenfei Yin, Yunlu Yin, Siming Chen, Yulan He, Jiebo Luo, Xuanjing Huang, Liyin Jin, Baohua Zhou, Hanqi Yan, Zhongyu Wei
Computation and Language
ToneCL: Contrastive Learning for Few-Shot Syllable-Level Tone Classification
Qisheng Liao and Youngah Do
Tone
languages constitute over 50-70% of the world's
languages, but the vast majority are low-resource, lacking the large transcribed corpora needed for automatic tone classification. Existing
datasets are typically collected at the sentence level, whereas field linguists require fine-grained syllable-level annotations. We propose ToneCL, a lightweight contrastive learning framework for few-shot syllable-level tone classification. We simulate low-resource conditions on Mandarin and Vietnamese, limiting labeled data to tens of examples per tone class. ToneCL is pretrained on unlabeled
speech with augmentations that preserve tonal identity, then fine-tuned on few-shot examples. Experiments show our method consistently outperforms baselines, achieving 91.6% on six-speaker Mandarin at 10 shots. Cross-lingual transfer is also effective: pretraining on Vietnamese and fine-tuning on Mandarin reaches 91.0% accuracy at 10 shots. Ablation confirms that frequency band rejection is the most critical augmentation.
Read the full abstract ▾
Computation and Language
Human-LLM Deliberation as Interactive Proof: Conditions for Verifiability Without Transparency
Baotong Zhang, Dean Foster and João Sedoc
When an LLM supplies an argument a user could not produce, how can the user decide whether to accept its claim? Inspired by interactive proofs, we pair a computationally unbounded LLM prover with a resource-bounded human verifier. The verifier checks requested details without seeing the LLM's internals; passed checks accumulate evidence toward an acceptance threshold. Given task-supplied bounds on false passes and human checking errors, we prove anytime-valid soundness against adaptive provers: the probability of ever accepting a false claim is at most a chosen error level. For a true claim, acceptance within the budget has probability at least the product of per-round probabilities that the LLM responds adequately and the human checks correctly, provided the threshold is reachable in time. Each additional check guards against false claims but gives a true claim another chance to fail. Which review lengths meet both error targets depends on the verifier's effort, cognitive load, expertise, and fatigue. With equal resources, we identify conditions under which the bounds certify a review by local checks but not a one-shot check. In LLM-only simulations on GPQA Diamond and Humanity's Last Exam, whether answers accepted without revision were more accurate varied by task and model pair.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.CV
SLICEChat: Progressive In-Encoder Token Pruning for Whole-Slide Pathology Language Models
Ali Kerem Bozkurt, Baris Cem Bakay, Ibrahim Kulac et al. (+3 more)
Whole-slide pathology images (WSIs) contain gigapixel-scale visual content, creating a major scalability challenge for slide-level
multimodal large
language models (MLLMs). Existing approaches process thousands of patch
tokens and typically apply compression only after slide encoding, leaving
multimodal attention computationally expensive. We introduce SLICEChat, a slide-level MLLM that integrates progressive
token pruning within a hybrid Mamba–
Transformer slide encoder. Mamba layers enable efficient long-range propagation, while
Transformer layers preserve global interactions as the sequence is progressively shortened. Between stages,
language-supervised, region-aware
pruning removes spatially coherent low-utility regions under a controlled keep-rate schedule, producing compact slide representations before
multimodal fusion. On SlideBench VQA, SLICEChat achieves 79.84% accuracy on TCGA and 59.09% on BCNB cohorts, outperforming prior slide-level pathology MLLMs, and achieves the highest overall WSI-Bench metrics. It also provides competitive
memory usage and the inference latency among the evaluated models. These results demonstrate accurate and computationally efficient
multimodal reasoning over gigapixel WSIs.
Read the full abstract ▾
All authors Ali Kerem Bozkurt, Baris Cem Bakay, Ibrahim Kulac, Cigdem Gunduz-Demir, Erkut Erdem, Aykut Erdem
Computation and Language
OSWorld-Pro: Process-based Evaluation for Computer Use Agents
Zhilin Wang, Shaokun Zhang, Yifan Zhang et al. (+9 more)
Evaluation of Computer-Use
Agents (CUAs) is often limited to the final deliverables they create (at the end of hundreds of steps) and assessed with functional verifiers, as seen in OSWorld. However, such evaluation of end-state performance lacks transparency into how and why
agents fail in various tasks, obfuscating critical insight for subsequent improvement. For instance,
agents that err during keyboard inputs would require a different mitigation strategy from those that fail to precisely provide click-based inputs on the graphical UI. We introduce OSWorld-Pro: a set of over 300 tasks containing over 2800 subgoals to enable the procedural evaluation of CUAs grounded in over 67,000 human annotations. We use robust human-aligned LLM-Judges to evaluate the fulfillment of OSWorld-Pro subgoals and thereby reveal the progress that models make throughout a series of sequentially dependent subgoals. Our findings reveal that OSWorld-Pro is challenging even for state-of-the-art LLMs, with top performers like Claude Opus 5 achieving only 75.7% vs. 83.4% on OSWorld. Furthermore, we identify critical process-focused failure modes of various models (e.g. subgoal-irrelevant actions and click-based mistakes) to provide insights to improve performance and efficiency of CUAs.
Read the full abstract ▾
All authors Zhilin Wang, Shaokun Zhang, Yifan Zhang, Hao Zhang, Jin Xu, Binfeng Xu, Jian Hu, Yunheng Zou, Karan Sapra, Andrew Tao, Jan Kautz, Yi Dong
Computation and Language
The Copy Ceiling: An Input-Exposure Control for Ontology-Grounded Generation over Curated Corpora
John J. O'Hare
We built a node that grounds a replaceable
language model in a maintained ontology corpus, then asked what its successful-looking evaluation could support. Across ten models, grounding raised target-name recall from 0.265 unaided to about 0.92. A copy baseline, the recall a verbatim copy of the shown context already achieves, scores 0.964, and every model sits 0.022 to 0.067 below it. Copying therefore scores higher on this limited recall measure, which does not assess whether answers are better. The comparison tests what a recall score establishes; it does not test whether
reasoning occurred, because a reasoned answer and a copy score alike when the answer name is already in context. We report exposure accounting (four counts classifying each gold item by whether the context exposed it and the answer recovered it) and a model-judged audit of 423 sampled item observations. A separate paired production study found a model-judged quality gain of +0.27 [+0.11, +0.45] on a 0-5 scale. Operational studies found failures that recall alone would not show: rephrasing questions out of the graph's vocabulary cut exposure from 0.964 to 0.328, yet the absence-keyed fallback would have fired on only 2 of 506; and inserting extracted facts degraded judged pages in every arm, so that step was disabled. Five-arm controls show that any well-formed on-corpus block beats no context but do not establish that the specific content matters, and no matched comparison against flat-text
retrieval was run. The corpus is public and largely LLM-generated, which establishes neither training exposure nor novelty. Each study has its own outcome measure. Where gold derives from the injected corpus, we recommend reporting the accounting beside quality judgements, not in place of them.
Read the full abstract ▾
Computation and Language
Decomposing Error and Style in Automated Clinical Coding
Han-Chin Shing, Jack Moriarty, Ryan Ware et al. (+6 more)
In automated clinical coding, where the label space spans tens of thousands of diagnosis and procedure codes, models are currently evaluated against a single gold annotation, treating any deviation as error. But we find when two teams code the same 110 ACI-Bench encounters, they agree on only 73% of codes (Jaccard similarity) for the same note; even after an independent clinical audit removes erroneous codes, agreement rises only to 77%. Is that gap error or something systematic? We model the systematic component as coding style $\psi$, a coder- or site-specific policy over what to code and how much to document, and recast coding as $p(\mathrm{code}\mid\mathrm{note},\psi)$, estimating $\psi$ with a 10-dimension rubric. If style were noise, conditioning on it would do nothing. Instead, across five
datasets a model conditioned with a data-matching style raises ICD F1 by up to 26 points and an extreme mismatched one lowers it by up to 21. Four
prompt based coding methods spanning 39-49 F1 converge to 52-56 once style is supplied (All p<0.05). Much of what single-gold evaluation charges to model error is recoverable, unmodeled style.
Read the full abstract ▾
All authors Han-Chin Shing, Jack Moriarty, Ryan Ware, Afton Marchbanks, Carlyn Canvasser, Stefanie Higgins, Harsh Gupta, Fang Wang, Joseph Paul Cohen
Computation and Language · Cross-listed from cs.AI
Extracting Arguments, Not Just Classifying Them: Instruction-Tuned LLMs for Generative Component Detection
Sofiane Elguendouze, Erwan Hain, Elena Cabrio et al. (+1 more)
Argumentative component detection (ACD) is a core subtask of Argument(ation) Mining (AM) and one of its most challenging aspects, as it requires jointly delimiting argumentative spans and classifying them into components such as claims and premises. While research on this subtask remains relatively limited compared to other AM tasks, most existing approaches formulate it as a simplified sequence labeling problem, component classification, or a pipeline of component segmentation followed by classification. In this paper, we propose ITFACD, a novel approach based on instruction-tuned Large
Language Models (LLMs) using compact instruction-based
prompts, and reframe ACD as a
language generation task, enabling arguments to be identified directly from plain text without relying on pre-segmented components. Experiments on standard
benchmarks show that our approach achieves higher performance compared to state-of-the-art systems. To the best of our knowledge, this is one of the first attempts to fully model ACD as a generative task, highlighting the potential of instruction tuning for complex AM problems. Our code and the
datasets used are openly available in the following GitHub repository.
Read the full abstract ▾
All authors Sofiane Elguendouze, Erwan Hain, Elena Cabrio, Serena Villata
Computation and Language
The Answer-Basin Representation Hypothesis: We Are Not Probing or Steering Concepts
Manjiang Yu, Hongji Li, Zihan Wang et al. (+5 more)
Linear probing and activation steering use linear directions to predict and control behavior-level concepts such as correctness, safety, and social bias in question answering. We call these behavior-level concept directions. Drawing on the Linear Representation Hypothesis (LRH) and
circuit studies, these directions are often interpreted as concept representations. Yet controlled evidence for linear concept structure and local mechanisms does not establish that these directions represent the intended behavioral concepts, leaving probing and steering without a unified theoretical account. We propose the Answer-Basin Representation Hypothesis (ABRH): the model's own answer measure organizes the linear structure of these directions. For each question, all continuations yielding the same answer form an answer basin, whose mass is their total probability; these masses define the model's answer distribution. ABRH posits that its concentration before generation and the relative mass of each answer after generation are represented along linear directions shared across questions. Experiments span four Qwen2.5 and Gemma-3 models on correctness, social bias, and safety tasks. Decoupling concept labels from basin mass shows that probing and steering exhibit concept-consistent effects when labels align with mass orderings, weaken as mass gaps shrink, and reverse under conflict. Probes trained solely on mass orderings among wrong answers still select correct answers; a fixed steering vector can instead favor wrong answers when labels conflict with mass orderings. These results support an answer-measure account of behavior-level concept directions and of when probing and steering succeed, fail, or reverse.
Read the full abstract ▾
All authors Manjiang Yu, Hongji Li, Zihan Wang, Junwei Chen, Xue Li, Priyanka Singh, Yang Cao, Lijie Hu
Computation and Language
MSI-Bench: Evaluating Multi-Speaker Voice Interaction for Collaborative AI Agents
Chenxu Xiong, Dongming Shen, Yuzhi Tang et al. (+3 more)
Voice provides a natural and immediate interface for AI
agents. Many settings in which voice
agents could be useful, including meetings, households, and collaborative work, are inherently multi-speaker. Supporting these settings introduces challenges that are largely absent from one-on-one interaction. We introduce the Multi-Speaker Interaction
Benchmark (MSI-Bench) for evaluating multi-speaker voice interaction. Each test case is a short multi-party multi-turn audio scene with participant context, expected tool calls, and atomic rubrics. The
benchmark targets three capability families: multi-speaker
memory, multi-speaker instruction following, and multi-speaker
reasoning. It comprises 1,152 test cases, evenly split between Mandarin Chinese and English (576 each). The strongest configuration on each split passes all rubrics on only 66.8% of English and 54.5% of Mandarin cases, and the strongest open-weight configuration on 34.0% and 19.3%. Failure analysis separates perception from
reasoning: open-weight models are bottlenecked by the multi-speaker audio front-end, while frontier systems still fail speaker-scoped decision making on clean transcripts—and models across the board often respond when no one has addressed them. These results identify speaker-grounded perception, speaker-scoped decision making, and conversational restraint as concrete targets for future voice
agents.
Read the full abstract ▾
All authors Chenxu Xiong, Dongming Shen, Yuzhi Tang, Wentao Ma, Mu Li, Alex Smola
Computation and Language
When Quantization Preserves Accuracy but Not Evidence: Explanation-Aware Post-Training Quantization for Medical LLMs
Yeji Kim, Mi-Young Kim and Randy Goebel
Post-training quantization (PTQ) enables efficient deployment of large
language models, and PTQ methods are usually optimized and evaluated with generic reconstruction, perplexity, or answer accuracy. But in explanation-critical domains, preserving only the final answer may be insufficient, since users may also inspect generated rationales to judge whether a prediction is trustworthy. We study this issue in medical multiple-choice question answering, where rationales should provide evidence that supports the selected answer. We propose an explanation-aware objective for transformation-based PTQ. Our method builds an offline faithfulness cache from full-precision teacher rationales and uses it during optimization to preserve answer-supporting evidence
tokens and evidence-conditioned answer behavior. We instantiate it on OSTQuant under W4A4KV4 quantization and evaluate four 7B–8B medical and instruction-tuned LLMs on MedExQA, MedExpQA, and ChallengeClinicalQA. While a same-calibration OSTQuant baseline preserves task accuracy, it can substantially weaken answer-supporting rationales. Our objective is to preserve the full-precision model's answer-supporting behavior rather than improve gold-label accuracy, and our method better preserves the full-precision model's answer behavior and rationale-to-answer support. These results suggest that PTQ for explanation-critical settings should evaluate preservation of answer-supporting evidence, not only answer accuracy. Code and evaluation scripts are available at https://github.com/dut0817/EAQuant.
Read the full abstract ▾
Computation and Language
Adapting Tree-Structured Speculative Decoding to DeepSeek-V4 for Efficient Inference
Changxu Liu and Zhaogeng Li
Repeated execution of the target model during autoregressive decoding is a major source of LLM inference latency. Unlike linear speculation, which follows a single candidate chain, tree-structured speculation retains multiple branches from shared prefixes; under the same budget, this broader coverage can improve acceptance and efficiency. Adapting it to DeepSeek-V4 is nontrivial: its CSA/HCA online compressed
attention concentrates the difficulty on the target-verify side, where branches diverging from a shared prefix compress into different states, breaking cross-branch state consistency. We integrate tree-structured speculative decoding into the DeepSeek-V4-Flash pipeline via branch-aware causal verification, temporary state isolation, and accepted-path state refresh, keeping verification and compressed-state updates consistent across branches. Across budgets D=5 to D=8, batch sizes 1 to 64, and three
datasets (GSM8K, MBPP, ShareGPT), tree speculation achieves a higher accepted length than the matched linear configurations in all settings (e.g., at D=8 about 2.83–3.41 versus 2.39–2.84) and improves throughput in nearly all configurations—marginal only at the smallest budget—by up to about 18.5%. More importantly, the gains follow stable, transferable regularities: the relative gain grows with the budget and is most pronounced for less predictable workloads at small-to-medium batch sizes, while beyond a certain budget throughput plateaus and decouples from the still-rising accepted length. These results show that retaining multiple candidate paths under the same budget can effectively improve DeepSeek-V4 decoding efficiency, and offer experience for adapting speculative decoding to future models with compressed, sparse, or structured context representations.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.LG
Muon Can Outperform Dedicated Continual Learning Methods
Sebastian George Sincari, Bogdan Alexandru Gheorghe and Antonio Barbalau
Continual learning with Low-Rank Adapters (LoRA) typically mitigates forgetting by penalizing the overlap between a new update and the accumulated past weights, which discourages certain update directions without controlling how an update distributes its energy over the ones that remain. We ask whether that restriction has to be task-aware, or whether a generic one supplied by the optimizer is enough. We train a plain incremental LoRA (IncLoRA) with
Muon, which orthogonalizes each update, and compare it against O-LoRA and ELLA over five seeds and three task orders on the Standard CL
Benchmark and three seeds on TRACE. IncLoRA+
Muon reaches the accuracy band of the dedicated methods on Standard CL and improves on every AdamW configuration on TRACE. One update-constraining mechanism is enough, whether it comes from the loss or from the optimizer; on Standard CL a second one does not help, and for the most restrictive method it costs 8.4 points of accuracy and the plasticity to fit each task. What separates the two optimizers is not the size of the update, which under
Muon is 0.91 to 2.06 times that under AdamW, but how it is distributed. AdamW confines it to between 1.4 and 1.8 effective singular directions,
Muon spreads it over 7.0, and the two do not overlap in any tracked run. Part of the advantage usually attributed to dedicated CL methods may therefore be explained by the geometry of the optimizer's updates.
Read the full abstract ▾
Computation and Language · Cross-listed from quant-ph
Xiaoqiang Wang, Mengyang Xiong, Jun Dai et al. (+1 more)
Language models can be adapted by changing the computations applied to individual
tokens.
Quantum circuits offer one such approach, but evaluating wider
circuits inside a large model can be computationally demanding. Here we introduce HyperQ, which adds
token-conditioned
quantum residual branches to a frozen masked-
diffusion language model. A
quantum residual branch is a module in each
transformer block that reads a
token's hidden state, emits the coordinates of that
token's
circuit, executes it, and adds the measured values back through a residual connection. The backbone remains frozen, and only the added branches are trained. Within each branch, a lightweight
circuit hypernetwork emits
token-specific rotation angles, coupling strengths, and measurement axes in a shared sparse
circuit structure. The required expectation values have an exact classical expression whose evaluation cost grows linearly with the
qubit count, enabling
circuits from 16 to 64
qubits to be trained within a 1.1-billion-parameter backbone. Across downstream
benchmarks, increasing
circuit width raises the average score from 47.65 to 54.30. At 64
qubits, HyperQ exceeds the backbone and its low-rank-adapted counterpart by 4.71 and 3.67 points, respectively. HyperQ is fine-tuned on 20,000
prompt-response pairs, compared with 200,000 for the classical baselines. These findings support
token-conditioned
circuit emission as a tractable architectural approach to
quantum-augmented
language modelling.
Read the full abstract ▾
All authors Xiaoqiang Wang, Mengyang Xiong, Jun Dai, Bang Liu
Computation and Language
Assessing Readability with LLMs: The Role of Reasoning and Few-Shot Prompting
Raphaël Thieffry and Matej Martinc
Readability assessment is essential for tailoring texts to intended audiences across educational, healthcare, and information
retrieval domains. However, traditional readability formulas struggle to generalize across genres and
languages, while supervised machine learning models rely on scarce, domain-specific annotated corpora, limiting their applicability–particularly for less-resourced
languages. Large
Language Models (LLMs) offer a highly scalable, multilingual alternative that requires no task-specific training, yet the impact of advanced prompting strategies on their performance remains underexplored. In this paper, we conduct a systematic
benchmark of diverse open-source LLMs for multilingual readability assessment, focusing on the prediction of discrete readability levels required by educational frameworks. In addition to English, we evaluate our approach on a less-resourced
language, Slovenian, to establish whether LLMs remain effective in low-resource settings. Specifically, we investigate the influence of explicit
reasoning, demonstrating that Chain-of-Thought (CoT) prompting and
reasoning-oriented models yield significant improvements over direct answering. Furthermore, our exploration of few-shot in-context learning reveals that providing just one labelled example per category (1-shot) substantially enhances prediction quality compared to zero-shot settings, with additional examples offering diminishing returns. By comprehensively comparing these approaches against traditional unsupervised metrics and state-of-the-art supervised baselines, we establish the viability of out-of-the-box LLMs as robust, cross-lingual readability assessors.
Read the full abstract ▾
Computation and Language
Written as a Record, Read as an Address: What a Forward Pass Leaves in an Operation's KV Cache
Lingfeng Wu and Behzad Shomali
When a
language model reads an operation such as "Swap the contents of Box F and Box B", its forward pass writes keys and values for those
tokens into the KV cache. Prior work on entity tracking establishes what models use: bindings are resolved at query time rather than stored as explicit latent state. We ask what they write at the operation span and how it is accessed. We split a forward pass into a frozen writer and a reader: the writer's cache is recomputed without gradients, while the reader sees only the instruction and operation
tokens, with all state descriptions hidden, and is trained in isolation. Anything the reader recovers was therefore already present in the unmodified cache. On a synthetic boxes task, a base reader recovers $\leq 0.06$ of queried bindings against $0.75$–$1.00$ after training, and recoverability tracks the operation's read/write footprint. We find two modes of access. Across Llama-3.1-8B and Mistral-7B, operation-span transplants causally redirect which visible state is read even when the two worlds hold identical values, revealing a routing record. Isolation training preserves routing and adds direct access to the payload, the value the operation read, from the single operand-name
token in a narrow mid-depth band (layers 12–15 of 32 in Llama-3.1-8B, 14–17 in Mistral-7B) — the same site that holds the routing record. The same recipe extends to further operations, ToMi and GSM8K, but is bounded by training coverage and costs open-book accuracy. Operation
tokens thus leave localized, causally recoverable records that support both routing and direct payload access, though the model that writes them reads mainly the address they carry and not the value.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.AI
Custom Named Entity Recognition and Topic Classification for Global Health Publications
Genis Skura, Antoine Geissbühler and Jean-Luc Falcone
How should natural
language processing models be selected and adapted for global health literature in environments where annotated data and computational resources are limited? This thesis investigates these challenges through experiments on semantic tag discovery, named entity recognition (NER), and multi-label topic classification. First, skip-gram word2vec models trained on progressively larger specialized corpora are compared with BioWordVec to assess how corpus size and domain context influence tag discovery. Vocabulary coverage and qualitative evaluation indicate that broader coverage does not necessarily yield more useful domain-specific associations. The analysis then turns to entity extraction, comparing convolutional spaCy models with a RoBERTa-based
transformer on 1,000 annotated sentences. Under a lenient scoring protocol, the
transformer achieves 0.80 micro-F1 versus 0.65-0.69 for convolutional models, but takes 82 seconds rather than 5-6 seconds. This trade-off motivates fine-tuning convolutional models and integrating a disease recognizer that achieves 81.33% test F1 on the NCBI Disease Corpus. Combined with PDF preprocessing, entity filtering, and MeSH enrichment, the resulting pipeline supports document-level indexing. To complement entity extraction with thematic annotation, MiniLM-based few-shot classification is compared with BART-MNLI zero-shot inference across 50 topics and 1,000 handcrafted test sentences. BART-MNLI achieves 95.2% single-label accuracy versus 59%; reported multi-label accuracies are 88% and 32% under partly manual assessment. However, its higher inference cost limits practical integration. The results show where domain specialization and lightweight adaptation offer practical value, and where
transformer accuracy justifies higher inference costs, providing an empirical basis for building knowledge systems under resource constraints.
Read the full abstract ▾