Computation and Language
Fast & Faithful Function Vectors
Minh An Pham, Anton Segeler, Thomas Wiegand et al. (+4 more)
Function vectors (FVs) are task representations elicited during in-context learning that can be used to steer Large
Language Models (LLMs). However, design choices in their formulation remain underexplored. In this work, we study the impact of varying FV definitions for instructions along two degrees of freedom:
attention head selection and steering. For head selection, using gradient-based attributions with Layer-wise Relevance Propagation (LRP) substantially improves efficiency as well as accuracy. For FV steering, applying it in a distributed manner yields a higher accuracy compared to simple aggregation. Our code is publicly available.
Read the full abstract ▾
All authors Minh An Pham, Anton Segeler, Thomas Wiegand, Wojciech Samek, Sebastian Lapuschkin, Patrick Kahardipraja, Reduan Achtibat
Computation and Language
Boosting Self-Consistency with Ranking
Maria Marina, Daniil Moskovskiy, Sergey Pletenev et al. (+3 more)
Self-consistency improves large
language models by sampling multiple
reasoning paths and selecting the most frequent answer, but majority voting often fails to recover correct answers that are already present among the samples. We address this limitation with Ranking-Improved Self-Consistency (RISC), which reformulates answer selection in self-consistency as a ranking problem. Instead of relying on a single uncertainty or confidence signal, RISC uses a lightweight LambdaRank model to score candidate answers with five carefully designed features that capture answer frequency, semantic centrality, and
reasoning-trace consistency. We evaluate RISC on three
datasets under a range of test-time budgets. Across
datasets, RISC consistently achieves a better accuracy-efficiency trade-off than standard self-consistency and strong baselines, with particularly large gains on question answering
benchmarks. Further analysis shows that the proposed features are individually useful and, more importantly, complementary, highlighting the value of learning to combine multiple informative signals for test-time answer selection.
Read the full abstract ▾
All authors Maria Marina, Daniil Moskovskiy, Sergey Pletenev, Mikhail Salnikov, Alexander Panchenko, Viktor Moskvoretskii
Computation and Language · Cross-listed from cs.LG
In-Context Graphical Inference
Zehua Cheng, Wei Dai and Jiahao Sun
Marginal inference in discrete graphical models forces a choice between exactness and scalability: exact
algorithms are intractable for high-treewidth graphs, while iterative approximations (Belief Propagation, variational methods) sacrifice convergence guarantees on frustrated
topologies. We argue that this dichotomy stems from a mismatched inductive bias: iterative methods abandon the sequential elimination structure that makes exact inference correct. We introduce In-Context Graphical Inference (ICG-I), an autoregressive Graph
Transformer that restores this structure by mimicking Variable Elimination with learned, Tensor- Train-compressed intermediate factors, paired with a Dirichlet output layer and Weighted Conformal Prediction for calibrated, distribution-free coverage guarantees under topological shift. We prove that TT compression errors propagate at most lincarly through the autoregressive chain, that the Dirichlet-Multinomial loss is a proper scoring rule, and that WCP maintains coverage with a quantifiable degradation under estimated density ratios. We conducted intensive experiments to evaluate ICG-I and achieved state-of-the-art performance across all
benchmarks. ICG-I reduces MAE from 0.041 (best baseline) to 0.020 on standard instances and achieves 0.048 on N=500 frustrated
spin glasses where BP diverges entirely.
Read the full abstract ▾
Computation and Language
Imbuing Large Language Models with Bidirectional Logic for Robust Chain Repair
Zehua Cheng, Wei Dai, Jiahao Sun et al. (+1 more)
Autoregressive chain-of-thought (CoT)
reasoning in large
language models (LLMs) is fundamentally forward-directed: each step conditions only on prior
tokens. This unidirectional inductive bias renders even capable models susceptible to error snowballing, wherein a single logical or arithmetic mistake in an early step irreversibly corrupts the entire
reasoning chain. We introduce Teleological
Reasoning Infilling (\TRI{}), a training framework that endows decoder-only
transformers with a native goal-conditioned bridging capability. The key insight is to reframe erroneous
reasoning segments as fill-in-the-middle (FIM) tasks: given a verified prefix premise $P$, a verified downstream milestone $S$, and the original query $Q$, the model must synthesise the logical bridge $M$ that connects $P$ to $S$ rigorously and completely. To achieve this with standard causal architectures, we introduce a Prefix-Suffix-Middle (PSM) sequence rearrangement with three non-overlapping sentinel
tokens, enabling $M$ to attend to both $P$ and $S$ without any structural modification to the self-
attention mechanism. Training proceeds in two stages: (i) Supervised Fine-Tuning (SFT) on symbolically verified $(P, S, M)$ triples extracted from formal mathematics corpora, and (ii) Direct Preference Optimisation (DPO) with a deterministic symbolic verifier (Lean 4 / Python) as the sole reward oracle, eliminating LLM-judge sycophancy. At inference, TRI operates as a
surgical repair module within a dual-system loop: a causal draft model generates an initial trace, the verifier pinpoints failures, and TRI infills only the damaged segment, leaving verified sections intact. Comprehensive experiments on three
benchmarks demonstrate that TRI achieves state-of-the-art performance across all tasks, while reducing per-problem
token expenditure by 31.2%.
Read the full abstract ▾
All authors Zehua Cheng, Wei Dai, Jiahao Sun, Thomas Lukasiewicz
Computation and Language · Cross-listed from cs.LG
Validity Threats for Foundation Model Research
Gunnar König, Martin Pawelczyk, Ulrike von Luxburg et al. (+1 more)
Controlled experiments are the backbone of machine learning research, but at the scale of modern foundation models, they have become prohibitively expensive. Instead, the community increasingly relies on research strategies that approximate the ideal experiment at a fraction of the cost: proxy experiments and scaling laws, observational studies with publicly available models, and single-run designs that leverage variation within individual training runs. In this work, we argue that there is no free lunch when approximating large-scale experiments on a compute budget. Specifically, savings in compute come at the cost of validity threats – hidden and sometimes untestable assumptions that, when violated, can invalidate research claims. To help navigate such threats, we propose an evaluation framework that casts foundation model research as a causal inference problem. Within this framework, we evaluate different research strategies through four types of validity adapted from the empirical social sciences – statistical, internal, external, and construct validity. We find that each strategy comes with a characteristic validity profile: proxy experiments trade external and construct validity for statistical and internal validity; observational studies face confounding and effect heterogeneity; and single-run designs are strained by interference between treated units. This analysis reveals several validity threats that have received insufficient
attention in the literature. Overall, our evaluation framework provides researchers with a practical toolkit for scrutinizing validity threats in foundation model research~designs.
Read the full abstract ▾
All authors Gunnar König, Martin Pawelczyk, Ulrike von Luxburg, Sebastian Bordt
Computation and Language
TaDA: Calibrated Probe Gating for Task-Domain LoRA Merging
Huy Quoc To, Fuyi Li, Guangyan Huang et al. (+1 more)
Combining a task LoRA adapter with a domain LoRA adapter into a single unified model is a practical yet largely unexplored challenge. Existing methods treat both adapters as symmetric peers, applying uniform weights across all layers. We argue that task and domain adapters exhibit a consistent depth-dependent asymmetry across
transformer architectures. Domain dominance increases with layer depth, while shallower layers retain stronger task-relevant signals. Motivated by this observation, we propose $\textbf{TaDA}$ ($\textbf{Ta}$sk-$\textbf{D}$omain LoR$\textbf{A}$ Merging), a training-free
algorithm that exploits this structure through calibrated probe-guided per-layer gating and per-component subspace-aware merging. The gating assigns individual weights per layer and projection type using a probe signal proved invariant to adapter weight magnitude. The merging discards conflicting singular directions before combining the remaining components. $\textbf{TaDA}$ produces a standard rank-$r$ LoRA adapter with zero inference overhead. On six scientific QA
benchmarks with Llama-2-7B, TaDA achieves an average accuracy of 0.452, outperforming DARE-TIES by +3.6 percentage points and obtaining the best result on all six
benchmarks. On six image classification
benchmarks with ViT-L/16, TaDA reaches 85.9% average accuracy, improving over the strongest merging baseline while leading in three of the six individual
benchmarks.
Read the full abstract ▾
All authors Huy Quoc To, Fuyi Li, Guangyan Huang, Ming Liu
Computation and Language
Depth-Attention: Cross-Layer Value Mixing for Language Models
Boyi Zeng, Yiqin Hao, Zitong Wang et al. (+7 more)
Self-
attention selects information freely across the sequence, but across depth,
Transformers merely add each layer's output to the residual stream, so later layers cannot selectively reuse earlier-layer representations. Recent cross-layer methods improve this flow but operate on hidden states outside
attention, adding state beyond the key-value cache at inference–a cost that becomes increasingly salient as modern LLMs compress the cache with grouped-query and multi-head latent
attention. We introduce Depth-
Attention, which performs this selection inside the
attention module itself: before a layer attends over the sequence, its query attends over the keys of earlier layers at the same
token position and mixes their values into the value that self-
attention then reads. Because Depth-
Attention reuses the standard
attention queries, keys, and value-cache slots, storing depth-mixed values in place of the original values, it adds no parameters and introduces no persistent inference state beyond the standard key-value cache–the same cache size as a vanilla decoder and less than hidden-state-based cross-layer methods. On Qwen3-style decoders at 1.5B and 3B parameters, Depth-
Attention attains the lowest perplexity and the highest average downstream accuracy, improving over the vanilla
Transformer by up to 2.3 accuracy points and surpassing strong cross-layer baselines in perplexity and average accuracy, while adding under 0.01% extra arithmetic FLOPs and no additional persistent inference state. The gains hold from 360M to 3B parameters and extend to looped
Transformers.
Read the full abstract ▾
All authors Boyi Zeng, Yiqin Hao, Zitong Wang, Shixiang Song, He Li, Feichen Song, Yifan Liu, Ziwei He, Xinbing Wang, Zhouhan Lin
Computation and Language
DAR: Deontic Reasoning with Agentic Harnesses
Guangyao Dou, William Jurayj, Nils Holzenberger et al. (+1 more)
Deontic
reasoning is the task of answering questions by applying explicit rules and policies to case-specific facts, for example computing tax liability under a statute or determining the outcome of an immigration appeal. A key technical challenge for LLM-based deontic
reasoning is that the relevant ruleset can be long and cross-referenced, so models may still fail to locate the rules needed for a particular
reasoning step. We introduce Deontic Agentic
Reasoning (DAR), an agentic
reasoning setup in which the model interacts with the statutes on demand. We evaluate DAR under multiple harnesses on hard subsets of DeonticBench. Across these settings, we find that agentic harnesses can push the frontier on deontic
reasoning tasks, but improvements are not uniform: weaker models often degrade on numerical tasks while consuming far more
tokens.
Read the full abstract ▾
All authors Guangyao Dou, William Jurayj, Nils Holzenberger, Benjamin Van Durme
Computation and Language · Cross-listed from cs.CV
M$^3$Eval: Multi-Modal Memory Evaluation through Cognitively-Grounded Video Tasks
Jie Huang, Ruixun Liu, Sirui Sun et al. (+4 more)
As multi-modal models advance towards long-form
video understanding,
memory emerges as a critical capability. Despite substantial efforts in developing
video datasets and
benchmarks, existing works primarily focus on perception and
reasoning, without systematically evaluating
memory: what models retain, how faithfully information is preserved, and how robust
memory remains under interference. To address this gap, we introduce M$^3$Eval, the first comprehensive evaluation framework and
benchmark for probing different
memory dimensions in multi-modal models. Grounded in cognitive psychology, our design features carefully constructed tasks that isolate key aspects of
memory. Leveraging M$^3$Eval, we conduct extensive experiments across representative multi-modal models, revealing consistent weaknesses and distinctive behaviors. We find that models struggle to maintain disentangled representations when processing parallel
video streams, exhibit interference patterns differing substantially from those observed in human
memory, ground
memory sources more reliably in the spatial domain than the temporal domain, and demonstrate limited symbolic
memory. Collectively, our
benchmark provides a valuable resource for future research, while our findings highlight
memory as a fundamental yet underexplored capability and offer insights for designing more effective
memory mechanisms in multi-modal models. Our code and
dataset are available at https://pku-value-lab.github.io/m3eval-homepage.
Read the full abstract ▾
All authors Jie Huang, Ruixun Liu, Sirui Sun, Xinyi Yang, Yin Li, Yixin Zhu, Yiwu Zhong
Computation and Language
GARL: Game-Theoretic Reinforcement Learning for Multi-Agent Strategic Prioritisation
Yuxiao Ye, Yiwen Zhang, Huiyuan Xie et al. (+2 more)
LLM-based multi-
agent systems are increasingly used for strategic decision-making tasks. In such settings, performance depends not only on individual model capabilities, but also on the policies by which
agents interact and adapt. Multi-
agent reinforcement learning can optimise these interaction policies, but its reward design often remains task-specific and weakly grounded in interaction structure. To address this gap, we propose GARL, a GAme-theoretic Reinforcement Learning framework for multi-
agent strategic prioritisation. GARL formalises strategic prioritisation as a two-stage game: competing
agents first allocate strategic resources over a shared candidate set, and a higher-level arbiter then produces the final ranking. The resulting game-theoretic utilities are converted into role-specific reinforcement signals, allowing policy optimisation to be guided by structured interaction. We instantiate GARL on issues-in-dispute ranking, where the goal is to prioritise core issues in legal proceedings. Experiments show that GARL improves ranking performance, enables small open-source LLMs to become competitive with a strong closed-source LLM under the same candidate-ranking setting, and yields gains in legal-domain competence and broader strategic decision-making. Overall, GARL demonstrates how game-theoretic interaction structure can be turned into reinforcement-learning objectives, providing a principled approach to policy optimisation in multi-
agent strategic prioritisation.
Read the full abstract ▾
All authors Yuxiao Ye, Yiwen Zhang, Huiyuan Xie, Yuqin Huang, Zhiyuan Liu
Computation and Language
DeliChess: A Multi-party Dialogue Dataset for Deliberation in Chess Puzzle Solving
Xiaochen Zhu, Georgi Karadzhov, Tom Stafford et al. (+1 more)
Multi-party dialogue is a critical setting for studying collaborative
reasoning and decision-making, yet existing
datasets rarely focus on structured,
reasoning-intensive tasks. We introduce DeliChess, a
dataset of group deliberation dialogues in which participants collaboratively solve multiple-choice chess puzzles. Participants first answer independently, then engage in multi-party deliberation and revise their individual answers. The
dataset comprises 107 dialogues with full transcripts, pre- and post-deliberation choices, and utterance-level annotations of communicative function, epistemic stance, and usefulness for supporting deliberation. Our analyses show that greater diversity in initial solution quality is associated with larger gains, while answer trajectories reveal how deliberation can recover, discover, or lose strong answers. Cases in which groups surpass every independent answer are associated with sustained
reasoning and epistemic openness. We further propose a diagnostic action-selection task and find that the tested LLMs show only weak agreement with human-attested helpful actions. Together, our
dataset provides a testbed for modelling group
reasoning, dialogue dynamics, and conditions for effective deliberation.
Read the full abstract ▾
All authors Xiaochen Zhu, Georgi Karadzhov, Tom Stafford, Andreas Vlachos
Computation and Language
Probing Outcome-Level Resemblance and Mechanism-Level Alignment in LLM Risk Decisions: Evidence from the St. Petersburg Game
Chensong Huang, Changyu Chen, Chenwei Lin et al. (+3 more)
LLMs can appear cautious in risk decision-making tasks, yet cautious-looking outputs do not necessarily indicate
alignment with human decision-making mechanisms. We investigate this distinction using the St. Petersburg game as a controlled testbed, a classical paradox in which the expected payoff is infinite, yet humans typically report low, finite willingness to pay. We evaluate 28 LLMs with a structured
prompt suite that includes the original game; controlled decision variants that perturb truncation, repeated play, numeric endowment, and occupational identity; a human-perspective
prompt that asks models to reason as human decision makers; and paired comparisons between base models and their instruction-tuned counterparts. In the original game, most models generate finite bids, creating the appearance of human-like risk behavior. However, this outcome-level resemblance masks substantial mechanism-level differences. The controlled variants reveal that rather than maintaining human-like behavior seen in the original game, models often shift to conditionally and computationally rational behavior. Human-cue prompting and instruction tuning often lower bids and reduce some visible pathologies, but most mechanism-level response patterns remain largely unchanged. These findings show that behavioral
alignment in risk decision-making can be surface-level: LLMs may produce human-like risk decisions without exhibiting human-consistent mechanisms. High-stakes evaluations of LLM decision-making should therefore move beyond outcome similarity and examine whether the
alignment is supported by mechanism-level consistency.
Read the full abstract ▾
All authors Chensong Huang, Changyu Chen, Chenwei Lin, Hanjia Lyu, Xian Xu, Jiebo Luo
Computation and Language
SAID: Accelerating Diffusion-Based Language Models via Scaffold-Aware Iterative Decoding
Na Li, Chengda Wang, Mingju Gao et al. (+1 more)
Diffusion large
language models (DLLMs) enable non-autoregressive generation by iteratively denoising corrupted
token sequences with bidirectional context. Despite their ability to update multiple positions in parallel, inference remains costly due to the many denoising steps required for high-quality generation. We propose SAID, a Scaffold-Aware Iterative Decoding framework that accelerates DLLMs by reallocating computation across
tokens. SAID first spends denoising computation on scaffold
tokens to establish the coarse semantic structure, and then completes predictable detail
tokens with fewer steps. We further adapt SAID to block-wise
diffusion decoding and introduce Confidence-Hierarchical Layered Generation (CHLG), which assigns additional steps only to low-confidence
tokens. Experiments on LLaDA-8B and LLaDA 1.5 across math, coding, and knowledge
benchmarks show that SAID significantly accelerates DLLM inference with a maximum speedup of 9.1x while maintaining competitive performance. Our code is publicly available: https://github.com/TH-AI-Lab-PKU/SAID.
Read the full abstract ▾
All authors Na Li, Chengda Wang, Mingju Gao, Hao Tang
Computation and Language
SemBlock: Semantic Boundary Dynamic Blocks for Diffusion LLMs
Xinrui Song, Zhuoran Wang, Mingju Gao et al. (+1 more)
Diffusion language models (DLMs) generate text through iterative denoising, and blockwise decoding improves their practicality by committing
tokens in local blocks. However, existing blockwise methods typically rely on fixed block sizes or delimiter-based runtime signals, which do not necessarily align with semantic boundaries. In this paper, we propose SemBlock, a semantic-boundary-driven dynamic block decoding framework for
diffusion LLMs. SemBlock formulates dynamic block construction as semantic boundary prediction and trains lightweight predictors on frozen LLaDA hidden states. To provide supervision, we construct SemBound, a semantic-boundary
dataset that derives boundary labels from discourse units,
reasoning steps, and implementation spans across natural
language, math, and code tasks. During inference, SemBlock uses predicted boundary probabilities to select the ending position of each dynamic block. Experiments on GSM8K, IFEval, MATH, and HumanEval show that SemBlock consistently improves over fixed-block decoding and AdaBlock. Our code is publicly available: https://github.com/TH-AI-Lab-PKU/SemBlock.
Read the full abstract ▾
All authors Xinrui Song, Zhuoran Wang, Mingju Gao, Hao Tang
Computation and Language · Cross-listed from cs.HC
Clinical Assistant for Remote Engagement Link (CARE-link): A Web-Based Electronic Health Records Software for Managing Diabetes
Prince Ebenezer Adjei, Joshua Teye Tettey, Toufiq Musah et al. (+2 more)
CARE-link is an open-source, web-based clinical support platform designed to improve the management of gestational diabetes by linking clinicians and patients through an LLM-mediated workflow. The system aggregates patient-generated data outside the hospital, summarizes relevant clinical information, and delivers context-aware decision support to clinicians. For patients, CARE-link provides clear explanations of management plans and delivers timely lifestyle guidance through a WhatsApp interface. The integrated dual-facing design aims to promote continuous monitoring, support individualized care, and reduce the burden of in-clinic follow-ups. Built with a modular architecture, the platform can be adapted to other chronic conditions requiring longitudinal tracking and behavioral support. CARE-link has the potential to enhance clinical oversight, promote patient compliance, and strengthen continuity of care particularly in resource-constrained settings.
Read the full abstract ▾
All authors Prince Ebenezer Adjei, Joshua Teye Tettey, Toufiq Musah, Audrey Agbeve, John Amuasi
Computation and Language · Cross-listed from cs.LG
Data Attribution in Large Language Models via Bidirectional Gradient Optimization
Frédéric Berdoz, Luca A. Lanzendörfer, Kaan Bayraktar et al. (+1 more)
Large
Language Models (LLMs) are increasingly deployed across diverse applications, raising critical questions for governance, accountability, and data provenance. Understanding which training data most influenced a model's output remains a fundamental open problem. We address this challenge through training data attribution (TDA) for auto-regressive LLMs by expanding upon the inverse formulation: How would training data be affected if the model had seen the generated output during training? Our method perturbs the base model using bidirectional gradient optimization (gradient ascent and descent) on a generated text sample and measures the resulting change in loss across training samples. Our framework supports attribution at arbitrary data granularity, enabling both factual and stylistic attribution. We evaluate our method against baselines on pretrained models with known
datasets, and show that it outperforms previous work on influence metrics, thereby enhancing model interpretability, an essential requirement for accountable AI systems.
Read the full abstract ▾
All authors Frédéric Berdoz, Luca A. Lanzendörfer, Kaan Bayraktar, Roger Wattenhofer
Computation and Language
Can Crowdsourcing Survive the LLM Era? A Community Survey on Human Data Collection
Aswathy Velutharambath, Neele Falk, Sofie Labat et al. (+2 more)
The widespread use of Large
Language Models (LLMs) as writing tools challenges the validity of crowdsourced data, as crowdworkers may outsource tasks to models. To better understand how this is addressed, we surveyed 155 researchers in NLP and related disciplines about their experiences and opinions on collecting free-text responses via crowdsourcing. This paper provides an overview of practitioners' challenges, mitigation strategies, and the foreseen implications on data quality. 44% of respondents reported observing LLM usage in their crowdsourced data. While 93% of them had anticipated this, half were unsure what precautions to take. The most prevalent detection strategies are distinctive textual style patterns and unusually fast completion times. Overall, survey responses show that the research community is aware of the problem and taking measures, but existing efforts remain insufficient to fully address it. Finally, we derive a set of considerations to guide future crowdsourced free-text data collection in the era of LLMs.
Read the full abstract ▾
All authors Aswathy Velutharambath, Neele Falk, Sofie Labat, Tarun Tater, Amelie Wuehrl
Computation and Language · Cross-listed from cs.LG
Reproducing, Analyzing, and Detecting Reward Hacking in Rubric-Based Reinforcement Learning
Xuekang Wang, Zhuoyuan Hao, Shuo Hou et al. (+3 more)
Rubric-based reinforcement learning (RL) uses an LLM-as-a-Judge (LaaJ) to score model outputs according to rubrics as rewards. However, policy models may exploit latent biases in the judge, leading to reward hacking and ineffective or unsafe training outcomes. In real-world rubric-based RL, such hacking behaviors are often subtle and entangled with multiple judge biases, making them difficult to analyze, detect, and mitigate. In this paper, we introduce CHERRL, a Controllable Hacking Environment for Rubric-based RL. By injecting known biases into LaaJ, CHERRL enables stable reproduction of reward hacking, explicit observation of reward divergence, and identification of hacking onset. This provides a clean experimental testbed for studying the mechanisms and mitigations of reward hacking in rubric-based RL. To demonstrate its utility, we analyze different judge biases from the perspectives of discoverability and exploitability, and explore an
agent for automatically detecting reward hacking onset from training logs. The code and environment are publicly available at https://github.com/THUAIS-Lab/CHERRL.
Read the full abstract ▾
All authors Xuekang Wang, Zhuoyuan Hao, Shuo Hou, Hao Peng, Juanzi Li, Xiaozhi Wang
Computation and Language
Caliper: Probing Lexical Anchors versus Causal Structure in LLMs
Zhenyu Yu and Shuigeng Zhou
Large
language models reach 50 to 70% accuracy on causal
reasoning benchmarks such as CLadder, but it is unclear whether this reflects structural
reasoning or lexical pattern matching. We introduce Caliper, a controlled perturbation that replaces semantic variable names with placeholder
tokens while preserving the causal graph and probabilistic specification of each question. Across nine instruction-tuned LLMs from 3.8B to 671B and three causal
reasoning benchmarks, lexical anonymization yields robust accuracy drops of +7.6, +27.0, and +11.1 pp on a local 3.8B-14B set, rising to +29.6 and +18.0 pp on CRASS and e-CARE across nine frontier models spanning the 2024-2026 generations. Of 40 engaged model-by-
benchmark cells, 39 show a positive gap, and the gap collapses by 17x on CLadder's pseudoword subset. Structured scaffolding and few-shot in-context learning each narrow the gap, but mainly by lowering P0 accuracy on smaller models rather than recovering P1. Current instruction-tuned LLMs, evaluated zero-shot, show little evidence of structural causal
reasoning once lexical anchors are removed.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.CV
BreastGPT: A Multimodal Large Language Model for the Full Spectrum of Breast Cancer Clinical Routine
Yang Liu, Jiajin Zhang, Danyang Tu et al. (+8 more)
Breast cancer remains a leading cause of cancer-related mortality among women. Its clinical management requires
multimodal reasoning across a clinical workflow that spans screening, diagnosis and treatment planning, where each stage involves distinct imaging modalities, task objectives, and
reasoning patterns. However, constrained by data scarcity and model versatility, existing medical MLLMs are typically evaluated on isolated modalities or narrow task families, limiting their ability to support workflow-level clinical
reasoning. In this work, we first introduce BreastStage, a workflow-aligned breast imaging instruction corpus comprising 1.86M instruction-following pairs curated from 17 sub-
datasets across 5 imaging modalities and 136 task templates. Its held-out split, BreastStage-Bench, provides a comprehensive
benchmark for evaluating
multimodal reasoning across the breast cancer care continuum. Building on this corpus, we propose BreastGPT, a unified MLLM equipped with a dual-branch visual encoder and concept-preserving
token compression to bridge the scale gap between standard radiology and gigapixel pathology. On BreastStage-Bench, BreastGPT achieves 75.66% closed-ended accuracy and 89.92% open-ended score, outperforming both general-purpose and medical-specific MLLMs across clinical stages and task formats. These results suggest that workflow-aligned data and cross-scale visual modeling are critical for clinically grounded medical MLLMs. All data, code, and model checkpoints are released at https://yangyy-liu.github.io/BreastGPT.io.
Read the full abstract ▾
All authors Yang Liu, Jiajin Zhang, Danyang Tu, Yaojun Hu, Jiao Qu, Jiuyu Zhang, Yu Shi, Wei Fang, Shi Gu, Ling Zhang, Yingda Xia
Computation and Language · Cross-listed from cs.IR
BEATS: Bootstrapping E-commerce Attribute Taxonomies for Search through Iterative Human-AI Collaboration
Yung-Yu Shih, Shang-Yu Su, Tzu-I Ho et al. (+2 more)
E-commerce platforms in emerging
markets often operate with underdeveloped product catalogs that contain only category taxonomies but lack structured attribute schemas. This absence of fine-grained product attributes limits search capabilities – preventing faceted filtering, degrading query understanding, and weakening semantic representations used by search systems. We present BEATS, a human-in-the-loop LLM framework for bootstrapping product attribute taxonomies entirely from scratch. Our approach extends a multi-stage LLM generation pipeline with two critical production stages: (1) proactive quality checking by model developers to filter erroneous outputs, and (2) human annotation by domain-expert local staff to validate generated attributes. The framework operates iteratively –
prompts at each generation stage are refined based on quality check observations and annotator feedback across successive rounds, progressively improving attribute quality. Once the attribute taxonomy is established, we employ LLMs to perform structured attribute tagging on individual product items, enriching their contextual representations. The enriched catalog directly benefits multiple components of the search system: enabling granular attribute-based filtering, providing structured features for ranking models, and improving semantic representations for dense
retrieval. We validate the generated taxonomy by training dense
retrieval models on attribute-enriched product data, demonstrating consistent improvements over baselines using original catalog information. Our system has been deployed at Rakuten Taiwan, enriching 9 major categories spanning 2,694 sub-categories with 67,277 generated attributes, and over 5.4 million products have been tagged with the generated attributes, with plans to enrich the entire product catalog.
Read the full abstract ▾
All authors Yung-Yu Shih, Shang-Yu Su, Tzu-I Ho, Dongzhe Wang, Yun-Nung Chen
Computation and Language
'Your AI Text is not Mine': Redefining and Evaluating AI-generated Text Detection under Realistic Assumptions
Nils Dycke, Marina Sakharova, Nico Daheim et al. (+1 more)
Although it is generally agreed that AI-generated text poses a broad societal risk, there is no common understanding in the AI-generated text detection literature on what constitutes harmful use. Rather, existing
datasets and approaches often define their own criteria and make their own assumptions, sometimes implicitly, and often only loosely related to real-world needs and applications. To address this gap, we here systematically define various notions of AI-generated text and their characteristics. To study these, we collect AITDNA - a new
benchmark of human-machine co-constructed texts that is annotated with detailed genesis information, such as the entire edit and AI-interaction history. We
benchmark various machine-generated text detectors and find that they often only perform well for specific notions but not as broad detectors. We release code and data publicly.
Read the full abstract ▾
All authors Nils Dycke, Marina Sakharova, Nico Daheim, Iryna Gurevych
Computation and Language
GRAIL: Gradient-Reweighted Advantages for Reinforcement Learning with Verifiable Rewards
Tej Deep Pala, Vernon Toh and Soujanya Poria
Reinforcement learning with verifiable rewards (e.g. GRPO) is now a common way to improve mathematical
reasoning in Large
Language Models (LLMs). However, current methods usually broadcast one sequence-level advantage to all
tokens, or use costly process reward models (PRMs) for step-level supervision. Uniform advantage distribution assumes that all
tokens contribute equally to the final reward. This dilutes the gradient signal, since flawed
reasoning steps and filler words are updated as strongly as valid logical inferences. To address this, we introduce Gradient-Reweighted Advantage (GRAIL), an intrinsic
token-wise advantage reweighting method. GRAIL uses gradient-activation saliency to place more weight on
tokens that are more locally sensitive to the final answer. Evaluations across five models from the Qwen3, R1-distilled and OctoThinker families show that GRAIL consistently outperforms GRPO. GRAIL achieved an average improvement of 3.60% in accuracy and 3.05% in Pass@3, demonstrating that fine-grained
reasoning alignment can be achieved without process-level supervision.
Read the full abstract ▾
Computation and Language
Optimizing the Cost-Quality Tradeoff of Agentic Theorem Provers in Lean
Kári Rögnvaldsson, Chenhao Sun, Jasper Dekoninck et al. (+1 more)
Large
language models (LLMs) are increasingly used in workflows for generating formal proofs in Lean. These workflows often decompose problems into smaller lemmas, sample many proof attempts, and use compiler feedback to guide search. However, they can be prohibitively expensive, often spending substantial compute on attempts that ultimately fail. In this work, we address this problem with an action routing
agent that consists of a data plane and a control plane. The data plane generates natural-
language lemma decompositions, formalizes them in Lean, and samples proof attempts for the resulting
theorem and lemma targets. The control plane observes previous failed Lean attempts, estimates both the likelihood of success and the cost of another attempt, and decides whether to continue proving the current target or restart from a new breakdown. On a subset of PutnamBench, our
agent decreases the cost by 28.9% over a fixed-step baseline on average, preserving performance while using substantially less compute. These results suggest that failed Lean trajectories provide actionable signals for cost-aware resource allocation in agentic
theorem proving.
Read the full abstract ▾
All authors Kári Rögnvaldsson, Chenhao Sun, Jasper Dekoninck, Martin Vechev
Computation and Language
Agent Planning Benchmark: A Diagnostic Framework for Planning Capabilities in LLM Agents
Haoyu Sun, Wenxuan Wang, Mingyang Song et al. (+5 more)
Planning is central to LLM
agents: before acting, an
agent must decompose goals, select tools, reason over constraints, and decide when a task is infeasible. Yet existing
agent evaluations often report only end-to-end success, making it difficult to determine whether failures stem from planning or execution. We introduce
Agent Planning
Benchmark (APB), a planning-specific diagnostic
benchmark with 4,209
multimodal cases across 22 domains and five settings, covering holistic planning, feedback-conditioned step-wise planning, and robustness under extraneous tools, broken tools, and unsolvable tasks. Across 12 MLLMs, APB reveals systematic weaknesses in long-horizon planning, tool-noise robustness, calibrated refusal, and inference-time refinement. We further validate APB on 200 ToolSandbox tasks and 200 $\tau^2$-bench tasks, where APB-guided refinement consistently improves plan correctness, plan grade, and downstream execution metrics across three representative models. APB thus serves as an upstream diagnostic complement to execution
benchmarks. The APB
benchmark and code are available in \href{https://github.com/Mikivishy/AgentPlanningBenchmark}{this URL}.
Read the full abstract ▾
All authors Haoyu Sun, Wenxuan Wang, Mingyang Song, Jujie He, Weinan Zhang, Yang Liu, Yang Yang, Yu Cheng
Computation and Language · Cross-listed from cs.CV
MusaCoder: Native GPU Kernel Generation with Full-Stack Training on Moore Threads GPU
Kun Cheng, Songshuo Lu, Sicong Liao et al. (+7 more)
Native GPU kernel generation turns high-level tensor programs into executable, efficient low-level code. Existing Large
Language Models (LLMs) struggle with this task, while execution-based reinforcement learning suffers from sparse rewards, reward hacking, and training instability. We present MusaCoder, a full-stack training framework for native GPU kernel generation on CUDA and MUSA backends. MusaCoder combines progressive kernel-oriented data synthesis, diversity-preserving rejection fine-tuning, and execution-feedback Reinforcement Learning (RL) through MooreEval, a distributed verifier and reward environment. To stabilize RL, MusaCoder introduces PrimeEcho for first-turn-anchored multi-turn rewards, Buffered Dynamic Retry for recovering signals from all-failed hard samples, and MirrorPop for off-policy sequence filtering. Experiments on KernelBench and a MUSA-ported variant show that MusaCoder outperforms strong open-source and proprietary baselines in both correctness and empirical speedup, with the 9B model matching or exceeding frontier closed-source models and the 27B model establishing a new state of the art. These results demonstrate not only the effectiveness of full-stack execution-feedback training for native kernel generation, but also the capability of Moore Threads GPUs to support the complete LLM post-training stack, providing a practical foundation for large-model training and optimization on emerging accelerators.
Read the full abstract ▾
All authors Kun Cheng, Songshuo Lu, Sicong Liao, Tankun Li, Yafei Zhang, Dong Yang, Qiheng Lv, Hua Wang, Zhi Chen, Yaohua Tang
Computation and Language
Large Language Models in K-12 Education: Alignment with State Curriculum Standards and Student Personas
Lisa Korver, Tomo Lazovich and Sherief Reda
As Large
Language Models (LLMs) become increasingly popular in educational settings, they raise important questions about the ethical implications of their use. Publicly available online chatbots are quickly improving in capability and accuracy leading to more widespread use, including among students looking for help with their homework. This makes it crucial to consider whether these models are aligned with educational standards. Because curriculum standards in the United States are set at the state level, they differ significantly in required content, emphasis, and narrative focus. In this work, we develop an LLM-based pipeline to identify variations in U.S. History curricula across states and evaluate the extent to which different LLMs reflect these state-specific curricular differences. In addition, we conduct controlled experiments that vary user personas by stating user attributes such as geographic location, grade level, gender and race to evaluate the sensitivity of LLM responses to user characteristics. We find that while models are able to adjust their presentation of historical topics, these shifts may come from the perceived political leanings of states and do not necessarily reflect actual curriculum content. Additionally, models successfully adapt to a student's grade level while showing minimal sensitivity to race or gender, suggesting they are capable of useful adaptation to student personas with limited demographic bias. Together, these findings highlight potential risks that open access to LLM chatbots may cause to student learning outcomes stemming from misalignment with state curriculum standards and highlight the need for more robust
alignment techniques.
Read the full abstract ▾
Computation and Language
A French Corpus Annotated for Multiword Expressions with Adverbial Function
Eric Laporte, Takuya Nakamura and Stavroula Voyatzi
This paper presents a French corpus annotated for multiword expressions (MWEs) with adverbial function. This corpus is designed for investigation on information
retrieval and extraction, as well as on deep and shallow syntactic parsing. We delimit which kind of MWEs we annotated, we describe the resources and methods we used for the annotation, and we briefly comment the results. The annotated corpus is available at http://infolingu.univ-mlv.fr/ under the LGPLLR license.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.AI
R-APS: Compositional Reasoning and In-Context Meta-Learning for Constrained Design via Reflective Adversarial Pareto Search
João Pedro Gandarela, Thiago Rios, Stefan Menzel et al. (+1 more)
Large
language models (LLMs) are fluent on open-ended tasks, yet in agentic settings, where a system must plan, use tools, and act over extended horizons, fluency does not ensure reliable delivery. We trace this gap to three coupled structural failures: errors propagate without localization, worst-case perturbations go unevaluated, and accumulated knowledge is never invalidated. We argue these share a root cause: abductive, counterfactual, meta-inductive, corrective, and inductive
reasoning pull a shared context in incompatible directions. We introduce Reflective
Adversarial Pareto Search (R-APS), to our knowledge the first method addressing all three failures jointly via
reasoning-mode decomposition, allocating each
reasoning mode its own context and orchestrating interaction across three timescales: staged compositional
reasoning with a typed validation critic (failure localization), sensitivity-guided counterfactual stress-testing as a first-class Pareto objective (robustness), and meta-inductive rule extraction with explicit invalidation (persistent
memory). R-APS requires no fine-tuning and operates on a frozen LLM purely via structured protocol design. We evaluate on planar mechanism synthesis (robotics, prosthetics, mechanical design), with every candidate checked by a kinematic solver. On 32 target trajectories, R-APS delivers robustness certificates 3.5x tighter than uniform-perturbation baselines, 46% faster iterations-to-first-admission, and 2.1x Chamfer-distance reduction over Enum+GA while jointly controlling bar-count and worst-case robustness. Small 4B
reasoning-specialized models prove competitive with general-purpose 70B backbones inside the protocol, suggesting structured protocols can partially offset model scale.
Read the full abstract ▾
All authors João Pedro Gandarela, Thiago Rios, Stefan Menzel, André Freitas
Computation and Language · Cross-listed from cs.AI
BiasGRPO: Stabilizing Bias Mitigation in High-Variance Reward Landscapes via Group-Relative Policy Optimization
Saket Reddy, Ke Yang and ChengXiang Zhai
Mitigating social bias in Large
Language Models (LLMs) presents a distinct
alignment challenge: unlike verifiable tasks, bias lacks a single ground truth, creating a high-variance, subjective reward landscape. Previous preference-based fine-tuning methods have major trade-offs: Direct Preference Optimization (DPO) is limited by the lack of exploration inherent in offline training, while Proximal Policy Optimization (PPO) can lead to training instability due to potentially unreliable critic estimates. In this paper, we propose BiasGRPO, a framework using Group Relative Policy Optimization (GRPO) to stabilize
alignment by normalizing rewards across a group of sampled completions. By substituting the value function with a group-relative baseline, our approach reduces instability while maintaining the exploration benefits of online training. We find that BiasGRPO outperforms DPO and PPO across multiple
benchmarks, indicating its effectiveness. To adapt GRPO, we synthetically extend a
dataset spanning multiple domains and contexts. We also create and release a custom bias reward model that effectively guides generation while being highly compute-efficient and avoiding knowledge degradation, providing a valuable resource that can be seamlessly integrated into multi-objective RLHF pipelines.
Read the full abstract ▾