Computation and Language · Cross-listed from cs.LG
CosmicFish-HRM: Adaptive Reasoning via Hierarchical Recurrent Mechanisms in Compact Language Models
Venkat Akhil Lakkapragada
Large
language models have achieved strong
reasoning capabilities, though often at the cost of massive parameter counts and expensive inference. In this work, we explore a different direction: adaptive
reasoning depth in compact
language models. We present CosmicFish-HRM, a compact
language model built around a Hierarchical
Reasoning Module (HRM) that dynamically allocates computational effort during inference. Instead of applying fixed computation to every input, the model iterates through high-level and low-level
reasoning cycles and learns when to halt based on input complexity. CosmicFish-HRM combines this adaptive
reasoning core with modern
transformer components including Grouped Query
Attention, RoPE, and SwiGLU activations. While the additional
reasoning infrastructure introduces overhead at small scale, we hypothesize that this tradeoff becomes increasingly favorable as model size grows and the relative cost of the HRM core diminishes. Our results show that the model learns non-uniform
reasoning behavior, allocating different numbers of
reasoning steps across tasks and inputs. These findings suggest that adaptive
reasoning depth may offer a promising alternative to relying solely on parameter scale for
reasoning capability.
Read the full abstract ▾
Computation and Language
Reasoning that Travels: Dissecting How Chain-of-Thought Transfers Across Models
Xinyuan Cheng, Beiduo Chen, Philipp Mondorf et al. (+1 more)
Large
reasoning models (LRMs) often generate extensive chain-of-thought (CoT) traces before producing a final answer. As explicit textual artifacts, these traces can be passed to other models to solve the same task, enabling cross-model
reasoning transfer. Yet successful transfer alone does not reveal how the provided CoT contributes to another model's answer. We study this question with a controlled provider–receiver framework, where a provider generates a
reasoning trace and a receiver solves the same problem from increasingly longer trace prefixes. We compare force-answer, where the receiver answers directly from the prefix, with free-generation, where it may continue
reasoning before answering. Across models and
benchmarks, full traces often transfer successfully, but prefix trajectories reveal distinct mechanisms. In force-answer mode, AIME transfer is largely driven by explicit answer availability. MMLU-Pro instead reflects a larger role for receiver competence, while ZebraLogic depends on partial structured-answer information rather than complete-answer leakage alone. In free-generation mode, partial CoTs improve performance across
benchmarks, indicating that prefixes can guide continued
reasoning. Finally, answer agreement among receivers provides a gold-free signal for stopping provider
reasoning early. Overall, cross-model CoT transfer is not a single phenomenon: it can reflect answer extraction,
reasoning scaffolding, or receiver-dependent competence.
Read the full abstract ▾
All authors Xinyuan Cheng, Beiduo Chen, Philipp Mondorf, Barbara Plank
Computation and Language
Hallucination Detection-Guided Preference Optimization for Clinical Summarization
Shamanth Kuthpadi Seethakantha, Dung Ngoc Thai, Vara Prasad Gudi et al. (+6 more)
Large
language models (LLMs) have shown promise on summarization tasks, but they often produce
hallucinations, which are unsupported or incorrect statements that limit their reliability in specialized healthcare applications. We introduce
Hallucination Detection Guided Self-Refinement (HDSR), an inference-time method that leverages
hallucination detectors to guide iterative summary revisions toward factual corrections. Building on this, we propose HDSR for Preference Learning (HDSR-PL), which converts detector-guided refinement trajectories into preference pairs for model finetuning. Extensive experiments show that our methods substantially reduce
hallucinations for Llama and Gemma models in summarizing real-world clinical notes from MIMIC-IV-Note v2.2. For example, HDSR reduces 24% and HDSR-PL reduces 48%
hallucinations in Llama-3.1-8B-Instruct. Importantly, both methods preserve summary fluency, coherence, and relevance according to human expert and LLM-Jury evaluations. Together, these results demonstrate that detection-informed refinement and preference learning offer an automated solution for improving factual faithfulness in clinical summarization.
Read the full abstract ▾
All authors Shamanth Kuthpadi Seethakantha, Dung Ngoc Thai, Vara Prasad Gudi, Simran Tiwari, Rami Matar, Avijit Mitra, Wenlong Zhao, Andrew McCallum, Wael Salloum
Computation and Language · Cross-listed from cs.LG
PEFT-Arena: Understanding Parameter-Efficient Finetuning from a Stability-Plasticity Perspective
Yangyi Huang, Ruotian Peng, Zeju Qiu et al. (+4 more)
Parameter-efficient finetuning (PEFT) has become the standard approach for adapting large
language models, yet evaluations largely emphasize downstream accuracy while overlooking the retention of pretrained capabilities. We argue that PEFT should be assessed through the stability-plasticity dilemma: the trade-off between target-task adaptation and resistance to forgetting. We introduce PEFT-Arena, a
benchmark that jointly measures downstream performance and general capability retention. Across methods, we find distinct stability-plasticity profiles; under comparable parameter budgets, orthogonal finetuning achieves the most favorable Pareto frontier. To explain these differences, we analyze PEFT updates from two
geometric perspectives. In weight space, spectral analysis reveals how parameterizations interact with the pretrained singular-value structure. In activation space, retention metrics show whether finetuning preserves or distorts general-capability representations, with forgetting linked to non-isometric representation distortion. Finally, an analysis shows that final SFT checkpoints often overshoot a better target-retention operating point. Inspired by this, we present case studies of a post-hoc improvement with path-wise rewinding.
Read the full abstract ▾
All authors Yangyi Huang, Ruotian Peng, Zeju Qiu, Jiale Kang, Yandong Wen, Bernhard Schölkopf, Weiyang Liu
Computation and Language
Do VLMs Align Better with Humans than LLMs during Natural Reading?
Jinzhou Wu, Zhengwu Ma, Jixing Li et al. (+2 more)
Large
language models have become increasingly useful computational models of human
language processing, but it remains open whether
vision-
language learning makes text representations more human-like during natural reading. We address this question by comparing matched LLM and
vision-
language model pairs under strictly text-only input and evaluating
alignment with human
brain activity (whole-cortex fMRI) and human behavior (synchronized regressive saccades). We identify a selective, rather than global, effect of
vision-
language training on human-model
alignment. In the two within-lineage model pairs, VLMs more accurately predicted human regressive saccades, whereas VLMs and LLMs showed comparable whole-cortex fMRI
alignment. However, sentence-level analyses revealed that the VLM advantage in fMRI
alignment increased with the visual evocative strength of the sentences. Together, these findings provide a controlled in-silico comparison of
multimodal training histories, showing that
vision-
language pretraining selectively improves model-human
alignment via reading behavior and visually grounded content.
Read the full abstract ▾
All authors Jinzhou Wu, Zhengwu Ma, Jixing Li, Baoping Tang, Zitong Lu
Computation and Language
Self-Improving Language Models with Bidirectional Evolutionary Search
Guowei Xu, Zhenting Qi, Huangyuan Su et al. (+4 more)
Search has been proposed as an effective method for self-improving
language models and agentic systems, both for post-training sample generation and for inference. However, widely used methods such as best-of-N sampling and tree search face two fundamental limitations: they are guided by sparse verification signals, and they construct candidates primarily through autoregressive expansion, restricting exploration to regions with substantial model probability mass. To address these, we propose Bidirectional Evolutionary Search (BES), a search framework that couples forward candidate
evolution with backward goal decomposition. In the forward search, BES augments standard expansion with
evolution operators that recombine partial trajectories to generate candidates that are difficult to obtain from a single model rollout. In the backward search, BES recursively decomposes the original task into checkable sub-goals, producing dense intermediate feedback that guides forward search. We provide motivation illustrating that candidates generated by expansion-only search may be confined to a narrow
entropy shell, while
evolution operators may escape it. Experiments show that on challenging post-training tasks where mainstream post-training
algorithms fail to improve, BES enables consistent gains. At inference time, BES outperforms existing open-source frameworks on three open problem solving
benchmarks in both average and best-case performance, and outperforms majority voting and Monte Carlo Tree Search on mathematical
reasoning and competitive programming
benchmarks. Code and trained models are available at https://github.com/
Embodied-Minds-Lab/BES.
Read the full abstract ▾
All authors Guowei Xu, Zhenting Qi, Huangyuan Su, Weirui Ye, Himabindu Lakkaraju, Sham M. Kakade, Yilun Du
Computation and Language · Cross-listed from cs.CV
Personal Visual Memory from Explicit and Implicit Evidence
Viet Nguyen, Thao Nguyen, Vishal M. Patel et al. (+1 more)
Long-term
memory is increasingly important for personalized AI
agents, yet existing
benchmarks and methods remain largely text-centric. Even when images are included, the user-specific information needed for later questions is typically recoverable from text alone, and most
memory systems reduce image turns to generic captions. Yet images often carry personal information that text rarely states – both explicit evidence, such as recurring user-associated entities, and implicit evidence, such as latent user facts inferred from visual or
multimodal cues. We introduce a
benchmark for personal visual
memory that targets both forms of evidence, and propose VisualMem, a hybrid visual–text architecture that augments a text-
memory backend with a structured personal visual
memory module. Rather than collapsing images into captions, VisualMem uses conversational context to resolve identity, ownership, and durable user facts. Experiments show that VisualMem substantially outperforms prior
memory systems on our
benchmark while remaining competitive on standard text-
memory benchmarks, indicating that personal visual
memory is a distinct and important component of long-term
memory for personalized AI
agents.
Read the full abstract ▾
All authors Viet Nguyen, Thao Nguyen, Vishal M. Patel, Yuheng Li
Computation and Language
OmniVerifier-M1: Multimodal Meta-Verifier with Explicit Structured Recalibration
Xinchen Zhang, Bowei Liu, Jiale Liu et al. (+7 more)
Visual outcomes are increasingly central to
multimodal large
language models, making reliable and fine-grained verification essential for scaling generalist foundation models. In this work, we investigate
multimodal meta-verification, which leverages verifier-generated rationales rather than decision-only signals, and explore how to effectively incorporate meta-verification feedback into
multimodal verifier training. We identify two key findings. First, symbolic verifier outputs (e.g., bounding boxes) outperform textual explanations as meta-verification rationales, enabling efficient rule-based reinforcement learning rewards while avoiding reliance on model-based rewards from auxiliary judge models. Second, decoupling reinforcement learning objectives for binary judgment and meta-verification substantially outperforms joint reward optimization, due to intrinsic differences in output structure and learning dynamics. Based on these insights, we train OmniVerifier-M1, a generalist visual verifier leveraging symbolic meta-verification and decoupled reinforcement learning. OmniVerifier-M1 provides robust verification and fine-grained error localization, and further enables M1-TTS, a verifier-driven agentic generation system achieving dynamic region-level self-correction. This approach paves the way for more reliable, interpretable, and fine-grained
multimodal verification, supporting safer and more controllable foundation model deployment.
Read the full abstract ▾
All authors Xinchen Zhang, Bowei Liu, Jiale Liu, Chufan Shi, Yizhen Zhang, Junhong Liu, Youliang Zhang, Zhiheng Li, Yujiu Yang, Ling Yang
Computation and Language
Human Label Variation as Stable Signal: Learning Annotator-Specific Explanation Behavior via Cross-Annotator Preference Optimization
Beiduo Chen, Pingjun Hong, Ziyun Zhang et al. (+3 more)
Free-text explanations extend human label variation (HLV) beyond label disagreement by revealing the
reasoning and preferences behind annotators' decisions. We study whether large
language models (LLMs) can learn and reproduce such annotator-specific label-explanation behavior. Using two sentence-pair tasks with four annotators each – natural
language inference and paraphrase judgment – we first analyze whether annotators exhibit stable individual patterns. We find that such patterns are weak at the single-annotation level due to strong input-content effects, but become detectable after input-content reduction and annotator-level aggregation. We then compare prompting and supervised fine-tuning (SFT) baselines and propose cross-annotator preference optimization (CAPO), which contrasts a target annotator's response with other valid but less target-specific annotations for the same input. Experiments show that prompting is limited and unstable, SFT better captures annotator-specific behavior, and CAPO further improves aggregation-aware imitation and judge-based attribution while preserving target-specific
reasoning patterns under human validation. Overall, our results show that HLV can be learned as annotator-specific label-explanation behavior, suggesting a path toward scalable explanation-based annotation grounded in annotator histories rather than labels alone.
Read the full abstract ▾
All authors Beiduo Chen, Pingjun Hong, Ziyun Zhang, Benjamin Roth, Anna Korhonen, Barbara Plank
Computation and Language
Jiazhen Huang, Xiao Chen, Xiao Luo et al. (+3 more)
On-policy self-
distillation (SD) improves LLM
reasoning by using teacher-side privileged information (PI) to turn sparse verifier outcomes into dense
token-level supervision. Existing methods usually assume trusted PI, such as reference answers or successful traces. We ask whether PI can instead come from an experience-derived skill bank, where retrieved skills are compact and reusable but may also be irrelevant or misleading. We propose Skill-Conditioned Gated Self-
Distillation (SGSD), which formulates skill-based SD as teacher hypothesis validation rather than unconditional imitation. SGSD retrieves skill-mistake pairs, constructs a multi-teacher pool, and lets all skill-conditioned teachers score the same plain-
prompt student rollout. The verifier validates each teacher's polarity: supporting a success or suppressing a failure gives positive supervision, while the opposite stance is reversed. A robust gated objective then distills informative teacher-student disagreements while suppressing uncertain or extreme signals. Experiments on multiple mathematical
reasoning benchmarks show that SGSD consistently improves over GRPO and remains competitive with answer-conditioned OPSD under a weaker PI assumption. For example, on Qwen3-1.7B, SGSD outperforms GRPO by 6.2% and OPSD by 1.7% on average on AIME24, AIME25, and HMMT25.
Read the full abstract ▾
All authors Jiazhen Huang, Xiao Chen, Xiao Luo, Yong Dai, Senkang Hu, Yuzhi Zhao
Computation and Language
Can Large Language Models Handle Discourse Particles? A Case Study of Colloquial Malay
Mariah Al Giptiah Binte Yusoff, Jakin Tan, Bocheng Chen et al. (+2 more)
Discourse particles, such as well and kind of, are crucial components that enable LLMs to "speak" more like humans. They are used to convey
emotions, intentions, and interpersonal attitudes. However, existing studies have not yet built a comprehensive understanding of LLMs' capabilities in handling discourse particles. Moreover, the limited number of research focuses primarily on high-resource
languages such as English, with little
attention paid to Southeast Asian
languages. In this paper, we (1) propose MalayPrag, a
benchmark designed to systematically evaluate and analyze LLMs' capabilities in handling discourse particles in colloquial Malay; (2) introduce five attributes that provide a theoretically grounded, unified framework for interpreting pragmatic functions of discourse particles. Applying these two, we
prompt ten off-the-shelf LLMs to perform three prediction tasks. The experimental results reveal substantial challenges for current LLMs to accurately connect discourse particles and their pragmatic functions in Malay. The provision of the five attributes designed in this study is found to significantly improve the connections, highlighting the need for structured scaffolding for models' pragmatic competence.
Read the full abstract ▾
All authors Mariah Al Giptiah Binte Yusoff, Jakin Tan, Bocheng Chen, Guangliang Liu, Xi Chen
Computation and Language
Chinh Hoang and Mohammad Rashedul Hasan
Vision-
language models (VLMs) generate fluent causal explanations, but current evaluations cannot distinguish linguistic plausibility from faithful causal
reasoning. We introduce a dual-probe methodology that isolates these properties. The Text-Only Probe measures linguistic quality. The Chain-Text Probe requires models to first generate explicit causal chains. The Abstraction Gap (AG) metric quantifies the normalized performance difference. Evaluating eight VLMs on CAGE (Causal Abstraction Gap Evaluation), a
benchmark of 49,500 questions across 5,500 images spanning Pearl's causal hierarchy, we find seven models exhibit AG exceeding 0.50 with text scores of 6–8 but chain scores below 2.5. Fine-tuning on 45,000 chain-annotated examples fails to close the gap. However, one model achieves near-zero AG. The capability exists within current VLM architectures and depends on pretraining and architectural choices. CAGE provides a diagnostic tool for assessing faithful causal
reasoning in VLMs.
Read the full abstract ▾
Computation and Language
Can LLMs Use Linguistic Uncertainty Markers to Reliably Reflect Intrinsic Confidence?
Gabrielle Kaili-May Liu and Arman Cohan
LLMs' linguistically expressed confidence should faithfully reflect their intrinsic uncertainty. While recent work shows LLMs struggle to use epistemic markers (e.g., "it is likely...") in a human-aligned fashion, it remains unclear whether models can apply their own linguistic confidence framework to associate markers with specific confidence levels in a stable and generalizable way, and how contextual features impact this ability. We conduct the first systematic study of this question, formalizing _marker internal confidence_ (MIC) as the estimated intrinsic confidence a model associates with a specific epistemic marker in a given task domain. We present 7 metrics to evaluate the stability of MICs within and across distributions. Applying our analysis framework to diverse models and tasks, we find that LLMs remain faithfully miscalibrated even under model-centric interpretation of marker meanings, struggling to differentiate markers by internal confidence across distributions despite preserving a somewhat consistent ranking order across tasks. This supplies critical, complementary evidence to existing work toward a holistic understanding of faithful calibration in LLMs, emphasizing the need for more aligned and stable marker use to improve trustworthiness and reliability.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.LG
Learn from Weaknesses: Automated Domain Specialization for Small Computer-Use Agents
Suji Kim, Kangsan Kim and Sung Ju Hwang
Computer-use
agents (CUAs) have recently made substantial progress, but deploying a separate large expert for each software domain remains expensive. Small open computer-use
agents are more practical specialization targets, but they remain substantially weaker and exhibit uneven domain-specific failures. A straightforward remedy is to synthesize large-scale training data for the target domain, yet we find that this naive approach yields only marginal improvements. Building on this observation, we introduce LearnWeak, an annotation-free specialization framework for small computer-use
agents that uses a stronger reference
agent to identify the student's weaknesses in the target domain, synthesize targeted tasks, and construct supervision automatically. LearnWeak further introduces an error-aware specialization objective that disentangles planning and execution errors, enabling more behaviorally precise updates than broad uniform supervision. On OSWorld, LearnWeak achieves average gains of 11.6 and 11.1 percentage points over EvoCUA-8B and OpenCUA-7B, respectively, across eight domains. We also validate that our student-aware
dataset generation and training approaches outperform existing autonomous trajectory generation and training baselines. Our work highlights the importance of student awareness in both data synthesis and
agent training, pointing toward a more principled and efficient path for specializing small computer-use
agents in diverse domains.
Read the full abstract ▾
Computation and Language
Minki Kang, Shizhe Diao, Ryo Hachiuma et al. (+4 more)
Vision-
language models with extended
reasoning succeed on complex problems, but many real-world problems require external tools that internal
reasoning alone often cannot resolve. Agentic
reasoning therefore interleaves two behaviors with a structural asymmetry: thinking (the self-contained default) and tool use (a high-variance auxiliary acting). We refer to this asymmetry as the Thinking-Acting Gap. Under standard RL recipes like GRPO, the gap manifests as two diagnostic symptoms during training: tool use is attempted on only ~30% of rollouts, and when attempted, the tool-using rollouts within a group are all-wrong on ~40% of questions, suppressing the learning signal at the tool calls that needed it. We propose AXPO (
Agent eXplorative Policy Optimization): for each all-wrong tool-using subgroup, AXPO fixes the thinking prefix and resamples the tool call and its continuation, paired with uncertainty-based prefix selection. Across nine
multimodal benchmarks and three scales of Qwen3-VL-Thinking, SFT+AXPO outperforms SFT+GRPO at average (+1.8pp Pass@1 and +1.8pp Pass@4 at 8B on average) and 8B with SFT+AXPO surpasses the 32B Base on Pass@4 with 4 times fewer parameters.
Read the full abstract ▾
All authors Minki Kang, Shizhe Diao, Ryo Hachiuma, Sung Ju Hwang, Pavlo Molchanov, Yu-Chiang Frank Wang, Byung-Kwan Lee
Computation and Language
Rethinking Memory as Continuously Evolving Connectivity
Jizhan Fang, Buqiang Xu, Zhixian Wang et al. (+12 more)
Existing
memory-augmented LLM
agents often treat
memory as a static repository with pre-defined representations and fixed
retrieval pipelines, which is brittle in dynamic agentic environments where feedback, task variation, and heterogeneous signals continuously reshape what should be remembered and how it should be connected. To address this, we propose FluxMem, a connectivity-evolving
memory framework that models
memory as a heterogeneous graph and progressively refines its
topology through three stages: initial connection formation, feedback-driven refinement, and long-term consolidation. During execution, FluxMem repairs missing links, prunes interference, aligns abstraction granularity, and distills recurrent successful trajectories into reusable procedural
circuits, guided by one metric for
memory generalizability and evolutionary maturity. Across three fundamentally distinct
benchmarks including LoCoMo, Mind2Web, and
GAIA, FluxMem achieves consistent state-of-the-art performance, demonstrating strong adaptation and generalization in complex agentic environments. The code will be open-sourced in https://github.com/zjunlp/LightMem.
Read the full abstract ▾
All authors Jizhan Fang, Buqiang Xu, Zhixian Wang, Haoliang Cao, Xinle Deng, Baohua Dong, Hangcheng Zhu, Ruohui Huang, Gang Yu, Ying Wei, Guozhou Zheng, Feiyu Xiong, Haofen Wang, Huajun Chen, Ningyu Zhang
Computation and Language · Cross-listed from cs.LG
Extrapolative Weight Averaging Reveals Correctness-Efficiency Frontiers in Code RL
Kunhao Zheng, Pierre Chambon, Juliette Decugis et al. (+4 more)
Linear interpolation between fine-tuned checkpoints has been shown to trace the Pareto front between competing objectives, but whether extrapolative weight averaging can extend such frontiers to new checkpoints useful at inference time, without additional RL training, remains unclear. We study this question in RL for competitive programming, where hidden unit tests under time and
memory limits enforce both functional correctness and computational efficiency. Starting from a shared initialization, we train checkpoints under nested unit-test coverage: low-coverage rewards require passing smaller-input tests, while high-coverage rewards require passing progressively larger tests up to the full suite. This sweep reveals the emergence of a correctness-efficiency frontier: on hard problems, higher-coverage reward reduces optimization failures but increases correctness failures, leaving solve rate nearly unchanged. Interpolation between low- and high-coverage checkpoints recovers this frontier, while extrapolation extends it beyond the trained endpoints. Both the frontier and its extrapolative continuation appear across three inference settings, pure
reasoning, tool use, and agentic coding, and across two model scales, 32B and 7B. At the problem level, moving along the frontier changes which problems are solved, making extrapolated checkpoints complementary policies in inference-time scaling. Ensembles with extrapolative weight averaging broaden coverage and improve pass@250 on LCB/hard by 3.3% over the best single checkpoint at matched sample budget. These results show that nested unit-test coverage in code RL induces a frontier that extrapolative weight averaging can navigate, extend, and exploit.
Read the full abstract ▾
All authors Kunhao Zheng, Pierre Chambon, Juliette Decugis, Jonas Gehring, Taco Cohen, Benjamin Negrevergne, Gabriel Synnaeve
Computation and Language
Stance Detection in Prediction Markets: Addressing Imbalanced Trader Commentary via Counterfactual Augmentation and Market Context
Thomas Mbrice
Prediction
markets such as Polymarket aggregate crowd beliefs into real-time probability estimates, and the comments traders post beneath each
market contain rich directional stance signals that prices alone cannot capture. This work introduces the first stance detection study applied to prediction
market commentary, a domain characterized by extreme brevity, trader- specific vernacular, and severe class imbalance (only 8.7% of comments oppose the
market outcome). RoBERTa-base is fine-tuned across a 4 x 3 ablation: four input configurations ({2- class, 3-class} x {with/without
market context}) and three augmentation conditions (baseline, 50% synthetic, 100% synthetic). Synthetic minority-class samples are generated via LLM-driven Pro -> Anti counterfactual flips using the Anthropic API. Results show that (1)
market context is the single most impactful factor, raising 3-class Anti recall from 0.10 to 0.45; (2) counterfactual augmentation is conditionally effective, improving Anti F1 in weak configurations (0.10 -> 0.24) while degrading strong ones (2-class-ctx macro F1: 0.68 -> 0.50 at full dose); and (3) 50% augmentation is the optimal dose, with 100% consistently hurting performance.
Attention-based interpretability analysis provides mechanistic support for all three findings.
Read the full abstract ▾
Computation and Language
Reverse Probing: Supervised Token-level Uncertainty Quantification for Large Language Models in Clinical Text
Bushi Xiao, Sarvesh Soni and Daisy Zhe Wang
As large
language models are increasingly deployed for clinical text, ensuring they can reliably signal their own uncertainty becomes critical. Most existing uncertainty quantification (UQ) methods are designed for open-domain generation and cannot localize uncertainty at the
token or span level in long clinical text. We propose Reverse Probing, the first UQ framework specialized for clinical summarization, which estimates
token-level uncertainty directly from pre-existing labeled summaries. Rather than sampling new outputs, Reverse Probing treats the text as a probe into the model's internal state, extracting uncertainty signals from four categories of internal activations. We evaluate on two expert-annotated clinical
datasets and outperform eight adapted baselines on all metrics, achieving up to 4 times higher AUPRC while reducing inference time and computational costs. Feature analysis reveals that delta energy and neighborhood context are the most consistent predictors across all models. This study offers interpretable insights into how models internally respond to unsupported clinical content.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.CR
Code as a Weapon: A Consensus-Labeled Prompt Bank for Measuring Coding-Model Compliance with Malicious-Code Requests
Richard J. Young and Gregory D. Moody
A general-purpose
language model that answers a harmful question returns text; a coding model that complies with a malicious request can return a working weapon: a keylogger, ransomware, an exploit that runs as written. This asymmetry in the severity of a single act of compliance implies coding-specialized models should clear a higher refusal bar than general-purpose chat models, not a lower one, yet the field cannot tell whether they do. Refusal
benchmarks for malicious code are fragmented: they mix requests for executable software with requests for harmful security knowledge and report refusal rates over non-comparable corpora. This paper's central result is that the CODE-versus-KNOWLEDGE classification axis established in a prior four-corpus release remains stable under a substantially expanded corpus pool and an independently refreshed judge panel, evidence that it measures a real construct rather than an artifact of the
prompts or judges. Eight corpora spanning diverse elicitation paradigms (direct, jailbreak-decorated, indirect, and
agent/interpreter: ASTRA, CySecBench, AdvBench/harmful_behaviors, JailbreakBench, MalwareBench, RedCode, RMCBench, Scam2
Prompt) are classified under a five-judge consensus protocol (6,675
prompts x 5 judges = 33,375 calls), reaching Fleiss' kappa = 0.767 [95% CI 0.755, 0.777] ("substantial"). Critically, the panel shares no judge with the prior release (five paid commercial APIs replaced by five open-weight models from five vendors), yet the two panels agree on 94.45% of the 3,133 shared
prompts and reach Cohen's kappa = 0.952 [0.942, 0.963] on the 3,031-
prompt binary overlap: the axis survives near-total panel replacement. The released bank comprises 4,748 consensus-CODE and 1,923 consensus-KNOWLEDGE
prompts, a reliability-quantified
benchmark whose central classification axis is shown stable across corpus expansion and judge-panel replacement.
Read the full abstract ▾
Computation and Language
MemTrace: Tracing and Attributing Errors in Large Language Model Memory Systems
Xinle Deng, Ruobin Zhong, Hujin Peng et al. (+15 more)
Memory is essential for enabling large
language models to support long-horizon
reasoning, yet existing
memory systems remain unreliable and difficult to debug. Tracing
memory's dynamic
evolution is crucial to understand how information is synthesized, propagated, or corrupted over time. In this work, we study the new problem of error tracing and attribution in LLM
memory systems. We propose a novel framework that transforms
memory pipelines into executable
memory evolution graphs, enabling fine-grained tracing of operational information flow. We then construct MemTraceBench, a
benchmark collected from representative
memory systems such as Long-Context, RAG, Mem0, and EverMemOS, to systematically study
memory failure modes. We further introduce an automatic attribution method that iteratively traces operation subgraphs to pinpoint the root cause of any failed case. Our analysis reveals that
memory failures are systematic, stemming from operation-level issues like information loss and
retrieval misalignment. Crucially, we leverage these fine-grained attribution signals to guide downstream
prompt optimization, establishing a closed-loop system that automatically corrects faults and boosts end-task performance by up to 7.62 percentage points. Code has ben released at https://github.com/zjunlp/MemTrace.
Read the full abstract ▾
All authors Xinle Deng, Ruobin Zhong, Hujin Peng, Xiaoben Lu, Yanzhe Wu, Guang Li, Buqiang Xu, Yunzhi Yao, Jizhan Fang, Haoliang Cao, Junjie Guo, Yuan Yuan, Ziqing Ma, Yuanqiang Yu, Rui Hu, Baohua Dong, Hangcheng Zhu, Ningyu Zhang
Computation and Language
IPO-Mine: A Toolkit and Dataset for Section-Structured Analysis of Long, Multimodal IPO Documents
Michael Galarnyk, Siddharth Lohani, Vidhyakshaya Kannan et al. (+8 more)
An Initial Public Offering (IPO) filing is a document released when a private firm goes public, allowing individual (retail) investors to purchase its shares. These filings describe a firm's business, financials, and risks and are long,
multimodal documents with narrative text and images. Despite their importance to financial
markets, there is no large-scale, standardized
dataset or
benchmark for studying IPO filings with modern
language and
multimodal models. These documents pose significant challenges: filings frequently exceed 500,000
tokens and lack consistent structural organization. We introduce the IPO-Toolkit, an open-source framework for downloading and parsing IPO filings into standardized section-structured text and extracted images. The toolkit segments filings, extracts embedded images, and produces structured outputs that enable large-scale, reproducible analysis workflows over long,
multimodal documents. Using this infrastructure, we construct the IPO-
Dataset, a large, section-structured,
multimodal dataset covering more than 109,000 IPO filings and amendments from 1994 to 2026 and containing over 76,000 images. We establish structured evaluation tasks over extracted financial charts, including chart quality and misleadingness assessment. Our experiments show that state-of-the-art
multimodal models often diverge from expert human judgments on these tasks, exposing
alignment challenges in
multimodal reasoning over long, real-world regulatory documents. Beyond benchmarking, the IPO-
Dataset enables large-scale analysis of section-level textual variation and cross-industry differences in visual and textual disclosure practices. Our code,
dataset, and website are publicly available under CC-BY-4.0.
Read the full abstract ▾
All authors Michael Galarnyk, Siddharth Lohani, Vidhyakshaya Kannan, Sagnik Nandi, Aman Patel, Liqin Ye, Arnav Hiray, Rutwik Routu, Prasun Banerjee, Siddhartha Somani, Sudheer Chava
Computation and Language
Towards Reliable Multilingual LLMs-as-a-Judge: An Empirical Study
Irune Zubiaga, Aitor Soroa and Rodrigo Agerri
Large
language models (LLMs) are increasingly used for the automatic evaluation of generated text, yet most prior work focuses on English. Despite the growing demand for
multilingual evaluation, extending LLM-based evaluators to
multilingual settings remains challenging, particularly for low-resource
languages and scenarios where in-domain data is scarce. This work explores several strategies for developing
multilingual LLMs-as-a-judge, considering whether in-domain data is available for fine-tuning or not. We systematically analyze English, Spanish, and Basque, representing high-, mid-, and low-resource
languages, considering instruction translation, monolingual versus
multilingual supervision, and model size. For evaluation, we extend two existing meta-evaluation
datasets to Basque and Spanish. Our results reveal key trade-offs: When in-domain data is available, fine-tuned smaller models can achieve performance comparable to proprietary models, whereas zero-shot evaluation with larger models proves more effective in out-of-domain settings. We also observe that fine-tuning on out-of-domain data can adversely affect model performance. These findings provide practical guidance for building efficient, reliable
multilingual evaluation pipelines. The data and code are publicly available at hitz-zentroa/mJudge.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.AI
The Importance of Being Statistically Earnest: A Critical Re-evaluation of GSM-Symbolic
Dominika Agnieszka Długosz, Arlindo Oliveira and Natalia Díaz-Rodríguez
The GSM-Symbolic
benchmark (Mirzadeh et al., 2025) reported consistent performance drops across 25 Large
Language Models (LLMs) when tested on template-generated variants of GSM8K problems, concluding that the models lack genuine
reasoning capabilities. We argue that this conclusion rests on shaky statistical ground. Re-evaluating 20 open-weight models using bootstrapped Generalised Linear Mixed Models with per-question random effects, we find that only 8 exhibit statistically significant performance changes under the original
prompt format. Moreover, we identify a previously unacknowledged factor: the distribution of integers in problem texts of the main GSM-Symbolic
dataset is systematically shifted towards larger values relative to the original GSM8K (K-S statistic = 0.12, p < 0.001), contradicting the original authors' claims. Controlling for this large-number effect accounts for significance in half of the remaining cases. Among models with statistically significant performance deltas, we identify distinct, model-specific behavioural failure profiles – including fragility of variable binding, arithmetic limitations, and dual-task interference – underscoring that blanket claims about LLM
reasoning risk being both statistically premature and mechanistically misleading.
Read the full abstract ▾
Computation and Language
Sense Representations Are Inducible Interfaces
Jan Christian Blaise Cruz and Alham Fikri Aji
Sense representations (explicit, per-
token meaning decompositions) are useful for disambiguation, steering, and cross-lingual
alignment, but existing approaches require models to be pretrained with sense structure baked in. We introduce ACROS, which induces an explicit sense pathway into a frozen pretrained decoder LM through a gated residual addition. On SmolLM2-360M, ACROS preserves base LM quality while supporting three uses of the same induced variables: zero-shot word-sense disambiguation (64.95 F1 on Raganato ALL, competitive with the WordNet first-sense heuristic), low-KL lexical steering across 5,161 CoInCo cases where a simple non-oracle proxy recovers about 90% of positive shifts, and SENSIA cross-lingual adaptation to four
languages (mean R@1 0.988, target FLORES PPL 7.94). ACROS makes sense representations an inducible interface for ordinary pretrained LMs.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.LG
Activation Steering for Synthetic Data Generation: The Role of Diversity in Downstream Safety Detection
Vijeta Deshpande, Tootiya Giyahchi, Veena Padmanabhan et al. (+2 more)
Safety detection models require examples of HHH (Helpful, Harmless, Honest)-violating outputs for robust generalization, however such examples are scarce. Activation Steering (AS) has emerged as a data-efficient method for generating target-concept-aligned responses. We investigate whether AS can generate high-quality training
datasets for downstream classifiers, a question that remains untested. We present a two-fold study with intrinsic and extrinsic evaluation across $4$ concepts $\times\,2$ models $\times\,4$ steering methods. Intrinsically, beyond the field-standard rubric of steering success (concept
alignment) and coherence, we introduce sample- and set-level diversity as a quality axis previously absent from the literature, and find that increasing steering strength reduces response diversity. Extrinsically, we replace HHH-violating examples in the available training data with steered generations and fine-tune detection classifiers. AS-generated data results in a better classifier than the prompting-generated data on $3$ of $4$ concepts. However, only $41$ of $136$ AS configurations outperform prompting, indicating that downstream utility lies in a narrow regime that jointly satisfies success, coherence, and diversity. The harmonic mean of these three axes correlates with downstream AUROC more consistently across concepts than success and coherence alone, providing a practical heuristic target for practitioners tuning AS hyperparameters. Together, our results highlight the potential of AS in synthetic data generation for improving safety detection and identify diversity as a critical, previously overlooked axis for tuning AS.
Read the full abstract ▾
All authors Vijeta Deshpande, Tootiya Giyahchi, Veena Padmanabhan, Leman Akoglu, Anna Rumshisky
Computation and Language · Cross-listed from cs.LG
Interpretability-Guided Layer Selection over Subspace Projection: SAEs as Stethoscopes, Not Scalpels, for Raw Task Vector Model Editing
Li Lei, Madalina Ciobanu, Qingqing Mao et al. (+1 more)
LLMs increasingly require
surgical model editing to enhance domain-specific capabilities without incurring the computational cost or catastrophic forgetting associated with full fine-tuning. Sparse Autoencoders (SAEs) have emerged as a promising tool in this setting, in principle allowing for feature-level identification of where to intervene. In this work, we rigorously evaluate an SAE-guided editing pipeline for mathematical
reasoning on Gemma-3-4B-IT and uncover a fundamental failure mode: the intuitively appealing approach of projecting task vectors onto SAE feature subspaces acts as an information bottleneck that discards approximately 97% of the modification energy, yielding no statistically significant improvements across seven math subjects. We show that this failure stems from a
geometric misalignment between activation-space SAE directions and weight-space task vectors. We then propose a shift in perspective: SAE as a Stethoscope, Not a Scalpel, where SAEs are used for layer-level diagnosis rather than intervention-level filtering. By injecting unfiltered raw task vectors only into layers identified by an SAE-derived specificity score, we improve Number Theory accuracy from 29.6% to 39.4% (z=+3.41, p=0.0007) on the Minerva Math
benchmark; 5 of 7 math subjects significantly improved and none significantly degraded. Our method is fully deterministic, requires no additional inference cost, and provides a principled framework for interpretability-guided model editing.
Read the full abstract ▾
All authors Li Lei, Madalina Ciobanu, Qingqing Mao, Ritankar Das
Computation and Language · Cross-listed from cs.CR
MaskClaw: Edge-Side Personalized Privacy Arbitration for GUI Agents with Behavior-Driven Skill Evolution
Yanqiu Zhao, Dongying Zheng, Kaibo Huang et al. (+3 more)
GUI
agents rely on screenshots to infer intent and operate across applications, but these screenshots often contain private messages, medical records, payment credentials, and workplace-specific workflows.
Privacy decisions in this setting depend on task, recipient, application state, and user role, yet static PII detectors miss these boundaries and cloud-side VLM
reasoning can upload the raw screen before deciding what should be protected. We present MaskClaw, an edge-side
privacy arbitrator for GUI
agents. MaskClaw extracts local visual evidence, retrieves user- and task-specific policy
memory, and decides Allow, Mask, or Ask before raw screenshots leave a trusted user- or organization-controlled environment. In five designed skill-
evolution scenarios, it turns corrections, cancellations, and edits into reusable
privacy skills checked by a sandbox gate. We introduce P-GUI-Evo, a
benchmark built from real UI patterns, reconstructed HTML screens, and sanitized labels. Experiments show that pattern matching, cloud
reasoning, and routing alone tend to over-confirm, over-mask, or expose raw screenshots under the same protocol. The artifact is available at https://github.com/Theodora-Y/MaskClaw.
Read the full abstract ▾
All authors Yanqiu Zhao, Dongying Zheng, Kaibo Huang, Yukun Wei, Zhongliang Yang, Linna Zhou
Computation and Language · Cross-listed from cs.CR
GraphSteal: Structural Knowledge Stealing from Graph RAG via Traversal Reconstruction
Jinze Gu, Qinghua Mao, Xi Lin et al. (+1 more)
Retrieval-Augmented Generation (RAG) enhances LLMs by grounding generation in query-relevant external evidence. Beyond unstructured text corpora, Graph RAG integrates knowledge graphs into the
retrieval pipeline, enabling LLMs to access entities, relations, and multi-hop dependencies encoded in structured knowledge. However, the same structured knowledge that empowers Graph RAG also creates a new
privacy attack surface. We demonstrate that Graph RAG systems can be turned into structural oracles: through adaptive black-box interactions, an adversary can elicit sufficient relational evidence to reconstruct substantial portions of the hidden knowledge graph. We propose a structure-oriented reconstruction framework that recovers targeted graphs from both local and global perspectives. Specifically, Depth-Wise Heuristic Search extracts fine-grained node attributes by recursively expanding entity-centered evidence, while Breadth-Wise
Diffusion Search infers graph
topology by propagating across relation-induced neighborhoods. Experiments on generic and healthcare scenarios demonstrate that our method can recover over 90% of the original knowledge graph from representative Graph RAG systems, revealing sensitive entities, relations, and structural dependencies with high fidelity. Existing guradrails provide limited defense against our attack, highlighting the inherent difficulty of safeguarding structural
privacy in Graph RAG pipelines.
Read the full abstract ▾
All authors Jinze Gu, Qinghua Mao, Xi Lin, Jun Wu
Computation and Language
GraphLit: Learning Text-Enriched Dynamic Character Network Representations for Literary Study
Gaspard Michel, Elena V. Epure, Romain Hennequin et al. (+2 more)
Methods to represent literary texts as graphs or sequences of graphs mainly focus on representing character interactions, and often overlook another crucial aspect: the textual context in which characters interact. We introduce Dynamic Heterogeneous Character Networks (DHCNs), which organize long novels into temporally localized heterogeneous graphs that align characters with their textual contexts. We extract around 20,000 DHCNs from Project Gutenberg, and propose GraphLit, a self-supervised learning framework that learns rich literary representations through a masked graph autoencoder objective. Across a wide range of 12 character-related tasks, GraphLit improves over text-only, graph-only and prior hybrid baselines. Ablations over different kinds of dynamic graph structures and architectural elements show that grounding characters in their context is the main performance driver, while explicitly encoding narrative order and character relationships provide task-dependent improvements. Finally, we demonstrate the applicability of DHCNs and GraphLit for literary analysis by studying the link between narrative non-linearity and dynamic social features.
Read the full abstract ▾
All authors Gaspard Michel, Elena V. Epure, Romain Hennequin, Christophe Cerisara, Mirella Lapata