Computation and Language
MegaTrain: Full Precision Training of 100B+ Parameter Large Language Models on a Single GPU
Zhengqing Yuan, Hanchi Sun, Lichao Sun et al. (+1 more)
We present MegaTrain, a
memory-centric system that efficiently trains 100B+ parameter large
language models at full precision on a single GPU. Unlike traditional GPU-centric systems, MegaTrain stores parameters and optimizer states in host
memory (CPU
memory) and treats GPUs as transient compute engines. For each layer, we stream parameters in and compute gradients out, minimizing persistent device state. To battle the CPU-GPU bandwidth bottleneck, we adopt two key optimizations. 1) We introduce a pipelined double-buffered execution engine that overlaps parameter prefetching, computation, and gradient offloading across multiple CUDA streams, enabling continuous GPU execution. 2) We replace persistent autograd graphs with stateless layer templates, binding weights dynamically as they stream in, eliminating persistent graph metadata while providing flexibility in scheduling. On a single H200 GPU with 1.5TB host
memory, MegaTrain reliably trains models up to 120B parameters. It also achieves 1.84$\times$ the training throughput of DeepSpeed ZeRO-3 with CPU offloading when training 14B models. MegaTrain also enables 7B model training with 512k
token context on a single GH200.
Read the full abstract ▾
All authors Zhengqing Yuan, Hanchi Sun, Lichao Sun, Yanfang Ye
Computation and Language
Multilingual Language Models Encode Script Over Linguistic Structure
Aastha A K Verma, Anwoy Chatterjee, Mehak Gupta et al. (+1 more)
Multilingual language models (LMs) organize representations for typologically and orthographically diverse
languages into a shared parameter space, yet the nature of this internal organization remains elusive. In this work, we investigate which linguistic properties - abstract
language identity or surface-form cues - shape
multilingual representations. To do so, we analyze
language-associated units across different model families and scales using the
Language Activation Probability
Entropy (LAPE) metric, and further decompose activations with Sparse Autoencoders. We find that these units are strongly conditioned on orthography: romanization induces near-disjoint representations that align with neither native-script inputs nor English, while word-order shuffling has limited effect on unit identity. Probing shows that typological structure becomes increasingly accessible in deeper layers, while causal interventions indicate that generation is most sensitive to units that are invariant to surface-form perturbations rather than to units identified by typological
alignment alone. Overall, our results suggest that
multilingual LMs organize representations around surface form, with linguistic abstraction emerging gradually without collapsing into a unified interlingua.
Read the full abstract ▾
All authors Aastha A K Verma, Anwoy Chatterjee, Mehak Gupta, Tanmoy Chakraborty
Computation and Language
Document Optimization for Black-Box Retrieval via Reinforcement Learning
Omri Uzan, Ron Polonsky, Douwe Kiela et al. (+1 more)
Generative large
language models (LLMs) are increasingly used as inference-time components in
retrieval pipelines, for tasks such as query rewriting and document reranking. However, these online approaches place costly autoregressive computation directly on the latency-critical
retrieval path. We explore an alternative axis: using LLMs to improve documents instead, rewriting them into better representations and shifting computation offline. Yet producing a useful document rewrite is not straightforward:
retrieval is inherently discriminative, so an effective rewrite must make a document more similar to relevant queries than competing candidates under the retriever's notion of similarity. We therefore formulate document transformation as an optimization problem, directly training an LLM or VLM to produce rewrites that improve
retrieval. Our approach, DocOpt, uses GRPO with retriever ranking improvements as rewards, requires only black-box access to
retrieval ranks, and applies across single-vector, multi-vector, and lexical retrievers. We evaluate zero-shot LLM rewriting and DocOpt on code and visual
retrieval tasks, finding that document rewriting can improve
retrieval and that optimizing rewrites yields further gains. For example, OpenAI text-
embedding-3-small achieves 58.35 nDCG@5 on average with direct
retrieval; zero-shot rewriting improves this to 60.83 with GPT-5.4-mini, 63.75 with Claude Haiku 4.5, and 64.23 with Qwen3. DocOpt further improves performance to 67.94, surpassing the 6.5X more expensive text-
embedding-3-large retriever at 66.15.
Read the full abstract ▾
All authors Omri Uzan, Ron Polonsky, Douwe Kiela, Christopher Potts
Computation and Language
Beyond LLM-as-a-Judge: Deterministic Metrics for Multilingual Generative Text Evaluation
Firoj Alam, Gagan Bhatia, Sahinur Rahman Laskar et al. (+1 more)
While Large
Language Models (LLMs) are increasingly adopted as automated judges for evaluating generated text, their outputs are often costly, and highly sensitive to
prompt design,
language, and aggregation strategies, severely, which limits reproducibility. To address these challenges, we propose \textbf{OmniScore}, a family of complementary, deterministic learned metrics developed using small size ($<$1B) parameter models. OmniScore approximates LLM-judge behavior while preserving the low latency and consistency of traditional model-based scoring. We trained the models large-scale synthetic supervision ($\sim$564k instances, in 107
languages) and evaluated using 8,617 manually annotated instances. The OmniScore family supports reliable, multi-dimensional scores across a variety of settings, including reference-based, source-grounded, and hybrid evaluations. We evaluate these models across question answering (QA), translation, and summarization in 6
languages. Our results demonstrate that lightweight, deterministic learned metrics provide a highly practical and scalable alternative to frontier LLMs. Our models and
datasets can be found at https://huggingface.co/collections/QCRI/omniscore
Read the full abstract ▾
All authors Firoj Alam, Gagan Bhatia, Sahinur Rahman Laskar, Shammur Absar Chowdhury
Computation and Language · Cross-listed from cs.AI
MMORF: A Multi-agent Framework for Designing Multi-objective Retrosynthesis Planning Systems
Frazier N. Baker, Trieu Nguyen, Reza Averly et al. (+4 more)
Multi-objective retrosynthesis planning is a critical chemistry task requiring dynamic balancing of quality, safety, and cost objectives.
Language model-based multi-
agent systems (MAS) offer a promising approach for this task: leveraging interactions of specialized
agents to incorporate multiple objectives into retrosynthesis planning. We present MMORF, a framework for constructing MAS for multi-objective retrosynthesis planning. MMORF features modular agentic components, which can be flexibly combined and configured into different systems, enabling principled evaluation and comparison of different system designs. Using MMORF, we construct two representative MAS: MASIL and RFAS. On a newly curated
benchmark consisting of 218 multi-objective retrosynthesis planning tasks, MASIL achieves strong safety and cost metrics on soft-constraint tasks, frequently Pareto-dominating baseline routes, while RFAS achieves a 48.6% success rate on hard-constraint tasks, outperforming state-of-the-art baselines. Together, these results show the effectiveness of MMORF as a foundational framework for exploring MAS for multi-objective retrosynthesis planning. Code and data are available at https://github.com/ninglab/MMORF.
Read the full abstract ▾
All authors Frazier N. Baker, Trieu Nguyen, Reza Averly, Botao Yu, Daniel Adu-Ampratwum, Huan Sun, Xia Ning
Computation and Language
Memory Dial: A Training Framework for Controllable Memorization in Language Models
Xiangbo Zhang and Ali Emami
Memorization in
language models is widely studied but remains difficult to isolate and control. Understanding when and what models memorize is essential for explaining their predictions, yet existing approaches are post-hoc: they can detect memorization in trained models, but cannot disentangle its effects from architecture, data, or optimization. We introduce
Memory Dial, a training framework that makes memorization pressure an explicit, controllable variable.
Memory Dial interpolates between standard cross-
entropy and a temperature-sharpened objective via a single parameter $\alpha$, producing a family of models identical in architecture and training setup (within each sweep), differing only in memorization pressure. Experiments across six architectures and five
benchmarks demonstrate that: (1) $\alpha$ reliably controls memorization pressure, with seen-example accuracy increasing monotonically while unseen accuracy remains stable; (2) larger models are more responsive to memorization pressure; and (3) frequent sequences are easier to memorize than rare ones. Additional analyses show that the effect is robust across a range of sharpening temperatures, differs qualitatively from single-temperature cross-
entropy, transfers to
multilingual settings, and is detectable even on naturally occurring single-occurrence sequences.
Memory Dial provides a controlled experimental framework for studying how memorization behavior emerges and interacts with generalization in
language models.
Read the full abstract ▾
Computation and Language
This Treatment Works, Right? Evaluating LLM Sensitivity to Patient Question Framing in Medical QA
Hye Sun Yun, Geetika Kapoor, Michael Mackert et al. (+4 more)
Patients are increasingly turning to large
language models (LLMs) with medical questions that are complex and difficult to articulate clearly. However, LLMs are sensitive to
prompt phrasings and can be influenced by the way questions are worded. Ideally, LLMs should respond consistently regardless of phrasing, particularly when grounded in the same underlying evidence. We investigate this through a systematic evaluation in a controlled
retrieval-augmented generation (RAG) setting for medical question answering (QA), where expert-selected documents are used rather than retrieved automatically. We examine two dimensions of patient query variation: question framing (positive vs. negative) and
language style (technical vs. plain
language). We construct a
dataset of 6,614 query pairs grounded in clinical trial abstracts and evaluate response consistency across eight LLMs. Our findings show that positively- and negatively-framed pairs are significantly more likely to produce contradictory conclusions than same-framing pairs. This framing effect is further amplified in multi-turn conversations, where sustained persuasion increases inconsistency. We find no significant interaction between framing and
language style. Our results demonstrate that LLM responses in medical QA can be systematically influenced through query phrasing alone, even when grounded in the same evidence, highlighting the importance of phrasing robustness as an evaluation criterion for RAG-based systems in high-stakes settings.
Read the full abstract ▾
All authors Hye Sun Yun, Geetika Kapoor, Michael Mackert, Ramez Kouzy, Wei Xu, Junyi Jessy Li, Byron C. Wallace
Computation and Language
Phase-Associative Memory: Sequence Modeling in Complex Hilbert Space
Gowrav Vishwakarma and Christopher J. Agostino
Experiments probing natural
language processing by both humans and LLMs suggest that the meaning of a semantic expression is indeterminate prior to the act of interpretation rather than being specifiable simply as the sum of its parts (i.e. compositionality). This observer-dependent act dynamically actualizes meaning under genuine contextuality more consistent with
quantum logical mechanisms than with classical Boolean approaches that assume separability, motivating an approach to
language modeling that utilizes a
Hilbert space formalism. In this work, we introduce Phase-Associative
Memory (PAM) – a complex-valued sequence model whose state S_t \in \mathbb{C}^{d \times d} accumulates outer products of complex
token embeddings retrieved through the conjugate inner product $\mathrm{Re}\langle K \mid Q\rangle / \sqrt{d}$ – and evaluate it against a structurally matched real-valued ablation. Both architectures train stably across a 5M–100M parameter sweep on WikiText-103 under identical conditions; PAM sits at higher absolute loss at every measured scale but improves more rapidly with parameter count, with power-law exponents of $-0.15$ vs.\ $-0.12$ in loss and $-0.65$ vs.\ $-0.49$ in perplexity that narrow the gap between the two architectures monotonically. Further investigation of complex-valued sequence modeling at larger scales could reveal that the loss plateau characteristic of real-valued state-of-the-art
language models (e.g.
transformers) is reachable with PAM-style architectures with an order of magnitude fewer parameters than the current frontier ($\sim$1T), implying that similar capabilities are achievable at sizes runnable on consumer-grade hardware.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.CY
EduIllustrate: Towards Scalable Automated Generation Of Multimodal Educational Content
Shuzhen Bi, Mingzi Zhang, Zhuoxuan Li et al. (+3 more)
Large
language models are increasingly used as educational assistants, yet evaluation of their educational capabilities remains concentrated on question-answering and tutoring tasks. A critical gap exists for multimedia instructional content generation – the ability to produce coherent, diagram-rich explanations that combine geometrically accurate visuals with step-by-step
reasoning. We present EduIllustrate, a
benchmark for evaluating LLMs on interleaved text-diagram explanation generation for K-12 STEM problems. The
benchmark comprises 230 problems spanning five subjects and three grade levels, a standardized generation protocol with sequential anchoring to enforce cross-diagram visual consistency, and an 8-dimension evaluation rubric grounded in multimedia learning theory covering both text and visual quality. Evaluation of ten LLMs reveals a wide performance spread: Gemini 3.0 Pro Preview leads at 87.8%, while Kimi-K2.5 achieves the best cost-efficiency (80.8% at \\$0.12/problem). Workflow ablation confirms sequential anchoring improves Visual Consistency by 13\% at 94\% lower cost. Human evaluation with 20 expert raters validates LLM-as-judge reliability for objective dimensions ($\rho \geq 0.83$) while revealing limitations on subjective visual assessment.
Read the full abstract ▾
All authors Shuzhen Bi, Mingzi Zhang, Zhuoxuan Li, Xiaolong Wang, Keqian Li, Aimin Zhou
Computation and Language
Beyond the Final Actor: Modeling the Dual Roles of Creator and Editor for Fine-Grained LLM-Generated Text Detection
Yang Li, Qiang Sheng, Zhengjia Wang et al. (+3 more)
The misuse of large
language models (LLMs) requires precise detection of synthetic text. Existing works mainly follow binary or ternary classification settings, which can only distinguish pure human/LLM text or collaborative text at best. This remains insufficient for the nuanced regulation, as the LLM-polished human text and humanized LLM text often trigger different policy consequences. In this paper, we explore fine-grained LLM-generated text detection under a rigorous four-class setting. To handle such complexities, we propose RACE (Rhetorical Analysis for Creator-Editor Modeling), a fine-grained detection method that characterizes the distinct signatures of creator and editor. Specifically, RACE utilizes Rhetorical Structure Theory (RST) to construct a logic graph for the creator's foundation while extracting Elementary Discourse Unit (EDU)-level features for the editor's style. Experiments show that RACE outperforms 12 baselines in identifying fine-grained types with low false alarms, offering a policy-aligned solution for LLM regulation.
Read the full abstract ▾
All authors Yang Li, Qiang Sheng, Zhengjia Wang, Yehan Yang, Danding Wang, Juan Cao
Computation and Language
Early Stopping for Large Reasoning Models via Confidence Dynamics
Parsa Hosseini, Sumit Nawathe, Mahdi Salmani et al. (+2 more)
Large
reasoning models rely on long chain-of-thought generation to solve complex problems, but extended
reasoning often incurs substantial computational cost and can even degrade performance due to overthinking. A key challenge is determining when the model should stop
reasoning and produce the final answer. In this work, we study the confidence of intermediate answers during
reasoning and observe two characteristic behaviors: correct
reasoning trajectories often reach high-confidence answers early, while incorrect rollouts tend to produce long, unproductive
reasoning traces and exhibit less reliable confidence dynamics. Motivated by these observations, we propose CoDE-Stop (Confidence Dynamics Early Stop), an early stopping method that leverages the dynamics of intermediate answer confidence to decide when to terminate
reasoning, requiring no additional training and easily integrating into existing models. We evaluate CoDE-Stop on diverse
reasoning and science
benchmarks across multiple models. Compared to prior early stopping methods, it achieves a more favorable accuracy-compute tradeoff and reduces total
token usage by 25-50% compared to standard full-length
reasoning. In addition, we provide analyses of confidence dynamics during
reasoning, offering insights into how confidence changes in both correct and incorrect trajectories.
Read the full abstract ▾
All authors Parsa Hosseini, Sumit Nawathe, Mahdi Salmani, Meisam Razaviyayn, Soheil Feizi
Computation and Language
TriAttention: Efficient Long Reasoning with Trigonometric KV Compression
Weian Mao, Xi Lin, Wei Huang et al. (+5 more)
Extended
reasoning in large
language models (LLMs) creates severe KV cache
memory bottlenecks. Leading KV cache compression methods estimate KV importance using
attention scores from recent post-RoPE queries. However, queries rotate with position during RoPE, making representative queries very few, leading to poor top-key selection and unstable
reasoning. To avoid this issue, we turn to the pre-RoPE space, where we observe that Q and K vectors are highly concentrated around fixed non-zero centers and remain stable across positions – Q/K concentration. We show that this concentration causes queries to preferentially attend to keys at specific distances (e.g., nearest keys), with the centers determining which distances are preferred via a trigonometric series. Based on this, we propose TriAttention to estimate key importance by leveraging these centers. Via the trigonometric series, we use the distance preference characterized by these centers to score keys according to their positions, and also leverage Q/K norms as an additional signal for importance estimation. On AIME25 with 32K-
token generation, TriAttention matches Full
Attention reasoning accuracy while achieving 2.5x higher throughput or 10.7x KV
memory reduction, whereas leading baselines achieve only about half the accuracy at the same efficiency. TriAttention enables OpenClaw deployment on a single consumer GPU, where long context would otherwise cause out-of-
memory with Full
Attention.
Read the full abstract ▾
All authors Weian Mao, Xi Lin, Wei Huang, Yuxin Xie, Tianfu Fu, Bohan Zhuang, Song Han, Yukang Chen
Computation and Language · Cross-listed from cs.CV
Vero: An Open RL Recipe for General Visual Reasoning
Gabriel Sarch, Linrong Cai, Qunzhong Wang et al. (+3 more)
What does it take to build a visual reasoner that works across charts, science, spatial understanding, and open-ended tasks? The strongest
vision-
language models (VLMs) suggest that broad visual
reasoning is within reach, yet their closed data and reinforcement learning (RL) pipelines make their gains difficult to study, reproduce, or extend. We introduce Vero, a family of fully open VLMs that match or exceed existing open-weight models across diverse visual
reasoning tasks. We scale RL data and rewards across six broad task categories, constructing Vero-600K, a 600K-sample
dataset from 59
datasets, and designing task-routed rewards that handle heterogeneous answers. Across VeroEval, our 30-
benchmark suite, Vero-600K outperforms existing RL
datasets under controlled comparisons. Applied to five starting models, Vero variants gain 2.9-5.4 points on average over their initial models. Notably, Vero-Qwen3I-8B, trained on the Instruct model, surpasses Qwen3-VL-8B-Thinking by 3.8 points on average without additional
distillation. Systematic ablations reveal that different task categories elicit distinct
reasoning patterns and that broad gains depend on learning them jointly rather than in isolation. All data, code, and models are publicly available.
Read the full abstract ▾
All authors Gabriel Sarch, Linrong Cai, Qunzhong Wang, Haoyang Wu, Danqi Chen, Zhuang Liu
Computation and Language · Cross-listed from cs.AI
QED-Nano: Teaching a Tiny Model to Prove Hard Theorems
LM-Provers, Yuxiao Qu, Amrith Setlur et al. (+6 more)
Proprietary AI systems have recently demonstrated impressive capabilities on complex proof-based problems, with gold-level performance reported at the 2025 International Mathematical Olympiad (IMO). However, the training pipelines behind these systems remain largely undisclosed, and their reliance on large "internal" models and scaffolds makes them expensive to run, difficult to reproduce, and hard to study or improve upon. This raises a central question: can small, open models also be trained to achieve competitive
reasoning performance on difficult Olympiad-level math? In this paper, we answer this question by building QED-Nano, a 4B model post-trained for Olympiad-level proofs. Our training recipe has three stages: (1) supervised fine-tuning to imbue good proof-writing styles by distilling from DeepSeek-Math-V2, (2) reinforcement learning (RL) with rubric-based rewards, and (3) expanding RL with a
reasoning cache, which decomposes long proofs into iterative summarize-and-refine cycles and enables stronger test-time
reasoning. QED-Nano surpasses the proof-generation performance of much larger open models, including Nomos-1 and GPT-OSS-120B, and approaches the performance of proprietary models like Gemini 3 Pro, at a fraction of the inference cost. To support further research on open mathematical
reasoning, we release the full QED-Nano pipeline, including the QED-Nano and QED-Nano-SFT models, the FineProofs-SFT and FineProofs-RL
datasets, and the training and evaluation code.
Read the full abstract ▾
All authors LM-Provers, Yuxiao Qu, Amrith Setlur, Jasper Dekoninck, Edward Beeching, Jia Li, Ian Wu, Lewis Tunstall, Aviral Kumar
Computation and Language
Asymmetric Advantage Modulation Calibrates Entropy Dynamics in RLVR
Hengrui Gu, Xiaotian Han, Yujing Bian et al. (+2 more)
Reinforcement learning with verifiable rewards (RLVR) has substantially improved the
reasoning ability of large
language models (LLMs), but it often suffers from restricted exploration, where the policy rapidly concentrates on a narrow set of solutions. A common remedy is
entropy regularization, which attempts to preserve exploration by increasing policy
entropy. However, for LLM-RL, this intervention is highly sensitive to its coefficient, can introduce semantically weak uncertainty, and often yields limited accuracy gains. This motivates a more precise question: which
entropy helps
reasoning, and which
entropy should be reduced? To study this, we parameterize the advantage
estimator in Group Relative Policy Optimization (GRPO) into positive and negative outcome-conditioned channels and analyze their
entropy dynamics. Our results show that positive-channel modulation raises productive
entropy associated with successful
reasoning trajectories, while negative-channel modulation removes noisy
entropy associated with failed rollouts and reduces interference with correct paths. Guided by this channel-wise view, we propose AsymGRPO, which decouples the modulation strengths of positive and negative advantages. This enables flexible control over how the model updates across
prompt difficulty levels, allowing stronger reinforcement of rare successes on harder
prompts or stronger suppression of residual failures on easier
prompts without forcing the two channels to share the same modulation strength. Experiments on five mathematical
reasoning benchmarks show that AsymGRPO outperforms strong RLVR baselines, with consistent gains across model backbones.
Read the full abstract ▾
All authors Hengrui Gu, Xiaotian Han, Yujing Bian, Feiyi Wang, Kaixiong Zhou
Computation and Language
Synthetic Sandbox for Training Machine Learning Engineering Agents
Yuhang Zhou, Lizhu Zhang, Yifan Wu et al. (+4 more)
As large
language model
agents advance beyond software engineering (SWE) tasks toward machine learning engineering (MLE), verifying
agent behavior becomes orders of magnitude more expensive: while SWE tasks can be verified via fast-executing unit tests, MLE verification requires running full ML pipelines – data preprocessing, model training, and metric evaluation – on large
datasets at each rollout step, rendering trajectory-wise on-policy reinforcement learning (RL) prohibitively slow. Existing approaches retreat to supervised fine-tuning (SFT) or offline proxy rewards, sacrificing the exploration and generalization benefits of on-policy RL. We observe that sandbox data size is the primary source of this bottleneck. Based on this insight, we introduce SandMLE, a multi-
agent framework that generates diverse, verifiable synthetic MLE environments from a small number of seed tasks, preserving the structural and technical complexity of real-world problems while constraining
datasets to micro-scale (each task is paired with only 50-200 training samples). Through extensive experiments, we show that SandMLE reduces execution time by over 13 times, enabling large-scale, on-policy trajectory-wise RL for the first time in the MLE domain. On MLE-bench-lite, SandMLE yields significant gains over SFT baselines across Qwen3-8B, 14B, and 30B-A3B, with relative medal rate improvements ranging from 20.3% to 66.9%. Furthermore, the trained policy generalizes across unseen agentic scaffolds, achieving up to 32.4% better HumanRank score on MLE-Dojo.
Read the full abstract ▾
All authors Yuhang Zhou, Lizhu Zhang, Yifan Wu, Jiayi Liu, Xiangjun Fan, Zhuokai Zhao, Hong Yan
Computation and Language · Cross-listed from eess.AS
Full-Duplex-Bench-v3: Benchmarking Tool Use for Full-Duplex Voice Agents Under Real-World Disfluency
Guan-Ting Lin, Chen Chen, Zhehuai Chen et al. (+1 more)
We introduce Full-Duplex-Bench-v3 (FDB-v3), a
benchmark for evaluating spoken
language models under naturalistic
speech conditions and multi-step tool use. Unlike prior work, our
dataset consists entirely of real human audio annotated for five disfluency categories, paired with scenarios requiring chained API calls across four task domains. We evaluate six model configurations – GPT-Realtime, Gemini Live 2.5, Gemini Live 3.1, Grok, Ultravox v0.7, and a traditional Cascaded pipeline (Whisper$\rightarrow$GPT-4o$\rightarrow$TTS) – across accuracy, latency, and turn-taking dimensions. GPT-Realtime leads on Pass@1 (0.600) and interruption avoidance (13.5%); Gemini Live 3.1 achieves the fastest latency (4.25~s) but the lowest turn-take rate (78.0%); and the Cascaded baseline, despite a perfect turn-take rate, incurs the highest latency (10.12~s). Across all systems, self-correction handling and multi-step
reasoning under hard scenarios remain the most consistent failure modes.
Read the full abstract ▾
All authors Guan-Ting Lin, Chen Chen, Zhehuai Chen, Hung-yi Lee
Computation and Language
Do No Harm: Exposing Hidden Vulnerabilities of LLMs via Persona-based Client Simulation Attack in Psychological Counseling
Qingyang Xu, Yaling Shen, Stephanie Fong et al. (+7 more)
The increasing use of large
language models (LLMs) in mental healthcare raises safety concerns in high-stakes therapeutic interactions. A key challenge is distinguishing therapeutic empathy from maladaptive validation, where supportive responses may inadvertently reinforce harmful beliefs or behaviors in multi-turn conversations. This risk is largely overlooked by existing red-teaming frameworks, which focus mainly on generic harms or optimization-based attacks. To address this gap, we introduce Personality-based Client Simulation Attack (PCSA), the first red-teaming framework that simulates clients in psychological counseling through coherent, persona-driven client dialogues to expose vulnerabilities in psychological safety
alignment. Experiments on seven general and mental health-specialized LLMs show that PCSA substantially outperforms four competitive baselines. Perplexity analysis and human inspection further indicate that PCSA generates more natural and realistic dialogues. Our results reveal that current LLMs remain vulnerable to domain-specific
adversarial tactics, providing unauthorized medical advice, reinforcing delusions, and implicitly encouraging risky actions.
Read the full abstract ▾
All authors Qingyang Xu, Yaling Shen, Stephanie Fong, Zimu Wang, Yiwen Jiang, Xiangyu Zhao, Jiahe Liu, Zhongxing Xu, Vincent Lee, Zongyuan Ge
Computation and Language
MERIT: Multilingual Expert-Reward Informed Tuning for Chinese-Centric Low-Resource Machine Translation
Zhixiang Lu, Chong Zhang, Chenyu Xue et al. (+4 more)
Neural machine translation (NMT) from Chinese to low-resource Southeast Asian
languages remains severely constrained by the extreme scarcity of clean parallel corpora and the pervasive noise in existing mined data. This chronic shortage not only impedes effective model training but also sustains a large performance gap with high-resource directions, leaving millions of speakers of
languages such as Lao, Burmese, and Tagalog with persistently low-quality translation systems despite recent advances in large
multilingual models. We introduce
Multilingual Expert-Reward Informed Tuning (MERIT), a unified translation framework that transforms the traditional English-centric ALT
benchmark into a Chinese-centric evaluation suite for five Southeast Asian low-resource
languages (LRLs). Our framework combines
language-specific
token prefixing (LTP) with supervised fine-tuning (SFT) and a novel group relative policy optimization (GRPO) guided by the semantic
alignment reward (SAR). These results confirm that, in LRL{\textrightarrow}Chinese translation, targeted data curation and reward-guided optimization dramatically outperform mere model scaling.
Read the full abstract ▾
All authors Zhixiang Lu, Chong Zhang, Chenyu Xue, Angelos Stefanidis, Chong Li, Jionglong Su, Zhengyong Jiang
Computation and Language
Plausibility as Commonsense Reasoning: Humans Succeed, Large Language Models Do not
Sercan Karakaş
Large
language models achieve strong performance on many
language tasks, yet it remains unclear whether they integrate world knowledge with syntactic structure in a human-like, structure-sensitive way during ambiguity resolution. We test this question in Turkish prenominal relative-clause attachment ambiguities, where the same surface string permits high attachment (HA) or low attachment (LA). We construct ambiguous items that keep the syntactic configuration fixed and ensure both parses remain pragmatically possible, while graded event plausibility selectively favors High Attachment vs.\ Low Attachment. The contrasts are validated with independent norming ratings. In a speeded forced-choice comprehension experiment, humans show a large, correctly directed plausibility effect. We then evaluate Turkish and
multilingual LLMs in a parallel preference-based setup that compares matched HA/LA continuations via mean per-
token log-probability. Across models, plausibility-driven shifts are weak, unstable, or reversed. The results suggest that, in the tested models, plausibility information does not guide attachment preferences as reliably as it does in human judgments, and they highlight Turkish RC attachment as a useful cross-linguistic diagnostic beyond broad
benchmarks.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.AI
ANX: Protocol-First Design for AI Agent Interaction with a Supporting 3EX Decoupled Architecture
Xu Mingze
AI
agents, autonomous digital actors, need
agent-native protocols; existing methods include GUI automation and MCP-based skills, with defects of high
token consumption, fragmented interaction, inadequate security, due to lacking a unified top-level framework and key components, each independent module flawed. To address these issues, we present ANX, an open, extensible, verifiable
agent-native protocol and top-level framework integrating CLI, Skill, MCP, resolving pain points via protocol innovation, architectural optimization and tool supplementation. Its four core innovations: 1)
Agent-native design (ANX Config, Markup, CLI) with high information density, flexibility and strong adaptability to reduce
tokens and eliminate inconsistencies; 2) Human-
agent interaction combining Skill's flexibility for dual rendering as
agent-executable instructions and human-readable UI; 3) MCP-supported on-demand lightweight apps without pre-registration; 4) ANX Markup-enabled machine-executable SOPs eliminating ambiguity for reliable long-horizon tasks and multi-
agent collaboration. As the first in a series, we focus on ANX's design, present its 3EX decoupled architecture with ANXHub and preliminary feasibility analysis and experimental validation. ANX ensures native security: LLM-bypassed UI-to-Core communication keeps sensitive data out of
agent context; human-only confirmation prevents automated misuse. Form-filling experiments with Qwen3.5-plus/GPT-4o show ANX reduces
tokens by 47.3% (Qwen3.5-plus) and 55.6% (GPT-4o) vs MCP-based skills, 57.1% (Qwen3.5-plus) and 66.3% (GPT-4o) vs GUI automation, and shortens execution time by 58.1% and 57.7% vs MCP-based skills.
Read the full abstract ▾
Computation and Language
LiveFact: A Dynamic, Time-Aware Benchmark for LLM-Driven Fake News Detection
Cheng Xu, Changhong Jin, Yingjie Niu et al. (+5 more)
The rapid development of Large
Language Models (LLMs) has transformed fake news detection and fact-checking tasks from simple classification to complex
reasoning. However, evaluation frameworks have not kept pace. Current
benchmarks are static, making them vulnerable to
benchmark data contamination (BDC) and ineffective at assessing
reasoning under temporal uncertainty. To address this, we introduce LiveFact a continuously updated
benchmark that simulates the real-world "fog of war" in misinformation detection. LiveFact uses dynamic, temporal evidence sets to evaluate models on their ability to reason with evolving, incomplete information rather than on memorized knowledge. We propose a dual-mode evaluation: Classification Mode for final verification and Inference Mode for evidence-based
reasoning, along with a component to monitor BDC explicitly. Tests with 22 LLMs show that open-source Mixture-of-Experts models, such as Qwen3-235B-A22B, now match or outperform proprietary state-of-the-art systems. More importantly, our analysis finds a significant "
reasoning gap." Capable models exhibit epistemic humility by recognizing unverifiable claims in early data slices-an aspect traditional static
benchmarks overlook. LiveFact sets a sustainable standard for evaluating robust, temporally aware AI verification.
Read the full abstract ▾
All authors Cheng Xu, Changhong Jin, Yingjie Niu, Nan Yan, Yuke Mei, Shuhao Guan, Liming Chen, M-Tahar Kechadi
Computation and Language
SkillX: Automatically Constructing Skill Knowledge Bases for Agents
Chenxi Wang, Zhuoyun Yu, Xin Xie et al. (+8 more)
Learning from experience is critical for building capable large
language model (LLM)
agents, yet prevailing self-evolving paradigms remain inefficient:
agents learn in isolation, repeatedly rediscover similar behaviors from limited experience, resulting in redundant exploration and poor generalization. To address this problem, we propose SkillX, a fully automated framework for constructing a plug-and-play skill knowledge base that can be reused across
agents and environments. SkillX operates through a fully automated pipeline built on three synergistic innovations: (i) Multi-Level Skills Design, which distills raw trajectories into three-tiered hierarchy of strategic plans, functional skills, and atomic skills; (ii) Iterative Skills Refinement, which automatically revises skills based on execution feedback to continuously improve library quality; and (iii) Exploratory Skills Expansion, which proactively generates and validates novel skills to expand coverage beyond seed training data. Using a strong backbone
agent (GLM-4.6), we automatically build a reusable skill library and evaluate its transferability on challenging long-horizon, user-interactive
benchmarks, including AppWorld, BFCL-v3, and $\tau^2$-Bench. Experiments show that SkillKB consistently improves task success and execution efficiency when plugged into weaker base
agents, highlighting the importance of structured, hierarchical experience representations for generalizable
agent learning. Our code will be publicly available soon at https://github.com/zjunlp/SkillX.
Read the full abstract ▾
All authors Chenxi Wang, Zhuoyun Yu, Xin Xie, Wuguannan Yao, Runnan Fang, Shuofei Qiao, Kexin Cao, Guozhou Zheng, Xiang Qi, Peng Zhang, Shumin Deng
Computation and Language
How Far Are We? Systematic Evaluation of LLMs vs. Human Experts in Mathematical Contest in Modeling
Yuhang Liu, Heyan Huang, Yizhe Yang et al. (+3 more)
Large
language models (LLMs) have achieved strong performance on
reasoning benchmarks, yet their ability to solve real-world problems requiring end-to-end workflows remains unclear. Mathematical modeling competitions provide a stringent testbed for evaluating such end-to-end problem-solving capability. We propose a problem-oriented, stage-wise evaluation framework that assesses LLM performance across modeling stages using expert-verified criteria. We validate the framework's reliability by comparing automatic scores with independent human expert judgments on problems from the China Postgraduate Mathematical Contest in Modeling, demonstrating substantially stronger
alignment than existing evaluation schemes. Using this framework, we reveal a comprehension-execution gap in state-of-the-art LLMs: while they perform well in early stages such as problem identification and formulation, they exhibit persistent deficiencies in execution-oriented stages including model solving, code implementation, and result analysis. These gaps persist even with increased model scale. We further trace these failures to insufficient specification, missing verification, and lack of validation, with errors propagating across stages without correction. Our findings suggest that bridging this gap requires approaches beyond model scaling, offering insights for applying LLMs to complex real-world problem solving.
Read the full abstract ▾
All authors Yuhang Liu, Heyan Huang, Yizhe Yang, Hongyan Zhao, Zhizhuo Zeng, Yang Gao
Computation and Language
HUKUKBERT: Domain-Specific Language Model for Turkish Law
Mehmet Utku Öztürk, Tansu Türkoğlu and Buse Buz-Yalug
Natural
language processing (NLP) advances have powered a generation of LegalTech systems, but Turkish law remains under-served by domain-specific data and models. English has legal encoders such as LEGAL-BERT; no comparable high-volume Turkish counterpart exists. We introduce HukukBERT, a Turkish legal
language model trained on a 19 GB cleaned corpus using a hybrid domain-adaptive pre-training (DAPT) recipe that mixes Whole-Word Masking,
Token Span Masking, Word Span Masking, and targeted Keyword Masking. We compared our 48K WordPiece tokenizer and DAPT pipeline against general-purpose and existing domain-specific Turkish models. On the Legal Cloze Test - a masked legal term prediction
benchmark over Turkish court decisions - HukukBERT reaches 84.40% Top-1 accuracy and beats every baseline we tested. The Legal Cloze Test is synthetically constructed, so its passages are absent from the pre-training corpus by construction, eliminating train-test contamination. On the downstream task of structural
segmentation of official Turkish court decisions, it reaches a 92.8% document pass rate. We release HukukBERT to support Turkish legal NLP work in named entity recognition, judgment prediction, and document classification.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.CV
MinerU2.5-Pro: Pushing the Limits of Data-Centric Document Parsing at Scale
Bin Wang, Tianyao He, Linke Ouyang et al. (+40 more)
Current document parsing methods advance primarily through model architecture innovation, while systematic engineering of training data remains underexplored. Yet state-of-the-art models spanning diverse architectures and parameter scales exhibit highly consistent failure patterns on the same set of hard samples, suggesting that the performance bottleneck stems from shared deficiencies in training data rather than from architectural differences. Building on this finding, we present MinerU2.5-Pro, which advances the state of the art purely through data engineering and training strategy design while retaining the 1.2B-parameter architecture of MinerU2.5 unchanged. At its core is a Data Engine co-designed around coverage, informativeness, and annotation accuracy: Diversity-and-Difficulty-Aware Sampling expands training data from under 10M to 65.5M samples while mitigating distribution shift; Cross-Model Consistency Verification leverages output consensus among heterogeneous models to assess sample difficulty and generate reliable annotations; the Judge-and-Refine pipeline improves annotation quality for hard samples through render-then-verify iterative correction. A three-stage progressive training strategy–large-scale pre-training, hard sample fine-tuning, and GRPO
alignment–sequentially exploits these data at different quality tiers. On the evaluation front, we rectify element-matching biases in OmniDocBench v1.5 and introduce a Hard subset, establishing the more discriminative OmniDocBench v1.6 protocol. Without any architectural modification, MinerU2.5-Pro achieves 95.69 on OmniDocBench v1.6, improving over the same-architecture baseline by 2.71 points and surpassing all existing methods, including those based on models with over 200x more parameters.
Read the full abstract ▾
All authors Bin Wang, Tianyao He, Linke Ouyang, Fan Wu, Zhiyuan Zhao, Tao Chu, Yuan Qu, Zhenjiang Jin, Weijun Zeng, Ziyang Miao, Bangrui Xu, Junbo Niu, Mengzhang Cai, Jiantao Qiu, Qintong Zhang, Dongsheng Ma, Yuefeng Sun, Hejun Dong, Wenzheng Zhang, Jutao Xiao, Jiayong Shi, Pengyu Liao, Xiaomeng Zhao, Huaping Zhong, Liqun Wei, Jing Yu, Jie Yang, Wei Li, Shasha Wang, Qianqian Wu, Xuanhe Zhou, Weijia Li, Zhenxiang Li, Zhongying Tu, Jiang Wu, Lijun Wu, Chao Xu, Kai Chen, Wentao Zhang, Yu Qiao, Bowen Zhou, Dahua Lin, Conghui He
Computation and Language · Cross-listed from cs.LG
Cog-DRIFT: Exploration on Adaptively Reformulated Instances Enables Learning from Hard Reasoning Problems
Justin Chih-Yao Chen, Archiki Prasad, Zaid Khan et al. (+4 more)
Reinforcement learning from verifiable rewards (RLVR) has improved the
reasoning abilities of LLMs, yet a fundamental limitation remains: models cannot learn from problems that are too difficult to solve under their current policy, as these yield no meaningful reward signal. We propose a simple yet effective solution based on task reformulation. We transform challenging open-ended problems into cognitively simpler variants – such as multiple-choice and cloze formats – that preserve the original answer while reducing the effective search space and providing denser learning signals. These reformulations span a spectrum from discriminative to generative tasks, which we exploit to bootstrap learning: models first learn from structured, easier formats, and this knowledge transfers back to improve performance on the original open-ended problems. Building on this insight, we introduce Cog-DRIFT, a framework that constructs reformulated variants and organizes them into an adaptive curriculum based on difficulty. Training progresses from easier to harder formats, enabling the model to learn from problems that previously yielded zero signal under standard RL post-training. Cog-DRIFT not only improves on the originally unsolvable hard problems (absolute +10.11% for Qwen and +8.64% for Llama) but also generalizes well to other held-out
datasets. Across 2 models and 6
reasoning benchmarks, our method consistently outperforms standard GRPO and strong guided-exploration baselines. On average, Cog-DRIFT shows +4.72% (Qwen) and +3.23% (Llama) improvements over the second-best baseline. We further show that Cog-DRIFT improves pass@k at test time, and the curriculum improves sample efficiency. Overall, our results highlight task reformulation and curriculum learning as an effective paradigm for overcoming the exploration barrier in LLM post-training.
Read the full abstract ▾
All authors Justin Chih-Yao Chen, Archiki Prasad, Zaid Khan, Joykirat Singh, Runchu Tian, Elias Stengel-Eskin, Mohit Bansal
Computation and Language · Cross-listed from cs.CR
Your Agent, Their Asset: A Real-World Safety Analysis of OpenClaw
Zijun Wang, Haoqin Tu, Letian Zhang et al. (+11 more)
OpenClaw, the most widely deployed personal AI
agent in early 2026, operates with full local system access and integrates with sensitive services such as Gmail, Stripe, and the filesystem. While these broad privileges enable high levels of automation and powerful personalization, they also expose a substantial attack surface that existing sandboxed evaluations fail to capture. To address this gap, we present the first real-world safety evaluation of OpenClaw and introduce the CIK taxonomy, which unifies an
agent's persistent state into three dimensions, i.e., Capability, Identity, and Knowledge, for safety analysis. Our evaluations cover 12 attack scenarios on a live OpenClaw instance across four backbone models (Claude Sonnet 4.5, Opus 4.6, Gemini 3.1 Pro, and GPT-5.4). The results show that poisoning any single CIK dimension increases the average attack success rate from 24.6% to 64-74%, with even the most robust model exhibiting more than a threefold increase over its baseline vulnerability. We further assess three CIK-aligned defense strategies alongside a file-protection mechanism; however, the strongest defense still yields a 63.8% success rate under Capability-targeted attacks, while file protection blocks 97% of malicious injections but also prevents legitimate updates. Taken together, these findings show that the vulnerabilities are inherent to the
agent architecture, necessitating more systematic safeguards to secure personal AI
agents. Our project page is https://ucsc-vlaa.github.io/CIK-Bench.
Read the full abstract ▾
All authors Zijun Wang, Haoqin Tu, Letian Zhang, Hardy Chen, Juncheng Wu, Xiangyan Liu, Zhenlong Yuan, Tianyu Pang, Michael Qizhe Shieh, Fengze Liu, Zeyu Zheng, Huaxiu Yao, Yuyin Zhou, Cihang Xie
Computation and Language · Cross-listed from cs.LG
Darkness Visible: Reading the Exception Handler of a Language Model
Peter Balogh
The final MLP of GPT-2 Small exhibits a fully legible routing program – 27 named
neurons organized into a three-tier exception handler – while the knowledge it routes remains entangled across ~3,040 residual
neurons. We decompose all 3,072
neurons (to numerical precision) into: 5 fused Core
neurons that reset vocabulary toward function words, 10 Differentiators that suppress wrong candidates, 5 Specialists that detect structural boundaries, and 7 Consensus
neurons that each monitor a distinct linguistic dimension. The consensus-exception crossover – where MLP intervention shifts from helpful to harmful – is statistically sharp (bootstrap 95% CIs exclude zero at all consensus levels; crossover between 4/7 and 5/7). Three experiments show that "knowledge
neurons" (Dai et al., 2022), at L11 of this model, function as routing infrastructure rather than fact storage: the MLP amplifies or suppresses signals already present in the residual stream from
attention, scaling with contextual constraint. A garden-path experiment reveals a reversed garden-path effect – GPT-2 uses verb subcategorization immediately, consistent with the exception handler operating at
token-level predictability rather than syntactic structure. This architecture crystallizes only at the terminal layer – in deeper models, we predict equivalent structure at the final layer, not at layer 11. Code and data: https://github.com/pbalogh/transparent-gpt2
Read the full abstract ▾
Computation and Language
Hallucination Basins: A Dynamic Framework for Understanding and Controlling LLM Hallucinations
Kalyan Cherukuri and Lav R. Varshney
Large
language models (LLMs) hallucinate: they produce fluent outputs that are factually incorrect. We present a
geometric dynamical systems framework in which
hallucinations arise from task-dependent basin structure in latent space. Using autoregressive hidden-state trajectories across multiple open-source models and
benchmarks, we find that separability is strongly task-dependent rather than universal: factoid settings can show clearer basin separation, whereas summarization and misconception-heavy settings are typically less stable and often overlap. We formalize this behavior with task-complexity and multi-basin
theorems, characterize basin emergence in L-layer
transformers, and show that geometry-aware steering can reduce
hallucination probability without retraining.
Read the full abstract ▾