Computation and Language
Latent Abstraction for Retrieval-Augmented Generation
Ha Lan N.T, Minh-Anh Nguyen and Dung D. Le
Retrieval-Augmented Generation (RAG) has become a standard approach for enhancing large
language models (LLMs) with external knowledge, mitigating
hallucinations, and improving factuality. However, existing systems rely on generating natural
language queries at each hop and maintaining a strict architectural separation between retriever and generator, preventing them from leveraging the full representational capacity of the LLM. We propose LAnR (Latent Abstraction for RAG), a unified framework in which a single LLM jointly performs encoding,
retrieval, and generation entirely within its own latent space. Rather than generating textual queries, LAnR produces dense
retrieval vectors from the hidden states of a designated [PRED]
token and uses them to match against encoded document representations from the same model. Furthermore, LAnR adaptively decides when sufficient evidence has been retrieved using a lightweight MLP control head over those same hidden states, eliminating both the separate retriever and explicit
token-level stopping
reasoning. This design is motivated by our empirical observation that answer
token entropy reliably signals
retrieval sufficiency. Extensive experiments on six QA
benchmarks spanning single-hop and multi-hop settings demonstrate that LAnR outperforms existing RAG methods, while achieving improved inference efficiency through reduced number of
retrieval calls and tighter model integration.
Read the full abstract ▾
Computation and Language
On the Emergence of Syntax by Means of Local Interaction
Zichao Wei
Can syntactic processing emerge spontaneously from purely local interaction? We present a concrete instance on a minimal system: an 18,658-parameter two-dimensional neural cellular automaton (NCA), supervised by nothing more than a 1-bit boundary signal, is trained on the membership problem of an arithmetic-expression grammar. After training, its internal $L \times L$ grid spontaneously self-organizes into an ordered, spatially extended representation that we name Proto-CKY. This representation satisfies three operational criteria for syntactic processing: expressive power beyond the regular
languages, structural generalization beyond the training distribution, and an internal organization quantitatively aligned with grammatical structure (Pearson $r \approx 0.71$). It emerges independently on four context-free grammars and regenerates spontaneously after perturbation. Proto-CKY is functionally aligned with the CKY
algorithm but formally distinct from it: it is a physical prototype, a concrete instantiation of a mathematical ideal on a physical substrate, and the systematic distance between the two carries information about the substrate itself.
Read the full abstract ▾
Computation and Language
QuickScope: Certifying Hard Questions in Dynamic LLM Benchmarks
Taylor Lundy, Narun K. Raman and Kevin Leyton-Brown
LLM
benchmarks are increasingly dynamic: instead of containing a fixed set of questions, they define templates and parameters that can generate an effectively unlimited number of question variants. This flexibility is valuable, but it makes evaluation expensive – especially when the goal is not just determining an average score, but reliably identifying a model's weak spots. This paper introduces a new methodology for identifying hard questions in dynamic
benchmarks. It leverages COUP, a recent
Bayesian optimization
algorithm (Graham, Velez & Leyton-Brown, 2026), after introducing several substantive modifications to make the
algorithm suitable for practical LLM pipelines. We also wrap it in a tool that supports flexible choices of
datasets and utility functions, enabling users to target the kinds of questions they care about (e.g., low-accuracy questions; questions that are unusually hard relative to their measured complexity). In experiments across a range of
benchmarks, we show that our method, dubbed $\texttt{QuickScope}$, discovers truly difficult questions more sample efficiently than standard baselines, while also reducing false positives from noisy outcomes.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.AI
Polysemantic Experts, Monosemantic Paths: Routing as Control in MoEs
Charles Ye, Bo Yuan and Lee Sharkey
An LLM's residual stream is both state and instruction: it encodes the current context and determines the next transformation. We introduce a parameter-free decomposition for Mixture-of-Experts models that splits each layer's hidden state into a control signal that causally drives routing and an orthogonal content channel invisible to the router. Across six MoE architectures, we find that models preserve surface-level features (
language,
token identity, position) in the content channel, while the control signal encodes an abstract function that rotates from layer to layer. Because each routing decision is low-bandwidth, this hand-off forces compositional specialization across layers. While individual experts remain polysemantic, expert paths become monosemantic, clustering
tokens by semantic function across
languages and surface forms. The same
token (e.g., ":") follows distinct trajectories depending on whether it serves as a type annotation, an introductory colon, or a time separator. Our decomposition identifies the source of this structure: clusters in the control subspace are substantially more monosemantic than those in the full representation. As a result, the natural unit of interpretability in MoEs is not the expert but the trajectory.
Read the full abstract ▾
Computation and Language
How Non-Linguistic Is the Indus Sign System? A Synthetic-Baseline Scorecard
Ashish Nair
Whether the Indus Valley sign system (c. 2600-1900 BCE) encodes spoken
language has been debated for decades. This paper introduces a multi-metric discrimination framework that tests the observed Indus corpus against two kinds of computer-generated non-linguistic baseline – one mimicking a heraldic emblem system, the other an administrative coding system – each calibrated with Zipfian frequency distributions, positional constraints, and bigram dependencies derived from six attested non-linguistic corpora. The scorecard evaluates four properties central to the Farmer-Sproat-Witzel (2004) critique: text brevity, repeated formulaic phrases, hapax legomenon rate, and positional
rigidity. Applying this framework to 1,916 deduplicated inscriptions (584 unique signs, 11,110
tokens) from the ICIT/Yajnadevam digitization, we find that the Indus corpus does not match either baseline cleanly. Across the four metrics examined, the Indus corpus occupies an intermediate position relative to the two baseline families, matching neither cleanly. Neither a heraldic nor an administrative generator can reproduce all four properties at once. We also compare against seven real-world non-linguistic corpora including Sproat's (2014)
datasets, finding that no attested non-linguistic system reproduces the full Indus statistical profile either. We replicate key prior results including a Zipf slope of -1.49 and conditional
entropy of 3.23 bits. All code and data are publicly available.
Read the full abstract ▾
Computation and Language
Local-First Intelligence via Proactive SLM Collaborating with Feedback-Providing LLM
Hang Zeng, Xiangyu Liu, Yong Hu et al. (+5 more)
Personal intelligence requires strong capabilities under strict
privacy and efficiency constraints. Unlike cloud-first intelligence, which relies on powerful but costly and
privacy-vulnerable large
language models (LLMs), we focus on local-first intelligence, emphasizing on-device small
language models (SLMs), which enable efficient, private inference but have limited capacity. To reconcile the capability gap with
privacy and efficiency, we propose a dynamic collaboration framework, where an SLM learns to proactively decide how to consult an LLM during multi-step
reasoning, while the LLM provides adaptive feedback instead of acting as a passive tool, modeling them as heterogeneous
agents under information asymmetry. We further systematically investigate how collaboration strategies are shaped by SLM and LLM capabilities as well as efficiency and
privacy constraints. Evaluation results reveal a distinct scaling effect: stronger SLMs become more self-reliant, while stronger LLMs enable fewer and more informative interactions. In addition, the learned dynamic collaboration strategies significantly outperform various baselines and transfer robustly to unseen LLMs as well as to real-world personal tasks without additional training.
Read the full abstract ▾
All authors Hang Zeng, Xiangyu Liu, Yong Hu, Chaoyue Niu, Jiarui Zhang, Shaojie Tang, Fan Wu, Guihai Chen
Computation and Language · Cross-listed from cs.SD
A novel LSTM music generator based on the fractional time-frequency feature extraction
Li Ya, Chen Wei, Li Xiulai et al. (+3 more)
In this paper, we propose a novel approach for generating music based on an artificial intelligence (AI) system. We analyze the features of music and use them to fit and predict the music. The fractional
Fourier transform (FrFT) and the long short-term
memory (LSTM) network are the foundations of our method. The FrFT method is used to extract the spectral features of a music piece, where the music signal is expressed on the time and frequency domains. The LSTM network is used to generate new music based on the extracted features, where we predict the music according to the hidden layer features and real-time inputs using GiantMIDI-Piano
dataset. The results of our experiments show that our proposed system is capable of generating high-quality music that is comparable to human-generated music.
Read the full abstract ▾
All authors Li Ya, Chen Wei, Li Xiulai, Yu Lei, Deng Xinyi, Chen Chaofan
Computation and Language
Algorithmic Primitives of Theory of Mind Benchmarks
Qiutong Tony Yi, Wang Bill Zhu, Robin Jia et al. (+1 more)
Do Theory of Mind (ToM)
benchmarks really test theory of mind, or are they actually testing parsing and algorithmic state tracking? While
benchmarks vary modality and dialogue length, this paper finds that they re-encode the same few state transition
algorithms. We create PDDL-Mind, a neuro-symbolic approach that recovers such
algorithms and upstream inductive biases of ToM
benchmarks explicitly by parsing narratives into the Planning Domain Definition
Language (PDDL) and executes candidate actions against a predefined set of update
algorithms. Doing so provides program-verified state traces to the model. We use our method's performance gain to audit how much of a
benchmark's difficulty our inductive bias absorbs. Across MMToM, MuMA, and FanToM, outsourcing parsing to programs alone brings performance close to strong agentic pipelines, while outsourcing state tracking to PDDL moves GPT-4o to close-to-human performance. We demonstrate that complex ToM
benchmarks primarily punishes Large
Language Models' unreliable implicit implementation of state tracking and parsing
algorithms rather than limitations in belief and counterfactual
reasoning.
Read the full abstract ▾
All authors Qiutong Tony Yi, Wang Bill Zhu, Robin Jia, Jesse Thomason
Computation and Language · Cross-listed from cs.HC
Navigating the Conceptual Multiverse
Andre Ye, Jenny Y. Huang, Alicia Guo et al. (+3 more)
When
language models answer open-ended questions, they implicitly make decisions that shape their answers. However, the models do not show these decisions to the user, so users cannot contextualize answers: they cannot see the assumptions an answer rests on or the alternatives to it. Drawing on multiverse analysis from statistics, we introduce the conceptual multiverse, a representation in which
language models express answers as the result of intermediate conceptual decisions. We build a system that generates such decisions explicitly, organizes them into a space of answers that users can see and change, and checks them against the norms of a specific domain. In a user study across three domains - philosophy, AI
alignment annotation, and poetry - participants say that our system helps them see the decisions leading to an answer, explore the space of answers, and understand the decisions structuring this space.
Read the full abstract ▾
All authors Andre Ye, Jenny Y. Huang, Alicia Guo, Rose Novick, Tamara Broderick, Mitchell L. Gordon
Computation and Language · Cross-listed from cs.AR
Enabling AI ASICs for Zero Knowledge Proof
Jianming Tong, Jingtian Dang, Simon Langowski et al. (+6 more)
Zero-knowledge proof (ZKP) provers remain costly because multi-scalar multiplication (MSM) and number-theoretic transforms (NTTs) dominate runtime as they need significant computation. AI ASICs such as TPUs provide massive
matrix throughput and SotA energy efficiency. We present MORPH, the first framework that reformulates ZKP kernels to match AI-ASIC execution. We introduce Big-T complexity, a hardware-aware complexity model that exposes heterogeneous bottlenecks and layout-transformation costs ignored by Big-O. Guided by this analysis, (1) at arithmetic level, MORPH develops an MXU-centric extended-RNS lazy reduction that converts high-precision modular arithmetic into dense low-precision GEMMs, eliminating all carry chains, and (2) at dataflow level, MORPH constructs a unified-sharding layout-stationary TPU Pippenger MSM and optimized 3/5-step NTT that avoid on-TPU shuffles to minimize costly
memory reorganization. Implemented in JAX, MORPH enables TPUv6e8 to achieve up-to 10x higher throughput on NTT and comparable throughput on MSM than GZKP. Our code: https://github.com/EfficientPPML/MORPH.
Read the full abstract ▾
All authors Jianming Tong, Jingtian Dang, Simon Langowski, Tianhao Huang, Asra Ali, Jeremy Kun, Jevin Jiang, Srinivas Devadas, Tushar Krishna
Computation and Language
Bridging the Reasoning Gap in Vietnamese with Small Language Models via Test-Time Scaling
Bui The Trung, Do Minh Duc, Nguyen Van Vinh et al. (+1 more)
The democratization of ubiquitous AI hinges on deploying sophisticated
reasoning capabilities on resource-constrained devices. However, Small
Language Models (SLMs) often face a "
reasoning gap", particularly in non-English
languages like Vietnamese, where they struggle to maintain coherent chains of thought. This paper investigates Test-Time Scaling strategies for the Qwen3-1.7B architecture within the context of Vietnamese Elementary Mathematics. We introduce Vi-S1K, a high-fidelity
reasoning dataset localized via a Gemini 2.5 Flash-Lite powered pipeline, and Vi-Elementary-Bench, a dual-resource
benchmark for rigorous evaluation. Using an LLM-as-a-Judge protocol, we reveal that the base model possesses robust latent knowledge (Accuracy: 4.05/5.00) but suffers from a severe "formatting gap" in communication. Supervised Fine-Tuning (SFT) acts as a critical "
reasoning unlocker", yielding a 77% improvement in Explanation Quality and bridging the gap between raw calculation and pedagogical coherence. Furthermore, our analysis of prompting strategies uncovers a significant trade-off: structured frameworks like ReAct impose a "cognitive tax" on the 1.7B parameter capacity, degrading performance relative to pure Chain-of-Thought (CoT) combined with Self-Consistency. These findings establish a deployment hierarchy for SLMs, demonstrating that SFT combined with simplified test-time scaling is superior to complex agentic workflows for edge-based
reasoning.
Read the full abstract ▾
All authors Bui The Trung, Do Minh Duc, Nguyen Van Vinh, Bui Nguyen Quoc Trinh
Computation and Language · Cross-listed from cs.CV
DuQuant++: Fine-grained Rotation Enhances Microscaling FP4 Quantization
Haokun Lin, Xinle Jia, Haobo Xu et al. (+7 more)
The MXFP4 microscaling format, which partitions tensors into blocks of 32 elements sharing an E8M0 scaling factor, has emerged as a promising substrate for efficient LLM inference, backed by native hardware support on NVIDIA Blackwell Tensor Cores. However, activation outliers pose a unique challenge under this format: a single outlier inflates the shared block scale, compressing the effective dynamic range of the remaining elements and causing significant quantization error. Existing rotation-based remedies, including randomized Hadamard and learnable rotations, are data-agnostic and therefore unable to specifically target the channels where outliers concentrate. We propose DuQuant++, which adapts the outlier-aware fine-grained rotation of DuQuant to the MXFP4 format by aligning the rotation block size with the microscaling group size (B{=}32). Because each MXFP4 group possesses an independent scaling factor, the cross-block variance issue that necessitates dual rotations and a zigzag permutation in the original DuQuant becomes irrelevant, enabling DuQuant++ to replace the entire pipeline with a single outlier-aware rotation, which halves the online rotation cost while simultaneously smoothing the weight distribution. Extensive experiments on the LLaMA-3 family under MXFP4 W4A4 quantization show that DuQuant++ consistently achieves state-of-the-art performance. Our code is available at https://github.com/Hsu1023/DuQuant-v2.
Read the full abstract ▾
All authors Haokun Lin, Xinle Jia, Haobo Xu, Bingchen Yao, Xianglong Guo, Yichen Wu, Zhichao Lu, Ying Wei, Qingfu Zhang, Zhenan Sun
Computation and Language
Forget What Matters, Keep the Rest: Selective Unlearning of Informative Tokens
Seunghee Koh, Sunghyun Baek, Youngdong Kim et al. (+1 more)
Unlearning in large
language models (LLMs) has emerged as a promising safeguard against
adversarial behaviors. When the forgetting loss is applied uniformly without considering
token-level semantic importance, model utility can be unnecessarily degraded. Recent studies have explored
token-wise loss regularizers that prioritize informative
tokens, but largely rely on ground-truth confidence or external linguistic parsers, which limits their ability to capture contextual information or the model's overall predictive state. Intuitively, function words like "the" primarily serve syntactic roles and are highly predictable with little ambiguity, but informative words admit multiple plausible alternatives with greater uncertainty. Based on this intuition, we propose
Entropy-guided
Token Weighting (ETW), a
token-level unlearning regularizer that uses
entropy of the predictive distribution as a proxy for
token informativeness. We demonstrate that informative
tokens tend to have higher
entropy, whereas structural
tokens tend to have lower
entropy. This behavior enables ETW to achieve more effective unlearning while better preserving model utility than existing
token-level approaches.
Read the full abstract ▾
All authors Seunghee Koh, Sunghyun Baek, Youngdong Kim, Junmo Kim
Computation and Language
SPENCE: A Syntactic Probe for Detecting Contamination in NL2SQL Benchmarks
Mohammadtaher Safarzadeh, Hitesh Laxmichand Patel, Afshin Orojlooyjadid et al. (+2 more)
Large
language models (LLMs) have achieved strong performance on natural
language to SQL (NL2SQL)
benchmarks, yet their reported accuracy may be inflated by contamination from
benchmark queries or structurally similar patterns seen during training. We introduce SPENCE (Syntactic Probing and Evaluation of NL2SQL Contamination Effects), a controlled syntactic probing framework for detecting and quantifying such contamination. SPENCE systematically generates syntactic variants of test queries for four widely used NL2SQL
datasets-Spider, SParC, CoSQL, and the newer BIRD
benchmark. We use SPENCE to evaluate multiple high-capacity LLMs under execution-based scoring. For each model, we measure changes in execution accuracy across increasing levels of syntactic divergence and quantify rank sensitivity using Kendall's tau with bootstrap confidence intervals. By aligning these robustness trends with
benchmark release dates, we observe a clear temporal gradient: older
benchmarks such as Spider exhibit the strongest negative values and thus the highest likelihood of training leakage, whereas the more recent BIRD
dataset shows minimal sensitivity and appears largely uncontaminated. Together, these findings highlight the importance of temporally contextualized, syntactic-probing evaluation for trustworthy NL2SQL benchmarking.
Read the full abstract ▾
All authors Mohammadtaher Safarzadeh, Hitesh Laxmichand Patel, Afshin Orojlooyjadid, Graham Horwood, Dan Roth
Computation and Language
Reverse Constitutional AI: A Framework for Controllable Toxic Data Generation via Probability-Clamped RLAIF
Yuan Fang, Yiming Luo, Aimin Zhou et al. (+1 more)
Ensuring the safety of large
language models (LLMs) requires robust red teaming, yet the systematic synthesis of high-quality toxic data remains under-explored. We propose Reverse Constitutional AI (R-CAI), a framework for automated and controllable
adversarial data generation that moves beyond isolated jailbreak
prompts. By inverting a harmless constitution into a constitution of toxicity and iteratively refining model outputs through a critique–revision pipeline, R-CAI enables scalable synthesis of multi-dimensional
adversarial data without human annotation. Optimizing solely for toxicity-related rewards, however, can lead to reward hacking and degraded semantic coherence. To address this challenge, we introduce probability clamping within reinforcement learning from AI feedback, which stabilizes
adversarial optimization while preserving
adversarial intent. Experiments demonstrate that R-CAI generates diverse, high-quality toxic data and that probability clamping substantially improves semantic coherence (15%) without sacrificing
adversarial strength. Overall, R-CAI provides a fully automated framework for red teaming data generation and systematic safety evaluation of aligned
language models.
Read the full abstract ▾
All authors Yuan Fang, Yiming Luo, Aimin Zhou, Fei Tan
Computation and Language · Cross-listed from cs.AI
Contrastive Attribution in the Wild: An Interpretability Analysis of LLM Failures on Realistic Benchmarks
Rongyuan Tan, Jue Zhang, Zhuozhao Li et al. (+3 more)
Interpretability tools are increasingly used to analyze failures of Large
Language Models (LLMs), yet prior work largely focuses on short
prompts or toy settings, leaving their behavior on commonly used
benchmarks underexplored. To address this gap, we study contrastive, LRP-based attribution as a practical tool for analyzing LLM failures in realistic settings. We formulate failure analysis as contrastive attribution, attributing the logit difference between an incorrect output
token and a correct alternative to input
tokens and internal model states, and introduce an efficient extension that enables construction of cross-layer attribution graphs for long-context inputs. Using this framework, we conduct a systematic empirical study across
benchmarks, comparing attribution patterns across
datasets, model sizes, and training checkpoints. Our results show that this
token-level contrastive attribution can yield informative signals in some failure cases, but is not universally applicable, highlighting both its utility and its limitations for realistic LLM failure analysis. Our code is available at: https://aka.ms/Debug-XAI.
Read the full abstract ▾
All authors Rongyuan Tan, Jue Zhang, Zhuozhao Li, Qingwei Lin, Saravan Rajmohan, Dongmei Zhang
Computation and Language · Cross-listed from cs.AI
Evolutionary Negative Module Pruning for Better LoRA Merging
Anda Cao, Zhuo Gou, Yi Wang et al. (+5 more)
Merging multiple Low-Rank Adaptation (LoRA) experts into a single backbone is a promising approach for efficient multi-task deployment. While existing methods strive to alleviate interference via weight interpolation or subspace
alignment, they rest upon the implicit assumption that all LoRA
matrices contribute constructively to the merged model. In this paper, we uncover a critical bottleneck in current merging paradigms: the existence of $\textit{negative modules}$ – specific LoRA layers that inherently degrade global performance upon merging. We propose $\textbf{E}$volutionary $\textbf{N}$egative $\textbf{M}$odule $\textbf{P}$runing ($\textbf{ENMP}$), a plug-and-play LoRA
pruning method to locate and exclude these detrimental modules prior to merging. By leveraging an evolutionary search strategy, ENMP effectively navigates the discrete, non-differentiable landscape of module selection to identify optimal
pruning configurations. Extensive evaluations demonstrate that ENMP consistently boosts the performance of existing merging
algorithms, achieving a new state-of-the-art across both
language and
vision domains. Code is available at https://github.com/CaoAnda/ENMP-LoRAMerging.
Read the full abstract ▾
All authors Anda Cao, Zhuo Gou, Yi Wang, Kaixuan Chen, Yu Wang, Can Wang, Mingli Song, Jie Song
Computation and Language · Cross-listed from cs.LG
HiP-LoRA: Budgeted Spectral Plasticity for Robust Low-Rank Adaptation
Lixian Chen and Jianhong Tan
Adapting foundation models under resource budgets relies heavily on Parameter-Efficient Fine-Tuning (PEFT), with LoRA being a standard modular solution. However, LoRA suffers from spectral interference. Low-rank updates often concentrate energy on the leading singular directions of pretrained weights, perturbing general capabilities and causing catastrophic forgetting and fragile multi-adapter merging. To resolve this, we propose HiP-LoRA, a spectrum-aware adaptation framework. Utilizing the cached singular value decomposition (SVD) of pretrained layers, HiP-LoRA decomposes updates into two channels: a principal channel within the dominant singular subspace, and a residual low-rank channel in the orthogonal complement. A singular-value-weighted stability budget on the principal channel continuously balances pretrained behavior preservation with task-specific plasticity. Experiments on Llama-3.1-8B demonstrate that under matched budgets, HiP-LoRA drastically reduces pretraining degradation and multi-adapter MergeFail, robustly outperforming baselines in interference-sensitive tasks like continual tuning and knowledge editing.
Read the full abstract ▾
Computation and Language
HiRAS: A Hierarchical Multi-Agent Framework for Paper-to-Code Generation and Execution
Hanhua Hong, Yizhi LI, Jiaoyan Chen et al. (+4 more)
Recent advances in large
language models have highlighted their potential to automate computational research, particularly reproducing experimental results. However, existing approaches still use fixed sequential
agent pipelines with weak global coordination, which limits their robustness and overall performance. In this work, we propose Hierarchical Research
Agent System (HiRAS), a hierarchical multi-
agent framework for end-to-end experiment reproduction that employs supervisory manager
agents to coordinate specialised
agents across fine-grained stages. We also identify limitations in the reference-free evaluation of the Paper2Code
benchmark and introduce Paper2Code-Extra (P2C-Ex), a refined protocol that incorporates repository-level information and better aligns with the original reference-based metric. We conduct extensive evaluation, validating the effectiveness and robustness of our proposed methods, and observing improvements, including >10% relative performance gain beyond the previous state-of-the-art using open-source backbone models and significantly reduced
hallucination in evaluation. Our work is available on GitHub: https://github.com/KOU-199024/HiRAS.
Read the full abstract ▾
All authors Hanhua Hong, Yizhi LI, Jiaoyan Chen, Sophia Ananiadou, Xiaoli Li, Jung-jae Kim, Chenghua Lin
Computation and Language · Cross-listed from cs.LG
Democratizing Tool Learning with Environments Fully Simulated by a Free 8B Language Model
Chenming Tang, Hsiu-Yuan Huang, Weijie Liu et al. (+3 more)
Reinforcement learning has become a prevalent paradigm for training tool calling
agents, which typically requires online interactive environments. Existing approaches either rely on training data with ground truth annotations or require advanced proprietary
language models (LMs) to synthesize environments that remain fixed once created. In this work, we propose TRUSTEE, a cost-friendly method for training tool calling
agents with dynamic environments fully simulated by open-source LMs that can be as small as 8B, including task generation, user simulation, tool simulation, and trajectory evaluation, paired with an adaptive curriculum learning mechanism that controls task difficulty dynamically. Our empirical results show that TRUSTEE outperforms baselines which require extra external resources in most cases. These confirm that, with a sufficiently sophisticated design, even simulated environments with a local 8B LM as the backbone could set a strong baseline for tool learning. We hope our proposed paradigm could democratize tool learning and inspire future research on environment scaling with limited resources.
Read the full abstract ▾
All authors Chenming Tang, Hsiu-Yuan Huang, Weijie Liu, Junqiang Zheng, Saiyong Yang, Yunfang Wu
Computation and Language
Mira-Embeddings-V1: Domain-Adapted Semantic Reranking for Recruitment via LLM-Synthesized Data
Zhaohua Liang, Zhilin Wang, Renjie Cao et al. (+1 more)
Candidate sourcing for recruiters is best viewed as a two-stage
retrieval and reranking pipeline with recall as the primary objective under a limited review budget. An upstream production retriever first returns a candidate shortlist for each job description (JD), and our goal is to rerank that shortlist so that qualified candidates appear as high as possible. We present mira-
embeddings-v1, a semantic reranking system for the recruitment domain that reshapes the
embedding space with LLM-synthesized training data and corrects boundary confusions with a lightweight reranking head. Starting from real JDs, we build a five-stage
prompt pipeline to generate diverse positive and hard negative samples that sculpt the semantic space from multiple angles. We then apply a two-round LoRA adaptation: JD–JD contrastive training followed by JD–CV triplet
alignment on a heterogeneous text
dataset. Importantly, these gains require no large-scale manually labeled industrial training pairs: a modest set of real JDs is expanded into supervision through LLM synthesis. Finally, a BoundaryHead MLP reranks the Top-K results to distinguish between roles that share the same title but differ in scope. On a local pool of 300 real JDs with candidates from an upstream production retriever, mira-
embeddings-v1 improves Recall@50 from 68.89% (baseline) to 77.55% while lifting Precision@10 from 35.77% to 39.62%. On a supportive global pool over 44,138 candidates judged by a Qwen3-32B rubric, it achieves Recall@200 of 0.7047 versus 0.5969 for the baseline. These results show that LLM-synthesized supervision with boundary-aware reranking yields robust gains without a heavy cross-encoder.
Read the full abstract ▾
All authors Zhaohua Liang, Zhilin Wang, Renjie Cao, Yining Zhang
Computation and Language
MHSafeEval: Role-Aware Interaction-Level Evaluation of Mental Health Safety in Large Language Models
Suhyun Lee, Palakorn Achananuparp, Neemesh Yadav et al. (+2 more)
Large
language models (LLMs) are increasingly explored as scalable tools for mental health counseling, yet evaluating their safety remains challenging due to the interactional and context-dependent nature of clinical harm. Existing evaluation frameworks predominantly assess isolated responses using coarse-grained taxonomies or static
datasets, limiting their ability to diagnose how harms emerge and accumulate over multi-turn counseling interactions. In this work, we introduce R-MHSafe, a role-aware mental health safety taxonomy that characterizes clinically significant harm in terms of the interactional roles an AI counselor adopts, including perpetrator, instigator, facilitator, or enabler, combined with clinically grounded harm categories. Then, we propose MHSafeEval, a closed-loop,
agent-based evaluation framework that formulates safety assessment as trajectory-level discovery of harm through
adversarial multi-turn interactions, guided by role-aware modeling. Using R-MHSafe and MHSafeEval, we conduct a large-scale evaluation across state-of-the-art LLMs. Our results reveal substantial role-dependent and cumulative safety failures that are systematically missed by existing static
benchmarks, and show that our framework significantly improves failure-mode coverage and diagnostic granularity.
Read the full abstract ▾
All authors Suhyun Lee, Palakorn Achananuparp, Neemesh Yadav, Ee-Peng Lim, Yang Deng
Computation and Language
RePrompT: Recurrent Prompt Tuning for Integrating Structured EHR Encoders with Large Language Models
Arya Hadizadeh Moghaddam, Drew Ross, Mohsen Nayebi Kerdabadi et al. (+2 more)
Large
Language Models (LLMs) have shown strong promise for mining Electronic Health Records (EHRs) by
reasoning over longitudinal clinical information to capture context-rich patient trajectories. However, leveraging LLMs for structured EHRs (e.g., standardized diagnosis and medication codes) presents two key challenges. First, translating time-stamped EHR sequences into plain text can obscure both temporal structure and code identities, weakening the ability to capture code co-occurrence and longitudinal regularities. Second, unlike cohort-trained predictive models that learn a shared, task-aligned representation space across patients, LLMs are often applied in a case-isolated inference setting where each patient is processed independently without leveraging population-level patterns. To address these challenges, we introduce RePrompT, a time-aware LLM framework that integrates structured EHR encoders through
prompt tuning, without modifying underlying architectures. Specifically, RePrompT recurrently incorporates latent states from prior visits to preserve longitudinal information, and injects population-level information through trainable
prompt tokens derived from a cohort-trained, task-aligned EHR encoder. Experiments on MIMIC-III and MIMIC-IV demonstrate that RePrompT consistently outperforms both EHR-based and LLM-based baselines across multiple clinical prediction tasks.
Read the full abstract ▾
All authors Arya Hadizadeh Moghaddam, Drew Ross, Mohsen Nayebi Kerdabadi, Dongjie Wang, Zijun Yao
Computation and Language
Do LLMs Use Cultural Knowledge Without Being Told? A Multilingual Evaluation of Implicit Pragmatic Adaptation
Mehwish Nasim, Sanjeevan Selvaganapathy, Neel Ganapathi Sabhahit et al. (+6 more)
Many
benchmarks show that large
language models can answer direct questions about culture. We study a different question: do they also change how they speak when culture is only implied by the situation? We evaluate 60 culturally grounded conversational scenarios across five
languages in three conditions: a neutral baseline (
Prompt A), an explicit cultural instruction (
Prompt B), and implicit situational cueing (
Prompt C). We score responses on 12 pragmatic features covering deference to authority, individual-versus-group framing, and uncertainty management. We define Pragmatic Context Sensitivity (PCS) as the fraction of the
Prompt A->B shift that reappears under
Prompt A->C. Across four deployed LLMs and five
languages (English, German, Hindi, Nepali, Urdu), the primary stable-only PCS mean is 0.196 (SD = 0.113), indicating that the models recover only about one-fifth of the pragmatic shift they can produce when instructed explicitly. Transfer is strongest for authority-related cues (0.299) and weakest for individual-versus-group framing (0.120). Uncertainty-related behaviour is mixed: hedging density exhibits negative explicit gaps in all five
languages, suggesting that
alignment training actively suppresses the target behaviour. Because Hindi and Urdu share core grammar yet index distinct cultural communities, we use them as a natural control; a paired analysis finds no reliable baseline difference (t = 0.96, p = 0.339, dz = 0.06), suggesting that models respond primarily to linguistic structure rather than to the cultural associations a
language carries. We argue that
multilingual cultural pragmatics is an explicit-versus-implicit deployment problem, not only a factual knowledge problem.
Read the full abstract ▾
All authors Mehwish Nasim, Sanjeevan Selvaganapathy, Neel Ganapathi Sabhahit, Marie Griesbach, Pranav Bhandari, Janina Lütke Stockdiek, Lennart Schäpermeier, Usman Naseem, Christian Grimme
Computation and Language
Concurrent Criterion Validation of a Validity Screen for LLM Confidence Signals via Selective Prediction
Jon-Paul Cacioli
The validity screen (Cacioli, 2026d, 2026e) classifies LLM confidence signals as Valid, Indeterminate, or Invalid. We test whether these classifications predict selective prediction performance. Twenty frontier LLMs from seven families were evaluated on 524 items across six cognitive tracks. Valid models show mean Type 2 AUROC = .624 (SD = .048). Invalid models show mean AUROC = .357 (SD = .231). Cohen's d = 2.81, p = .002. The tiers order monotonically: Invalid (.357) < Indeterminate (.554) < Valid (.624). Split-half cross-validation yields median d = 1.77, P(d > 0) = 1.0 across 1,000 splits. The three-tier classification accounts for 47% of the variance in AUROC. DeepSeek-R1 drops from 85.3% accuracy at full coverage to 11.3% at 10% coverage. The screen predicts the criterion. For selective prediction, the screen matters.
Read the full abstract ▾
Computation and Language
Screen Before You Interpret: A Portable Validity Protocol for Benchmark-Based LLM Confidence Signals
Jon-Paul Cacioli
LLM confidence signals are used for abstention, routing, and safety-critical decisions. No standard practice exists for checking whether a confidence signal carries item-level information before building on it. We transfer the validity screening principle from clinical personality assessment (PAI, MMPI-3) as a portable protocol for
benchmark-based LLM confidence data. The protocol specifies three core indices (L, Fp, RBS), a structural indicator (TRIN), and an item-sensitivity statistic, computed from a single 2x2 contingency table. A three-tier classification system (Invalid, Indeterminate, Valid) draws on four clinical traditions. Validated on 20 frontier LLMs across 524 items, four models are classified Invalid, two Indeterminate. Valid-profile models show mean r = .18 (15/16 significant). Invalid-profile models show mean r = -.20 (d = 2.48). Cross-
benchmark validation on 18 models using MMLU with verbalized confidence and on external data from Yang et al. (2024) confirms the screen transfers across
benchmarks and probe formats. All data and code: https://github.com/synthiumjp/validity-scaling-llm
Read the full abstract ▾
Computation and Language
DeInfer: Efficient Parallel Inferencing for Decomposed Large Language Models
You-Liang Huang, Xinhao Huang, Chengxi Liao et al. (+1 more)
Existing works on large
language model (LLM) decomposition mainly focus on improving performance on downstream tasks, but they ignore the poor parallel inference performance when trying to scale up the model size. To mitigate this important performance issue, this paper introduces DeInfer, a high-performance inference system dedicated to parallel inference of decomposed LLMs. It consists of multiple optimizations to maximize performance and be compatible with state-of-the-art optimization techniques. Extensive experiments are carried out to evaluate DeInfer's performance, where the results demonstrate its superiority, suggesting it can greatly facilitate the parallel inference of decomposed LLMs.
Read the full abstract ▾
All authors You-Liang Huang, Xinhao Huang, Chengxi Liao, Zeyi Wen
Computation and Language
Before You Interpret the Profile: Validity Scaling for LLM Metacognitive Self-Report
Jon-Paul Cacioli
Clinical personality assessment screens response validity before interpreting substantive scales. LLM evaluation does not. We apply the validity scaling framework from the PAI and MMPI-3 to metacognitive probe data from 20 frontier models across 524 items. Six validity indices are operationalised: L (maintaining confidence on errors), K (betting on errors), F (withdrawing consensus-endorsed items), Fp (withdrawing correct answers), RBS (inverted monitoring), and TRIN (fixed responding). A tiered classification system identifies four models as construct-level invalid and two as elevated. Valid-profile models produce item-sensitive confidence (mean r = .18, 14 of 16 significant). Invalid-profile models do not (mean r = -.20, d = 2.17, p = .001). Chain-of-thought training produces two opposite response distortions. Two latent dimensions account for 94.6% of index variance. Companion papers extract a portable screening protocol (Cacioli, 2026e) and validate it against selective prediction (Cacioli, 2026f). All data and code: https://github.com/synthiumjp/validity-scaling-llm
Read the full abstract ▾
Computation and Language · Cross-listed from cs.LG
The Geometric Canary: Predicting Steerability and Detecting Drift via Representational Stability
Prashant C. Raju
Reliable deployment of
language models requires two capabilities that appear distinct but share a common
geometric foundation: predicting whether a model will accept targeted behavioral control, and detecting when its internal structure degrades. We show that
geometric stability, the consistency of a representation's pairwise distance structure, addresses both. Supervised Shesha variants that measure task-aligned
geometric stability predict linear steerability with near-perfect accuracy ($\rho = 0.89$-$0.97$) across 35-69
embedding models and three NLP tasks, capturing unique variance beyond class separability (partial $\rho = 0.62$-$0.76$). A critical dissociation emerges: unsupervised stability fails entirely for steering on real-world tasks ($\rho \approx 0.10$), revealing that task
alignment is essential for controllability prediction. However, unsupervised stability excels at drift detection, measuring nearly $2\times$ greater
geometric change than CKA during post-training
alignment (up to $5.23\times$ in Llama) while providing earlier warning in 73% of models and maintaining a $6\times$ lower false alarm rate than Procrustes. Together, supervised and unsupervised stability form complementary diagnostics for the LLM deployment lifecycle: one for pre-deployment controllability assessment, the other for post-deployment monitoring.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.LG
MoE-nD: Per-Layer Mixture-of-Experts Routing for Multi-Axis KV Cache Compression
Libo Sun, Peixiong He, Po-Wei Harn et al. (+1 more)
KV cache
memory is the dominant bottleneck for long-context LLM inference. Existing compression methods each act on a single axis of the four-dimensional KV tensor –
token eviction (sequence), quantization (precision), low-rank projection (head dimension), or cross-layer sharing – but apply the same recipe to every layer. We show that this homogeneity leaves accuracy on the table: different layers respond very differently to each compression operation, and the optimal per-layer mix of eviction and quantization is far from uniform. We propose MoE-nD, a mixture-of-experts framework that routes each layer to its own (eviction-ratio, K-bits, V-bits) tuple under a global
memory budget. An offline-calibrated greedy solver chooses the routing that minimizes predicted quality loss; at inference time, per-layer heterogeneous eviction and quantization are applied jointly through a single
attention patch. On a 4-task subset of LongBench-v1 (16k inputs, n=50 per task, adapted
reasoning-model protocol; see section Experiments), MoE-nD's hetero variant matches our uncompressed 1.9~GB baseline at 14x compression (136~MB) while every other compressed baseline we tested (1d, 2d_uniform, 2d) at comparable or smaller
memory stays under 8/100. The gains hold on AIME
reasoning benchmarks (+6 to +27 pts over the strongest per-layer-quantization baseline across eight configurations). Two null results – MATH-500 and LongBench's TREC – share a principled cause (short inputs, solver picks keep=1.0 on most layers), cleanly characterizing when per-layer eviction routing has headroom to help.
Read the full abstract ▾
All authors Libo Sun, Peixiong He, Po-Wei Harn, Xiao Qin