Computation and Language
Do Language Models Encode Semantic Relations? Probing and Sparse Feature Analysis
Andor Diera and Ansgar Scherp
Understanding whether large
language models (LLMs) capture structured meaning requires examining how they represent concept relationships. In this work, we study three models of increasing scale: Pythia-70M, GPT-2, and Llama 3.1 8B, focusing on four semantic relations: synonymy, antonymy, hypernymy, and hyponymy. We combine linear probing with mechanistic interpretability techniques, including sparse autoencoders (SAE) and activation patching, to identify where these relations are encoded and how specific features contribute to their representation. Our results reveal a directional asymmetry in hierarchical relations: hypernymy is encoded redundantly and resists suppression, while hyponymy relies on compact features that are more easily disrupted by ablation. More broadly, relation signals are diffuse but exhibit stable profiles: they peak in the mid-layers and are stronger in post-residual/MLP pathways than in
attention. Difficulty is consistent across models (antonymy easiest, synonymy hardest). Probe-level causality is capacity-dependent: on Llama 3.1, SAE-guided patching reliably shifts these signals, whereas on smaller models the shifts are weak or unstable. Our results clarify where and how reliably semantic relations are represented inside LLMs, and provide a reproducible framework for relating sparse features to probe-level causal evidence.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.LG
Complementary RL: Towards Efficient Experience-Driven Agent Learning
Dilxat Muhtar, Jiashun Liu, Wei Gao et al. (+8 more)
Reinforcement Learning (RL) has emerged as a powerful paradigm for training LLM-based
agents, yet remains limited by low sample efficiency, stemming not only from sparse outcome feedback but also from the
agent's inability to leverage prior experience across episodes. While augmenting
agents with historical experience offers a promising remedy, existing approaches suffer from a critical weakness: the experience distilled from history is either stored statically or fail to coevolve with the improving actor, causing a progressive misalignment between the experience and the actor's evolving capability that diminishes its utility over the course of training. Inspired by complementary learning systems in neuroscience, we present Complementary RL to achieve seamless co-
evolution of an experience extractor and a policy actor within the RL optimization loop. Specifically, the actor is optimized via sparse outcome-based rewards, while the experience extractor is optimized according to whether its distilled experiences demonstrably contribute to the actor's success, thereby evolving its experience management strategy in lockstep with the actor's growing capabilities. Empirically, Complementary RL outperforms outcome-based agentic RL baselines that do not learn from experience, achieving 10% performance improvement in single-task scenarios and exhibits robust scalability in multi-task settings. These results establish Complementary RL as a paradigm for efficient experience-driven
agent learning.
Read the full abstract ▾
All authors Dilxat Muhtar, Jiashun Liu, Wei Gao, Weixun Wang, Shaopan Xiong, Ju Huang, Siran Yang, Wenbo Su, Jiamang Wang, Ling Pan, Bo Zheng
Computation and Language · Cross-listed from cs.SI
Temporal Narrative Monitoring in Dynamic Information Environments
David Farr, Stephen Prochaska, Jack Moody et al. (+4 more)
Comprehending the information environment (IE) during crisis events is challenging due to the rapid change and abstract nature of the domain. Many approaches focus on snapshots via classification methods or network approaches to describe the IE in crisis, ignoring the temporal nature of how information changed over time. This work presents a system-oriented framework for modeling emerging narratives as temporally evolving semantic structures without requiring prior label specification. By integrating semantic
embeddings, density-based clustering, and rolling temporal linkage, the framework represents narratives as persistent yet adaptive entities within a shared semantic space. We apply the methodology to a real-world crisis event and evaluate system behavior through stratified cluster validation and temporal lifecycle analysis. Results demonstrate high cluster coherence and reveal heterogeneous narrative lifecycles characterized by both transient fragments and stable narrative anchors. We ground our approach in situational awareness theory, supporting perception and comprehension of the IE by transforming unstructured social media streams into interpretable, temporally structured representations. The resulting system provides a methodology for monitoring and decision support in dynamic information environments.
Read the full abstract ▾
All authors David Farr, Stephen Prochaska, Jack Moody, Lynnette Hui Xian Ng, Iain Cruickshank, Kate Starbird, Jevin West
Computation and Language
VeriAgent: A Tool-Integrated Multi-Agent System with Evolving Memory for PPA-Aware RTL Code Generation
Yaoxiang Wang, Qi Shi, ShangZhan Li et al. (+6 more)
LLMs have recently demonstrated strong capabilities in automatic RTL code generation, achieving high syntactic and functional correctness. However, most methods focus on functional correctness while overlooking critical physical design objectives, including Power, Performance, and Area. In this work, we propose a PPA-aware, tool-integrated multi-
agent framework for high-quality verilog code generation. Our framework explicitly incorporates EDA tools into a closed-loop workflow composed of a Programmer
Agent, a Correctness
Agent, and a PPA
Agent, enabling joint optimization of functional correctness and physical metrics. To support continuous improvement without model retraining, we introduce an Evolved
Memory Mechanism that externalizes optimization experience into structured
memory nodes. A dedicated
memory manager dynamically maintains the
memory pool and allows the system to refine strategies based on historical execution trajectories. Extensive experiments demonstrate that our approach achieves strong functional correctness while delivering significant improvements in PPA metrics. By integrating tool-driven feedback with structured and evolvable
memory, our framework transforms RTL generation from one-shot
reasoning into a continual, feedback-driven optimization process, providing a scalable pathway for deploying LLMs in real-world hardware design flows.
Read the full abstract ▾
All authors Yaoxiang Wang, Qi Shi, ShangZhan Li, Qingguo Hu, Xinyu Yin, Bo Guo, Xu Han, Maosong Sun, Jinsong Su
Computation and Language · Cross-listed from physics.soc-ph
Modeling Changing Scientific Concepts with Complex Networks: A Case Study on the Chemical Revolution
Sofía Aguilar-Valdez and Stefania Degaetano-Ortlieb
While context
embeddings produced by LLMs can be used to estimate conceptual change, these representations are often not interpretable nor time-aware. Moreover, bias augmentation in historical data poses a non-trivial risk to researchers in the Digital Humanities. Hence, to model reliable concept trajectories in evolving scholarship, in this work we develop a framework that represents prototypical concepts through complex networks based on topics. Utilizing the Royal Society Corpus, we analyzed two competing theories from the Chemical Revolution (phlogiston vs. oxygen) as a case study to show that onomasiological change is linked to higher
entropy and topological density, indicating increased diversity of ideas and connectivity effort.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.IR
From Isolated Scoring to Collaborative Ranking: A Comparison-Native Framework for LLM-Based Paper Evaluation
Pujun Zheng, Jiacheng Yao, Jinquan Zheng et al. (+6 more)
Large
language models (LLMs) are currently applied to scientific paper evaluation by assigning an absolute score to each paper independently. However, since score scales vary across conferences, time periods, and evaluation criteria, models trained on absolute scores are prone to fitting narrow, context-specific rules rather than developing robust scholarly judgment. To overcome this limitation, we propose shifting paper evaluation from isolated scoring to collaborative ranking. In particular, we design a $\textbf{C}$omparison-$\textbf{N}$ative framework for $\textbf{P}$aper $\textbf{E}$valuation ($\textbf{CNPE}$), integrating comparison into both data construction and model learning. We first propose a graph-based similarity ranking
algorithm to facilitate the sampling of more informative and discriminative paper pairs from a collection. We then enhance relative quality judgment through supervised fine-tuning and reinforcement learning with comparison-based rewards. At inference, the model performs pairwise comparisons over sampled paper pairs and aggregates these preference signals into a global relative quality ranking. Experimental results demonstrate that our framework achieves an average relative improvement of 21.8% over the strong baseline DeepReview-14B, while exhibiting robust generalization to five previously unseen
datasets. Our code is available at https://github.com/ECNU-Text-Computing/ComparisonReview.
Read the full abstract ▾
All authors Pujun Zheng, Jiacheng Yao, Jinquan Zheng, Chenyang Gu, Guoxiu He, Jiawei Liu, Yong Huang, Tianrui Guo, Wei Lu
Computation and Language
KA2L: A Knowledge-Aware Active Learning Framework for LLMs
Haoxuan Yin, Chen Tang, Yangfan Wang et al. (+2 more)
Fine-tuning large
language models (LLMs) with high-quality knowledge has been shown to enhance their performance effectively. However, there is a paucity of research on the depth of domain-specific knowledge comprehension by LLMs and the application of targeted active learning to improve their expertise. To address this gap, we introduce the Knowledge-Aware Active Learning (KA2L) framework. This framework assesses LLMs' mastery of specific knowledge points to aid in constructing unanswerable or unknowable questions through latent space analysis. This active learning strategy enhances training efficiency by focusing on knowledge the model has yet to master, thereby minimizing redundancy in learning already acquired information. This study innovatively employs a knowledge distribution probing technique to examine the hidden states of specific
Transformer layers and identify the distribution of known and unknown knowledge within the LLM. Additionally, a hidden-state decoding method is proposed to generate numerous unknown questions in natural
language from the latent knowledge space. In our experiments, we selected nine open-source LLMs to validate the effectiveness of the proposed framework. Results indicate that KA2L not only significantly reduces 50% annotation and computation costs across two open-domain and one vertical-domain
dataset but also achieves better performance, offering valuable insights into active learning strategies for LLMs. The code is available at https://github.com/greenjerry/KA2L.
Read the full abstract ▾
All authors Haoxuan Yin, Chen Tang, Yangfan Wang, Lian Yan, Jingchi Jiang
Computation and Language
Zipper-LoRA: Dynamic Parameter Decoupling for Speech-LLM based Multilingual Speech Recognition
Yuxiang Mei, Delai Qiu, Shengping Liu et al. (+2 more)
Speech Large
Language Models (
Speech-LLMs) have emerged as a powerful approach for automatic
speech recognition (ASR) by aligning
speech encoders with large
language models. However, adapting these systems to
multilingual settings with imbalanced data distributions remains challenging. In such scenarios, a stability-plasticity dilemma often arises: fully shared Parameter-Efficient Fine-Tuning (PEFT) can cause negative inter-lingual interference for under-represented
languages, while fully
language-specific tuning limits the cross-lingual beneficial knowledge transfer needed for low-resource tasks. To address this, we propose Zipper-LoRA, a novel rank-level decoupling framework with three variants (Static, Hard, and Soft) that dynamically synthesizes LoRA updates from shared and
language-specific subspaces. By using a lightweight
language-conditioned router, Zipper-LoRA dynamically controls the contribution of each subspace at the LoRA rank level, enabling fine-grained sharing where
languages are compatible and strict decoupling when conflicts occur. To further stabilize optimization under imbalanced data, we propose a two-stage training strategy with an Initial-B warm start that significantly accelerates convergence. Experiments on a 12-
language mixed-resource setting show that Zipper-LoRA consistently outperforms both fully shared and independent baselines, particularly in extremely low-resource scenarios. Moreover, we demonstrate that these gains are robust across both chunked and non-chunked encoder configurations, confirming the framework's reliability for practical, large-scale
multilingual ASR. Our code and data will be available at https://github.com/YuCeong-May/Zipper-LoRA for reproducibility.
Read the full abstract ▾
All authors Yuxiang Mei, Delai Qiu, Shengping Liu, Jiaen Liang, Yanhua Long
Computation and Language
AURORA Model of Formant-to-Tongue Inversion for Didactic and Clinical Applications
Patrycja Strycharczuk and Sam Kirkham
This paper outlines the conceptual and computational foundations of the AURORA (Acoustic Understanding and Real-time Observation of Resonant Articulations) model. AURORA predicts tongue displacement and shape in vowel sounds based on the first two formant values. It is intended as a didactic aid helping to explain the relationship between formants and the underlying articulation, as well as a foundation for biofeedback applications. The model is informed by ultrasound tongue imaging and acoustic data from 40 native speakers of English. In this paper we discuss the motivation for the model, the modelling objectives as well as the model architecture. We provide a qualitative evaluation of the model, focusing on selected tongue features. We then present two tools developed to make the model more accessible to a wider audience, a Shiny app and a prototype software for real-time tongue biofeedback. Potential users include students of phonetics, linguists in fields adjacent to phonetics, as well as
speech and
language therapy practitioners and clients.
Read the full abstract ▾
Computation and Language
Detecting the Machine: A Comprehensive Benchmark of AI-Generated Text Detectors Across Architectures, Domains, and Adversarial Conditions
Madhav S. Baidya, S. S. Baidya and Chirag Chawla
The rapid proliferation of large
language models (LLMs) has created an urgent need for robust and generalizable detectors of machine-generated text. Existing
benchmarks typically evaluate a single detector on a single
dataset under ideal conditions, leaving open questions about cross-domain transfer, cross-LLM generalization, and
adversarial robustness. We present a comprehensive
benchmark evaluating diverse detection approaches across two corpora: HC3 (23,363 human-ChatGPT pairs) and ELI5 (15,000 human-Mistral-7B pairs). Methods include classical classifiers, fine-tuned
transformer encoders (BERT, RoBERTa, ELECTRA, DistilBERT, DeBERTa-v3), a CNN, an XGBoost stylometric model, perplexity-based detectors, and LLM-as-detector prompting. Results show that
transformer models achieve near-perfect in-distribution performance but degrade under domain shift. The XGBoost stylometric model matches performance while remaining interpretable. LLM-based detectors underperform and are affected by generator-detector identity bias. Perplexity-based methods exhibit polarity inversion, with modern LLM outputs showing lower perplexity than human text, but remain effective when corrected. No method generalizes robustly across domains and LLM sources.
Read the full abstract ▾
Computation and Language
Language on Demand, Knowledge at Core: Composing LLMs with Encoder-Decoder Translation Models for Extensible Multilinguality
Mengyu Bu and Yang Feng
Large
language models (LLMs) exhibit strong general intelligence, yet their
multilingual performance remains highly imbalanced. Although LLMs encode substantial cross-lingual knowledge in a unified semantic space, they often struggle to reliably interface this knowledge with low-resource or unseen
languages. Fortunately, pretrained encoder-decoder translation models already possess balanced
multilingual capability, suggesting a natural complement to LLMs. In this work, we propose XBridge, a compositional encoder-LLM-decoder architecture that offloads
multilingual understanding and generation to external pretrained translation models, while preserving the LLM as an English-centric core for general knowledge processing. To address the resulting representation misalignment across models, we introduce lightweight cross-model mapping layers and an optimal transport-based
alignment objective, enabling fine-grained semantic consistency for
multilingual generation. Experiments on four LLMs across
multilingual understanding,
reasoning, summarization, and generation indicate that XBridge outperforms strong baselines, especially on low-resource and previously unseen
languages, without retraining the LLM.
Read the full abstract ▾
Computation and Language
Inducing Epistemological Humility in Large Language Models: A Targeted SFT Approach to Reducing Hallucination
Cem Uluoglakci and Tugba Taskaya Temizel
Large
language models (LLMs) often hallucinate, producing fluent but false information, partly because supervised fine-tuning (SFT) implicitly rewards always responding. We introduce $\textit{HypoTermInstruct}$, an SFT
dataset (31,487 responses for 11,151 questions) designed to teach models epistemological humility-the ability to recognize the limits of their own knowledge and admit uncertainty. This is achieved through questions about non-existent "hypothetical" terms. We also release $\textit{HypoTermQA-Enhanced}$, a
benchmark for
hallucination tendency strengthened through multiple validations. We conducted 800 controlled LoRA SFT runs across $\textit{Llama3.1-8B}$ and $\textit{Gemma3-4B}$ (base and instruct), testing 100 fine-tuning configurations with paired controls. Our results demonstrate that replacing generic instruction data with $\textit{HypoTermInstruct}$ significantly improves the HypoTerm Score (median increases of 0.19% to 25.91%) and FactScore (+0.39% to +0.86%), while maintaining stable performance on MMLU (minimal decreases of 0.26% to 0.35%). Our work demonstrates that targeted, high-quality SFT data teaching meta-cognitive skills can effectively reduce
hallucination without preference/RL pipelines, providing mechanistic insights and a practical path toward more reliable AI systems.
Read the full abstract ▾
Computation and Language
Learning When to Attend: Conditional Memory Access for Long-Context LLMs
Sakshi Choudhary, Aditya Chattopadhyay, Luca Zancato et al. (+4 more)
Language models struggle to generalize beyond pretraining context lengths, limiting long-horizon
reasoning and
retrieval. Continued pretraining on long-context data can help but is expensive due to the quadratic scaling of
Attention. We observe that most
tokens do not require (Global)
Attention over the entire sequence and can rely on local context. Based on this, we propose L2A (Learning To Attend), a layer that enables conditional (
token-wise) long-range
memory access by deciding when to invoke global
attention. We evaluate L2A on Qwen 2.5 and Qwen 3 models, extending their effective context length from 32K to 128K
tokens. L2A matches the performance of standard long-context training to within 3% while skipping Global
Attention for $\sim$80% of
tokens, outperforming prior baselines. We also design custom Triton kernels to efficiently implement this
token-wise conditional
Attention on GPUs, achieving up to $\sim$2$\times$ improvements in training throughput and time-to-first-
token over FlashAttention. Moreover, L2A enables post-training
pruning of highly sparse Global
Attention layers, reducing KV cache
memory by up to 50% with negligible performance loss. Our code is released under Apache 2.0 at https://github.com/awslabs/hybrid-model-factory/tree/main/examples/research/L2A.
Read the full abstract ▾
All authors Sakshi Choudhary, Aditya Chattopadhyay, Luca Zancato, Elvis Nunez, Matthew Trager, Wei Xia, Stefano Soatto
Computation and Language · Cross-listed from cs.CV
UniSAFE: A Comprehensive Benchmark for Safety Evaluation of Unified Multimodal Models
Segyu Lee, Boryeong Cho, Hojung Jung et al. (+8 more)
Unified
Multimodal Models (UMMs) offer powerful cross-modality capabilities but introduce new safety risks not observed in single-task models. Despite their emergence, existing safety
benchmarks remain fragmented across tasks and modalities, limiting the comprehensive evaluation of complex system-level vulnerabilities. To address this gap, we introduce UniSAFE, the first comprehensive
benchmark for system-level safety evaluation of UMMs across 7 I/O modality combinations, spanning conventional tasks and novel
multimodal-context image generation settings. UniSAFE is built with a shared-target design that projects common risk scenarios across task-specific I/O configurations, enabling controlled cross-task comparisons of safety failures. Comprising 6,802 curated instances, we use UniSAFE to evaluate 15 state-of-the-art UMMs, both proprietary and open-source. Our results reveal critical vulnerabilities across current UMMs, including elevated safety violations in multi-image composition and multi-turn settings, with image-output tasks consistently more vulnerable than text-output tasks. These findings highlight the need for stronger system-level safety
alignment for UMMs. Our code and data are publicly available at https://github.com/segyulee/UniSAFE
Read the full abstract ▾
All authors Segyu Lee, Boryeong Cho, Hojung Jung, Seokhyun An, Juhyeong Kim, Jaehyun Kwak, Yongjin Yang, Sangwon Jang, Youngrok Park, Wonjun Chang, Se-Young Yun
Computation and Language
Humans and transformer LMs: Abstraction drives language learning
Jasper Jian and Christopher D. Manning
Categorization is a core component of human linguistic competence. We investigate how a
transformer-based
language model (LM) learns linguistic categories by comparing its behaviour over the course of training to behaviours which characterize abstract feature-based and concrete exemplar-based accounts of human
language acquisition. We investigate how lexical semantic and syntactic categories emerge using novel divergence-based metrics that track learning trajectories using next-
token distributions. In experiments with GPT-2 small, we find that (i) when a construction is learned, abstract class-level behaviour is evident at earlier steps than lexical item-specific behaviour, and (ii) that different linguistic behaviours emerge abruptly in sequence at different points in training, revealing that abstraction plays a key role in how LMs learn. This result informs the models of human
language acquisition that LMs may serve as an existence proof for.
Read the full abstract ▾
Computation and Language
TRiMS: Real-Time Tracking of Minimal Sufficient Length for Efficient Reasoning via RL
Tingcheng Bian, Jinchang Luo, Mingquan Cheng et al. (+5 more)
Large
language models achieve breakthroughs in complex
reasoning via long chain-of-thought sequences. However, this often leads to severe
reasoning inflation, causing substantial computational redundancy. To maximize Intelligence per
Token, we introduce a theoretical metric, MSL-Minimal Sufficient Length. MSL rigorously characterizes the shortest
reasoning length that preserves answer correctness. We provide a recursive definition based on independently sampled sequences and prove the existence of its limit, establishing the first measurable lower bound for
reasoning-chain compression. Building on an analysis of mainstream CoT compression strategies, we identify key structural factors enabling a model to approach MSL. Based on these insights, we propose TRiMS which employs the GRPO
algorithm in conjunction with MSL-based estimation during training, while mitigating instabilities during the training process through dynamic batch aggregation and advantage computation using batch-level standard deviation. TRiMS achieves over 80% CoT
token reduction with a minor accuracy boost across all
benchmarks.
Read the full abstract ▾
All authors Tingcheng Bian, Jinchang Luo, Mingquan Cheng, Jinyu Zhang, Xiaoling Xia, Ni Li, Yan Tao, Haiwei Wang
Computation and Language · Cross-listed from cs.AI
When Only the Final Text Survives: Implicit Execution Tracing for Multi-Agent Auditing
Yi Nian, Haosen Cao, Shenzhe Zhu et al. (+4 more)
When a multi-
agent system produces an incorrect or harmful answer, who is accountable if execution logs and
agent identifiers are unavailable? In practice, generated content is often detached from its execution environment due to
privacy or system boundaries, leaving the final text as the only auditable artifact. Existing attribution methods rely on full execution traces and thus become ineffective in such metadata-deprived settings. We propose Implicit Execution Tracing (IET), a provenance-by-design framework that shifts attribution from post-hoc inference to built-in instrumentation. Rather than inferring provenance after the fact, IET embeds
agent-specific, key-conditioned statistical signals into the
token generation process at generation time, turning the output text into a self-verifying provenance record. An offline auditor, holding a verification registry that maps each
agent to its key, then recovers segment-level provenance - segment boundaries and per-segment
agent attribution - from the final text alone without access to execution logs or private traces. Experiments across diverse multi-
agent coordination settings demonstrate that IET achieves accurate segment-level attribution and reliable transition recovery under identity removal, boundary corruption, and
privacy-preserving redaction, while maintaining generation quality. These results show that
embedding provenance into generation provides a practical foundation for accountability in multi-
agent language systems under metadata loss.
Read the full abstract ▾
All authors Yi Nian, Haosen Cao, Shenzhe Zhu, Henry Peng Zou, Qingqing Luan, Yudi Zhang, Yue Zhao
Computation and Language
Argument Reconstruction as Supervision for Critical Thinking in LLMs
Hyun Ryu, Gyouk Chu, Gregor Betz et al. (+3 more)
To think critically about arguments, human learners are trained to identify, reconstruct, and evaluate arguments. Argument reconstruction is especially important because it makes an argument's underlying inferences explicit. However, it remains unclear whether LLMs can similarly enhance their critical thinking ability by learning to reconstruct arguments. To address this question, we introduce a holistic framework with three contributions. We (1) propose an engine that automatically reconstructs arbitrary arguments (GAAR), (2) synthesize a new high-quality argument reconstruction
dataset (Arguinas) using the GAAR engine, and (3) investigate whether learning argument reconstruction benefits downstream critical thinking tasks. Our experimental results show that, across seven critical thinking tasks, models trained to learn argument reconstruction outperform models that do not, with the largest performance gains observed when training on the proposed Arguinas
dataset.
Read the full abstract ▾
All authors Hyun Ryu, Gyouk Chu, Gregor Betz, Eunho Yang, Carolyn Rose, Sean Welleck
Computation and Language · Cross-listed from cs.IR
Guangzhi Wang, Xiaohui Yang, Kai Li et al. (+4 more)
As
retrieval models converge on generic
benchmarks, the pressing question is no longer "who scores higher" but rather "where do systems fail, and why?" Person-job matching is a domain that urgently demands such diagnostic capability – it requires systems not only to verify explicit constraints but also to perform skill-transfer inference and job-competency
reasoning, yet existing
benchmarks provide no systematic diagnostic support for this task. We introduce PJB (Person-Job
Benchmark), a
reasoning-aware
retrieval evaluation
dataset that uses complete job descriptions as queries and complete resumes as documents, defines relevance through job-competency judgment, is grounded in real-world recruitment data spanning six industry domains and nearly 200,000 resumes, and upgrades evaluation from "who scores higher" to "where do systems differ, and why" through domain-family and
reasoning-type diagnostic labels. Diagnostic experiments using dense
retrieval reveal that performance heterogeneity across industry domains far exceeds the gains from module upgrades for the same model, indicating that aggregate scores alone can severely mislead optimization decisions. At the module level, reranking yields stable improvements while query understanding not only fails to help but actually degrades overall performance when combined with reranking – the two modules face fundamentally different improvement bottlenecks. The value of PJB lies not in yet another leaderboard of average scores, but in providing recruitment
retrieval systems with a capability map that pinpoints where to invest.
Read the full abstract ▾
All authors Guangzhi Wang, Xiaohui Yang, Kai Li, Jiawen He, Kai Yang, Ruixuan Zhang, Zhi Liu
Computation and Language
SafeTutors: Benchmarking Pedagogical Safety in AI Tutoring Systems
Rima Hazra, Bikram Ghuku, Ilona Marchenko et al. (+5 more)
Large
language models are rapidly being deployed as AI tutors, yet current evaluation paradigms assess problem-solving accuracy and generic safety in isolation, failing to capture whether a model is simultaneously pedagogically effective and safe across student-tutor interaction. We argue that tutoring safety is fundamentally different from conventional LLM safety: the primary risk is not toxic content but the quiet erosion of learning through answer over-disclosure, misconception reinforcement, and the abdication of scaffolding. To systematically study this failure mode, we introduce SafeTutors, a
benchmark that jointly evaluates safety and pedagogy across mathematics, physics, and chemistry. SafeTutors is organized around a theoretically grounded risk taxonomy comprising 11 harm dimensions and 48 sub-risks drawn from learning-science literature. We uncover that all models show broad harm; scale doesn't reliably help; and multi-turn dialogue worsens behavior, with pedagogical failures rising from 17.7% to 77.8%. Harms also vary by subject, so mitigations must be discipline-aware, and single-turn "safe/helpful" results can mask systematic tutor failure over extended interaction.
Read the full abstract ▾
All authors Rima Hazra, Bikram Ghuku, Ilona Marchenko, Yaroslava Tokarieva, Sayan Layek, Somnath Banerjee, Julia Stoyanovich, Mykola Pechenizkiy
Computation and Language · Cross-listed from cs.IR
Public Profile Matters: A Scalable Integrated Approach to Recommend Citations in the Wild
Karan Goyal, Dikshant Kukreja, Vikram Goyal et al. (+1 more)
Proper citation of relevant literature is essential for contextualising and validating scientific contributions. While current citation recommendation systems leverage local and global textual information, they often overlook the nuances of the human citation behaviour. Recent methods that incorporate such patterns improve performance but incur high computational costs and introduce systematic biases into downstream rerankers. To address this, we propose Profiler, a lightweight, non-learnable module that captures human citation patterns efficiently and without bias, significantly enhancing candidate
retrieval. Furthermore, we identify a critical limitation in current evaluation protocol: the systems are assessed in a transductive setting, which fails to reflect real-world scenarios. We introduce a rigorous Inductive evaluation setting that enforces strict temporal constraints, simulating the recommendation of citations for newly authored papers in the wild. Finally, we present DAVINCI, a novel reranking model that integrates profiler-derived confidence priors with semantic information via an adaptive vector-gating mechanism. Our system achieves new state-of-the-art results across multiple
benchmark datasets, demonstrating superior efficiency and generalisability.
Read the full abstract ▾
All authors Karan Goyal, Dikshant Kukreja, Vikram Goyal, Mukesh Mohania
Computation and Language
PACE-RAG: Patient-Aware Contextual and Evidence-Constrained RAG for Clinical Drug Recommendation
Chaeyoung Huh, Hyunmin Hwang, Jung Hwan Shin et al. (+3 more)
Drug recommendation requires a deep understanding of individual patient context, especially for complex conditions like Parkinson's disease. While LLMs possess broad medical knowledge, they fail to capture the subtle nuances of actual prescribing patterns. Existing RAG methods also struggle with these complexities because guideline-based
retrieval remains too generic and similar-patient
retrieval often replicates majority patterns without accounting for the unique clinical nuances of individual patients. To bridge this gap, we propose PACE-RAG (Patient-Aware Contextual and Evidence-Constrained RAG). Rather than directly copying frequent medications from retrieved patients, PACE-RAG personalizes recommendations by first extracting patient-specific clinical features, retrieving cases around these features, and then refining the final prescription using the patient's current symptoms, active medication history, and focus-specific prescribing tendencies. By analyzing treatment patterns tailored to specific clinical features, PACE-RAG generates patient-specific medication recommendations along with an explainable clinical summary. PACE-RAG achieved the strongest performance among the evaluated inference-only LLM-based methods, reaching F1 scores of 80.84% and 47.22% on the Parkinson's disease and MIMIC-IV cohorts, respectively. Our code is available at: https://github.com/ChaeYoungHuh/PACE-RAG.
Read the full abstract ▾
All authors Chaeyoung Huh, Hyunmin Hwang, Jung Hwan Shin, Sungyang Jo, Jinse Park, Jong Chul Ye
Computation and Language · Cross-listed from cs.LG
Beyond Outliers: A Data-Free Layer-wise Mixed-Precision Quantization Approach Driven by Numerical and Structural Dual-Sensitivity
Hengyuan Zhang, Xinrong Chen, Zunhai Su et al. (+10 more)
Layer-wise mixed-precision quantization (LMPQ) enables effective compression under extreme low-bit settings by allocating higher precision to sensitive layers. However, existing methods typically treat all intra-layer weight modules uniformly and rely on a single numerical property when estimating sensitivity, overlooking their distinct operational roles and structural characteristics. To address this, we propose NSDS, a novel calibration-free LMPQ framework driven by Numerical and Structural Dual-Sensitivity. Specifically, it first mechanistically decomposes each layer into distinct operational roles and quantifies their sensitivity from both numerical and structural perspectives. These dual-aspect scores are then aggregated into a unified layer-wise metric through a robust aggregation scheme based on MAD-Sigmoid and Soft-OR to guide bit allocation. Extensive experiments demonstrate that NSDS consistently achieves superior performance compared to various baselines across diverse models and downstream tasks, without relying on any calibration data.
Read the full abstract ▾
All authors Hengyuan Zhang, Xinrong Chen, Zunhai Su, Xiao Liang, Jing Xiong, Wendong Xu, He Xiao, Chaofan Tao, Wei Zhang, Ruobing Xie, Lei Jiang, Hayden Kwok-Hay So, Ngai Wong
Computation and Language
Grid Spatial Understanding: A Dataset for Textual Spatial Reasoning over Grids, Embodied Settings, and Coordinate Structures
Risham Sidhu and Julia Hockenmaier
We introduce GSU, a text-only grid
dataset to evaluate the spatial
reasoning capabilities of LLMs over 3 core tasks:
navigation, object localization, and structure composition. By forgoing visual inputs, isolating spatial
reasoning from perception, we show that while most models grasp basic grid concepts, they struggle with frames of reference relative to an
embodied agent and identifying 3D shapes from coordinate lists. We also find that exposure to a visual modality does not provide a generalizable understanding of 3D space that VLMs are able to utilize for these tasks. Finally, we show that while the very latest frontier models can solve the provided tasks (though harder variants may still stump them), fully fine-tuning a small LM or LORA fine-tuning a small LLM show potential to match frontier model performance, suggesting an avenue for specialized
embodied agents.
Read the full abstract ▾
Computation and Language
Ruyi2.5 Technical Report
Huan Song, Shuyu Tian, Qingfei Zhao et al. (+5 more)
We present Ruyi2.5, a
multimodal familial model built on the AI Flow framework. Extending Ruyi2's "Train Once, Deploy Many" paradigm to the
multimodal domain, Ruyi2.5 constructs a shared-backbone architecture that co-trains models of varying scales within a single unified pipeline, ensuring semantic consistency across all deployment tiers. Built upon Ruyi2.5, Ruyi2.5-Camera model is developed as a
privacy-preserving camera service system, which instantiates Ruyi2.5-Camera into a two-stage recognition pipeline: an edge model applies information-bottleneck-guided irreversible feature mapping to de-identify raw frames at the source, while a cloud model performs deep behavior
reasoning. To accelerate reinforcement learning fine-tuning, we further propose Binary Prefix Policy Optimization (BPPO), which reduces sample redundancy via binary response selection and focuses gradient updates on response prefixes, achieving a 2 to 3 times training speedup over GRPO. Experiments show Ruyi2.5 matches Qwen3-VL on the general
multimodal benchmarks, while Ruyi2.5-Camera substantially outperforms Qwen3-VL on
privacy-constrained surveillance tasks.
Read the full abstract ▾
All authors Huan Song, Shuyu Tian, Qingfei Zhao, Wenhao Hong, Jiang Liu, Ting Long, Jiawei Shao, Xuelong Li
Computation and Language · Cross-listed from cs.AI
InfoDensity: Rewarding Information-Dense Traces for Efficient Reasoning
Chengwei Wei, Jung-jae Kim, Longyin Zhang et al. (+2 more)
Large
Language Models (LLMs) with extended
reasoning capabilities often generate verbose and redundant
reasoning traces, incurring unnecessary computational cost. While existing reinforcement learning approaches address this by optimizing final response length, they neglect the quality of intermediate
reasoning steps, leaving models vulnerable to reward hacking. We argue that verbosity is not merely a length problem, but a symptom of poor intermediate
reasoning quality. To investigate this, we conduct an empirical study tracking the per-
token predictive
entropy of large
reasoning models across
reasoning trajectories. We find that high-quality
reasoning traces exhibit two consistent properties: low uncertainty convergence and fast uncertainty descent. These findings suggest that high-quality
reasoning traces are informationally dense, that is,
reasoning steps contribute to reaching a low uncertainty level relative to the total
reasoning length. Motivated by this, we propose InfoDensity, a reward framework for RL training that captures both properties through a single suffix-max envelope of the
entropy trajectory, weighted by a length scaling term that favors achieving equivalent quality more concisely. Experiments on mathematical and general
reasoning benchmarks demonstrate that InfoDensity outperforms state-of-the-art baselines on the accuracy-efficiency trade-off.
Read the full abstract ▾
All authors Chengwei Wei, Jung-jae Kim, Longyin Zhang, Shengkai Chen, Nancy F. Chen
Computation and Language
Evidence for systematic semantic structure in individual letters
Gexin Zhao
Associations between
speech sounds and meaning are well documented but have not been systematically mapped over a whole alphabet. Here we map them across the 26 English letters and find that each carries a structured, multidimensional semantic profile that is recoverable from text, perceived across
languages, and predicted by articulatory features. Three large
language models independently detected consistent semantic structure across nine perceptual dimensions in 220 pairwise letter contrasts, and the profiles they recovered were then tested in preregistered experiments with 1,388 human participants. Native English readers chose the predicted word above chance (85.3%, selected items; 65.7%, all contrasts), and the preference followed the letter rather than the words that carried it. Listeners of five typologically diverse
languages showed the same preference (76.7%, selected pairs; 68.4%, randomly drawn contrasts), regardless of their English proficiency. Articulatory features assigned to each letter predicted both the model profiles and the human judgments. Letter-meaning association is thus a systematic, multidimensional property of the alphabet rather than a set of isolated effects.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.AI
Contrastive Reasoning Alignment: Reinforcement Learning from Hidden Representations
Haozheng Luo, Yimin Wang, Jiahao Yu et al. (+2 more)
We propose CRAFT, a red-teaming
alignment framework that leverages model
reasoning capabilities and hidden representations to improve robustness against jailbreak attacks. Unlike prior defenses that operate primarily at the output level, CRAFT aligns large
reasoning models to generate safety-aware
reasoning traces by explicitly optimizing objectives defined over the hidden state space. Methodologically, CRAFT integrates contrastive representation learning with reinforcement learning to separate safe and unsafe
reasoning trajectories, yielding a latent-space geometry that supports robust,
reasoning-level safety
alignment. Theoretically, we show that incorporating latent-textual consistency into GRPO eliminates superficially aligned policies by ruling them out as local optima. Empirically, we evaluate CRAFT on multiple safety
benchmarks using two strong
reasoning models, Qwen3-4B-Thinking and R1-Distill-Llama-8B, where it consistently outperforms state-of-the-art defenses such as IPO and SafeKey. Notably, CRAFT delivers an average 79.0% improvement in
reasoning safety and 87.7% improvement in final-response safety over the base models, demonstrating the effectiveness of hidden-space
reasoning alignment.
Read the full abstract ▾
All authors Haozheng Luo, Yimin Wang, Jiahao Yu, Binghui Wang, Yan Chen
Computation and Language
From Words to Worlds: Benchmarking Cross-Cultural Cultural Understanding in Machine Translation
Bangju Han, Yingqi Wang, Huang Qing et al. (+6 more)
Culture-expressions, such as idioms, slang, and culture-specific items (CSIs), are pervasive in natural
language and encode meanings that go beyond literal linguistic form. Accurately translating such expressions remains challenging for machine translation systems. Despite this, existing
benchmarks remain fragmented and do not provide a systematic framework for evaluating translation performance on culture-loaded expressions. To address this gap, we introduce CulT-Eval, a
benchmark designed to evaluate how models handle different types of culturally grounded expressions. CulT-Eval comprises over 7,959 carefully curated instances spanning multiple types of culturally grounded expressions, with a comprehensive error taxonomy covering culturally grounded expressions. Through extensive evaluation of large
language models and detailed analysis, we identify recurring and systematic failure modes that are not adequately captured by existing automatic metrics. Accordingly, we propose a complementary evaluation metric that targets culturally induced meaning deviations overlooked by standard MT metrics. The results indicate that current models struggle to preserve culturally grounded meaning and to capture the cultural and contextual nuances essential for accurate translation. Our
benchmark and code are available at https://anonymous.4open.science/r/CulT-Eval-E75D/.
Read the full abstract ▾
All authors Bangju Han, Yingqi Wang, Huang Qing, Tiyuan Li, Fengyi Yang, Ahtamjan Ahmat, Abibulla Atawulla, Yating Yang, Xi Zhou
Computation and Language · Cross-listed from cs.CV
LED: A Benchmark for Evaluating Layout Error Detection in Document Analysis
Inbum Heo, Taewook Hwang, Jeesu Jung et al. (+1 more)
Recent advances in Large
Language Models (LLMs) and Large
Multimodal Models (LMMs) have improved Document Layout Analysis (DLA), yet structural errors such as region merging, splitting, and omission remain persistent. Conventional overlap-based metrics (e.g., IoU, mAP) fail to capture such logical inconsistencies. To overcome this limitation, we propose Layout Error Detection (LED), a
benchmark that evaluates structural
reasoning in DLA predictions beyond surface-level accuracy. LED defines eight standardized error types (Missing,
Hallucination, Size Error, Split, Merge, Overlap, Duplicate, and Misclassification) and provides quantitative rules and injection
algorithms for realistic error simulation. Using these definitions, we construct LED-
Dataset and design three evaluation tasks: document-level error detection, document-level error-type classification, and element-level error-type classification. Experiments with state-of-the-art
multimodal models show that LED enables fine-grained and interpretable assessment of structural understanding, revealing clear weaknesses across modalities and architectures. Overall, LED establishes a unified and explainable
benchmark for diagnosing the structural robustness and
reasoning capability of document understanding models.
Read the full abstract ▾
All authors Inbum Heo, Taewook Hwang, Jeesu Jung, Sangkeun Jung