Computation and Language · Cross-listed from cs.MA
G2CP: A Graph-Grounded Communication Protocol for Verifiable and Efficient Multi-Agent Reasoning
Karim Ben Khaled and Davy Monticolo
Multi-
agent systems powered by Large
Language Models face a critical challenge:
agents communicate through natural
language, leading to semantic drift,
hallucination propagation, and inefficient
token consumption. We propose G2CP (Graph-Grounded Communication Protocol), a structured
agent communication
language where messages are graph operations rather than free text.
Agents exchange explicit traversal commands, subgraph fragments, and update operations over a shared knowledge graph, enabling verifiable
reasoning traces and eliminating ambiguity. We validate G2CP within an industrial knowledge management system where specialized
agents (Diagnostic, Procedural, Synthesis, and Ingestion) coordinate to answer complex queries. Experimental results on 500 industrial scenarios and 21 real-world maintenance cases show that G2CP reduces inter-
agent communication
tokens by 73%, improves task completion accuracy by 34% over free-text baselines, eliminates cascading
hallucinations, and produces fully auditable
reasoning chains. G2CP represents a fundamental shift from linguistic to structural communication in multi-
agent systems, with implications for any domain requiring precise
agent coordination. Code, data, and evaluation scripts are publicly available.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.AI
Nanbeige4.1-3B: A Small General Model that Reasons, Aligns, and Acts
Chen Yang, Guangyue Peng, Jiaying Zhu et al. (+12 more)
We present Nanbeige4.1-3B, a unified generalist
language model that simultaneously achieves strong agentic behavior, code generation, and general
reasoning with only 3B parameters. To the best of our knowledge, it is the first open-source small
language model (SLM) to achieve such versatility in a single model. To improve
reasoning and preference
alignment, we combine point-wise and pair-wise reward modeling, ensuring high-quality, human-aligned responses. For code generation, we design complexity-aware rewards in Reinforcement Learning, optimizing both correctness and efficiency. In deep search, we perform complex data synthesis and incorporate turn-level supervision during training. This enables stable long-horizon tool interactions, allowing Nanbeige4.1-3B to reliably execute up to 600 tool-call turns for complex problem-solving. Extensive experimental results show that Nanbeige4.1-3B significantly outperforms prior models of similar scale, such as Nanbeige4-3B-2511 and Qwen3-4B, even achieving superior performance compared to much larger models, such as Qwen3-30B-A3B. Our results demonstrate that small models can achieve both broad competence and strong specialization simultaneously, redefining the potential of 3B parameter models.
Read the full abstract ▾
All authors Chen Yang, Guangyue Peng, Jiaying Zhu, Ran Le, Ruixiang Feng, Tao Zhang, Xiyun Xu, Yang Song, Yiming Jia, Yuntao Wen, Yunzhi Xu, Zekai Wang, Zhenwei An, Zhicong Sun, Zongchao Chen
Computation and Language · Cross-listed from cs.CV
Using Deep Learning to Generate Semantically Correct Hindi Captions
Wasim Akram Khan and Anil Kumar Vuppala
Automated image captioning using the content from the image is very appealing when done by harnessing the capability of computer
vision and natural
language processing. Extensive research has been done in the field with a major focus on the English
language which gives the scope for further developments in the same with consideration of popular foreign
languages. This research utilizes distinct models for translating the image caption into Hindi, the fourth most popular
language across the world. Exploring the multi-modal architectures this research comprises local visual features, global visual features,
attention mechanisms, and pre-trained models. Using google cloud translator on the image
dataset from Flickr8k, Hindi image descriptions have been generated. Pre-trained CNNs like VGG16, ResNet50, and Inception V3 helped in retrieving image characteristics, while the uni-directional and bi-directional techniques of text encoding are used for the text encoding process. An additional
Attention layer helps to generate a weight vector and, by multiplying it, combine image characteristics from each time step into a sentence-level feature vector. Bilingual evaluation understudy scores are used to compare the research outcome. Many experiments that serve as a baseline are done for the comparative analysis of the research. An image with a score of BLEU-1 is considered sufficient, whereas one with a score of BLEU-4 is considered to have fluid image captioning. For both BLEU scores, the
attention-based bidirectional LSTM with VGG16 produced the best results of 0.59 and 0.19 respectively. The experiments conclude that researchs ability to produce relevant, semantically accurate image captions in Hindi. The research accomplishes the goals and future research can be guided by this research model.
Read the full abstract ▾
Computation and Language
Semantic Chunking and the Entropy of Natural Language
Weishun Zhong, Doron Sivan, Tankut Can et al. (+2 more)
Humans and large
language models can predict next letter or word from its prior context much better than random guessing, indicating strong redundancy of
language viewed as a
stochastic process. Quantitatively this redundancy was estimated by Shannon to be around 80%, which means that every letter of a printed English text conveys approximately 1 bit of information and not 4.8 bits that 27 letters (including spaces) could potentially carry. This estimate was later confirmed by using autoregressive
token probabilies computed by large
language models. However, the statistical organization of
language that give rise to such a large redundancy remains unclear. Here we introduce a statistical framework of
language linking its redundancy to the hierarchical semantic organization of text. To this end, we use large
language models to recursively segment any given text into semantically coherent chunks, inducing a “semantic tree” that spans the whole range of text organization, beginning from its main idea to individual
tokens (words). For a large corpus of texts of a particular type, say fiction stories, the resulting ensemble of semantic trees is characterized by specific statistical regularities, giving rise to a “structural”
entropy rate defined in this study. Surprisingly, we discovered that for several
datasets considered in this work, semantic tree
entropy rate was quite close to LLM-measured quantity and exhibited a similar trend across corpus. In particular, simpler texts like children stories exhibit lower branching in their semantic trees and correspondingly lower
entropy rates, whereas fiction and poetry exhibit progressively larger branching factors and greater
entropy rates. These results suggest that hierarchical semantic organization of texts is an important factor in their overall information transmission rates.
Read the full abstract ▾
All authors Weishun Zhong, Doron Sivan, Tankut Can, Mikhail Katkov, Misha Tsodyks
Computation and Language · Cross-listed from cs.CV
CoPE-VideoLM: Leveraging Codec Primitives For Efficient Video Language Modeling
Sayan Deb Sarkar, Rémi Pautrat, Ondrej Miksik et al. (+4 more)
Video Language Models (VideoLMs) enable AI systems to understand temporal dynamics in
videos. To fit within the maximum context window constraint, current methods use keyframe sampling which often misses both macro-level events and micro-level details due to the sparse temporal coverage. Furthermore, processing full images and their
tokens for each frame incurs substantial computational overhead. We address these limitations by leveraging
video codec primitives (specifically motion vectors and residuals) which natively encode
video redundancy and sparsity without requiring expensive full-image encoding for most frames. To this end, we introduce lightweight
transformer-based encoders that aggregate codec primitives and align their representations with image encoder
embeddings through a pre-training strategy that accelerates convergence during end-to-end fine-tuning. Our approach, CoPE-VideoLM, reduces the time-to-first-
token by up to 86% and
token usage by up to 93% compared to standard VideoLMs. Moreover, by varying the keyframe and codec primitive densities we maintain or exceed performance on 14 diverse
video understanding
benchmarks spanning general question answering, temporal and motion
reasoning, long-form understanding, and spatial scene understanding.
Read the full abstract ▾
All authors Sayan Deb Sarkar, Rémi Pautrat, Ondrej Miksik, Marc Pollefeys, Iro Armeni, Mahdi Rad, Mihai Dusmanu
Computation and Language · Cross-listed from cs.LG
Quantization-Robust LLM Unlearning via Low-Rank Adaptation
João Vitor Boer Abitante, Joana Meneguzzo Pasquali, Luan Fonseca Garcia et al. (+4 more)
Large
Language Model (LLM) unlearning aims to remove targeted knowledge from a trained model, but practical deployments often require post-training quantization (PTQ) for efficient inference. However, aggressive low-bit PTQ can mask unlearning updates, causing quantized models to revert to pre-unlearning behavior. We show that standard full-parameter fine-tuning often induces parameter changes that are too small to survive 4-bit quantization. We propose quantization-robust unlearning via low-rank adaptation (LoRA): we freeze the base model and concentrate unlearning into trainable adapters so that the effective update is preserved after quantization. On Llama-2-7B evaluated with MUSE
dataset (BOOKS and NEWS), LoRA improves 4-bit utility by up to 7.93 points (NPO+GDR on BOOKS: 50.17 to 58.10) and yields higher 4-bit utility on NEWS for GA+GDR (40.06 to 44.82, increase of 4.76). LoRA also substantially reduces
privacy leakage under 4-bit PTQ, e.g., for GA+KLR on BOOKS, PrivLeak moves from -25.68 to -5.86 (closer to ideal 0), while maintaining strong forgetting (VerMem and KnowMem near 0). Thus, using LoRA for Machine Unlearning is beneficial for scenarios where quantization is necessary for model deployment.
Read the full abstract ▾
All authors João Vitor Boer Abitante, Joana Meneguzzo Pasquali, Luan Fonseca Garcia, Ewerton de Oliveira, Thomas da Silva Paula, Rodrigo C. Barros, Lucas S. Kupssinskü
Computation and Language
OpenLID-v3: Improving the Precision of Closely Related Language Identification – An Experience Report
Mariia Fedorova, Nikolay Arefyev, Maja Buljan et al. (+4 more)
Language identification (LID) is an essential step in building high-quality
multilingual datasets from web data. Existing LID tools (such as OpenLID or GlotLID) often struggle to identify closely related
languages and to distinguish valid natural
language from noise, which contaminates
language-specific subsets, especially for low-resource
languages. In this work we extend the OpenLID classifier by adding more training data, merging problematic
language variant clusters, and introducing a special label for marking noise. We call this extended system OpenLID-v3 and evaluate it against GlotLID on multiple
benchmarks. During development, we focus on three groups of closely related
languages (Bosnian, Croatian, and Serbian; Romance varieties of Northern Italy and Southern France; and Scandinavian
languages) and contribute new evaluation
datasets where existing ones are inadequate. We find that ensemble approaches improve precision but also substantially reduce coverage for low-resource
languages. OpenLID-v3 is available on https://huggingface.co/HPLT/OpenLID-v3.
Read the full abstract ▾
All authors Mariia Fedorova, Nikolay Arefyev, Maja Buljan, Jindřich Helcl, Stephan Oepen, Egil Rønningstad, Yves Scherrer
Computation and Language
From sunblock to softblock: Analyzing the correlates of neology in published writing and on social media
Maria Ryskina, Matthew R. Gormley, Kyle Mahowald et al. (+3 more)
Living
languages are shaped by a host of conflicting internal and external evolutionary pressures. While some of these pressures are universal across
languages and cultures, others differ depending on the social and conversational context:
language use in newspapers is subject to very different constraints than
language use on social media. Prior distributional semantic work on English word emergence (neology) identified two factors correlated with creation of new words by analyzing a corpus consisting primarily of historical published texts (Ryskina et al., 2020, arXiv:2001.07740). Extending this methodology to contextual
embeddings in addition to static ones and applying it to a new corpus of Twitter posts, we show that the same findings hold for both domains, though the topic popularity growth factor may contribute less to neology on Twitter than in published writing. We hypothesize that this difference can be explained by the two domains favouring different neologism formation mechanisms.
Read the full abstract ▾
All authors Maria Ryskina, Matthew R. Gormley, Kyle Mahowald, David R. Mortensen, Taylor Berg-Kirkpatrick, Vivek Kulkarni
Computation and Language
SCOPE: Selective Conformal Optimized Pairwise LLM Judging
Sher Badshah, Ali Emami and Hassan Sajjad
Large
language models (LLMs) are increasingly used as scalable judges in pairwise evaluation, but they remain prone to miscalibration and biases. We propose Scope (Selective Conformal Optimized Pairwise Evaluation), a framework that calibrates an acceptance threshold so that, under exchangeability, the error rate among non-abstained judgments is at most a user-specified level $\alpha$. To supply Scope with a bias-neutral uncertainty signal, we introduce Bidirectional Preference
Entropy (BPE), which queries the judge under both response positions and converts the order-averaged preference probability into an
entropy-based score. Across various pairwise judging
benchmarks, BPE outperforms standard confidence proxies in calibration and discrimination, while Scope consistently satisfies the target risk bound (empirical FDR $\approx 0.097$–$0.099$ at $\alpha=0.10$) and retains substantial coverage. Compared to vanilla baselines, Scope accepts up to $2.4\times$ more judgments under the same risk constraint, demonstrating that BPE enables reliable and high-coverage LLM-based evaluation.
Read the full abstract ▾
Computation and Language
Towards interpretable models for language proficiency assessment: Predicting the CEFR level of Estonian learner texts
Kais Allkivi
Using NLP to analyze authentic learner
language helps to build automated assessment and feedback tools. It also offers new and extensive insights into the development of second
language production. However, there is a lack of research explicitly combining these aspects. This study aimed to classify Estonian proficiency examination writings (levels A2-C1), assuming that careful feature selection can lead to more explainable and generalizable machine learning models for
language testing. Various linguistic properties of the training data were analyzed to identify relevant proficiency predictors associated with increasing complexity and correctness, rather than the writing task. Such lexical, morphological, surface, and error features were used to train classification models, which were compared to models that also allowed for other features. The pre-selected features yielded a similar test accuracy but reduced variation in the classification of different text types. The best classifiers achieved an accuracy of around 0.9. Additional evaluation on an earlier exam sample revealed that the writings have become more complex over a 7-10-year period, while accuracy still reached 0.8 with some feature sets. The results have been implemented in the writing evaluation module of an Estonian open-source
language learning environment.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.AI
Consistency of Large Reasoning Models Under Multi-Turn Attacks
Yubo Li, Ramayya Krishnan and Rema Padman
Large
reasoning models with
reasoning capabilities achieve state-of-the-art performance on complex tasks, but their robustness under multi-turn
adversarial pressure remains underexplored. We evaluate nine frontier
reasoning models under
adversarial attacks. Our findings reveal that
reasoning confers meaningful but incomplete robustness: most
reasoning models studied significantly outperform instruction-tuned baselines, yet all exhibit distinct vulnerability profiles, with misleading suggestions universally effective and social pressure showing model-specific efficacy. Through trajectory analysis, we identify five failure modes (Self-Doubt, Social Conformity, Suggestion Hijacking, Emotional Susceptibility, and
Reasoning Fatigue) with the first two accounting for 50% of failures. We further demonstrate that Confidence-Aware Response Generation (CARG), effective for standard LLMs, fails for
reasoning models due to overconfidence induced by extended
reasoning traces; counterintuitively, random confidence
embedding outperforms targeted extraction. Our results highlight that
reasoning capabilities do not automatically confer
adversarial robustness and that confidence-based defenses require fundamental redesign for
reasoning models.
Read the full abstract ▾
Computation and Language
Exploring a New Competency Modeling Process with Large Language Models
Silin Du, Manqing Xin and Raymond Jia Wang
Competency modeling is widely used in human resource management to select, develop, and evaluate talent. However, traditional expert-driven approaches rely heavily on manual analysis of large volumes of interview transcripts, making them costly and prone to randomness, ambiguity, and limited reproducibility. This study proposes a new competency modeling process built on large
language models (LLMs). Instead of merely automating isolated steps, we reconstruct the workflow by decomposing expert practices into structured computational components. Specifically, we leverage LLMs to extract behavioral and psychological descriptions from raw textual data and map them to predefined competency libraries through
embedding-based similarity. We further introduce a learnable parameter that adaptively integrates different information sources, enabling the model to determine the relative importance of behavioral and psychological signals. To address the long-standing challenge of validation, we develop an offline evaluation procedure that allows systematic model selection without requiring additional large-scale data collection. Empirical results from a real-world implementation in a software outsourcing company demonstrate strong predictive validity, cross-library consistency, and structural robustness. Overall, our framework transforms competency modeling from a largely qualitative and expert-dependent practice into a transparent, data-driven, and evaluable analytical process.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.LG
LCSB: Layer-Cyclic Selective Backpropagation for Memory-Efficient On-Device LLM Fine-Tuning
Juneyoung Park, Eunbeen Yoon and Seongwan Kim. Jaeho Lee
Memory-efficient backpropagation (MeBP) has enabled first-order fine-tuning of large
language models (LLMs) on mobile devices with less than 1GB
memory. However, MeBP requires backward computation through all
transformer layers at every step, where weight decompression alone accounts for 32–42% of backward time. We propose Layer-Cyclic Selective Backpropagation (LCSB), which computes gradients for only a subset of layers per step. Our key insight is that residual connections guarantee gradient flow through identity paths, while AdamW momentum provides implicit updates for non-selected layers. We interpret LCSB as Block Coordinate Descent on the LoRA parameter space, providing theoretical justification for convergence. LCSB achieves up to 1.40$\times$ speedup with less than 2% quality degradation across five models and three tasks. Surprisingly, in 4-bit quantized settings, LCSB exhibits superior stability: a 3B model that completely diverges under full backpropagation converges smoothly with LCSB, suggesting an implicit regularization effect from selective gradient computation.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.LG
Memory-Efficient Structured Backpropagation for On-Device LLM Fine-Tuning
Juneyoung Park, Yuri Hong, Seongwan Kim et al. (+1 more)
On-device fine-tuning enables
privacy-preserving personalization of large
language models, but mobile devices impose severe
memory constraints, typically 6–12GB shared across all workloads. Existing approaches force a trade-off between exact gradients with high
memory (MeBP) and low
memory with noisy estimates (MeZO). We propose
Memory-efficient Structured Backpropagation (MeSP), which bridges this gap by manually deriving backward passes that exploit LoRA's low-rank structure. Our key insight is that the intermediate projection $h = xA$ can be recomputed during backward at minimal cost since rank $r \ll d_{in}$, eliminating the need to store it. MeSP achieves 49% average
memory reduction compared to MeBP on Qwen2.5 models (0.5B–3B) while computing mathematically identical gradients. Our analysis also reveals that MeZO's gradient estimates show near-zero correlation with true gradients (cosine similarity $\approx$0.001), explaining its slow convergence. MeSP reduces peak
memory from 361MB to 136MB for Qwen2.5-0.5B, enabling fine-tuning scenarios previously infeasible on
memory-constrained devices.
Read the full abstract ▾
All authors Juneyoung Park, Yuri Hong, Seongwan Kim, Jaeho Lee
Computation and Language
TraceBack: Multi-Agent Decomposition for Fine-Grained Table Attribution
Tejas Anvekar, Junha Park, Rajat Jha et al. (+4 more)
Question answering (QA) over structured tables requires not only accurate answers but also transparency about which
cells support them. Existing table QA systems rarely provide fine-grained attribution, so even correct answers often lack verifiable grounding, limiting trust in high-stakes settings. We address this with TraceBack, a modular multi-
agent framework for scalable,
cell-level attribution in single-table QA. TraceBack prunes tables to relevant rows and columns, decomposes questions into semantically coherent sub-questions, and aligns each answer span with its supporting
cells, capturing both explicit and implicit evidence used in intermediate
reasoning steps. To enable systematic evaluation, we release CITEBench, a
benchmark with phrase-to-
cell annotations drawn from ToTTo, FetaQA, and AITQA. We further propose FairScore, a reference-less metric that compares atomic facts derived from predicted
cells and answers to estimate attribution precision and recall without human
cell labels. Experiments show that TraceBack substantially outperforms strong baselines across
datasets and granularities, while FairScore closely tracks human judgments and preserves relative method rankings, supporting interpretable and scalable evaluation of table-based QA.
Read the full abstract ▾
All authors Tejas Anvekar, Junha Park, Rajat Jha, Devanshu Gupta, Poojah Ganesan, Puneeth Mathur, Vivek Gupta
Computation and Language
False positive bias in AI-powered speech-based cognitive screening for multilingual English speakers in the UK
Madhurananda Pahar, Caitlin Illingworth, Dorota Braun et al. (+4 more)
Conversational
speech reveals early signs of cognitive decline, including dementia and mild cognitive impairment (MCI). AI models show promise for
speech-based screening, yet most research focuses on monolingual groups. In the UK, dementia is projected to rise fastest among Black and Asian communities, where multilingualism is common, making equity assessment critical. We recruited 1,395 participants (monolingual English speakers and
multilingual speakers from Sheffield/Bradford) and collected over 263 hours of
speech via the CognoMemory
agent.
Multilingual participants spoke English alongside Somali, Chinese, or South Asian
languages (Hindi, Urdu, Punjabi, Mirpuri, Arabic). We evaluated ASR (Whisper, Wav2Vec 2.0, NeMo) and downstream AI models for cognitive classification and MMSE regression. ASR accuracy showed no significant differences across groups. However, downstream models exhibited systematic disparities:
multilingual speakers were more often misclassified as impaired, especially in
memory, fluency, and reading tasks. False-positive rates were substantially higher for
multilingual (28 to 37%) than monolingual (12 to 16%) speakers, meaning
multilingual individuals were approximately 2.5 times more likely to receive incorrect impairment labels. These biases worsened when models were trained on DementiaBank. This is the first large-scale analysis of false-positive bias in
speech-based AI cognitive screening for UK
multilingual ethnic minorities. Despite strong overall performance, current models show measurable disparities affecting
multilingual speakers. Addressing these biases is essential for safe, equitable deployment in diverse healthcare settings.
Read the full abstract ▾
All authors Madhurananda Pahar, Caitlin Illingworth, Dorota Braun, Bahman Mirheidari, Lise Sproson, Daniel Blackburn, Heidi Christensen
Computation and Language · Cross-listed from cs.LG
Look Inward to Explore Outward: Learning Temperature Policy from LLM Internal States via Hierarchical RL
Yixiao Zhou, Yang Li, Dongzhou Cheng et al. (+2 more)
Reinforcement Learning from Verifiable Rewards (RLVR) trains large
language models (LLMs) from sampled trajectories, making decoding strategy a core component of learning rather than a purely inference-time choice. Sampling temperature directly controls the exploration–exploitation trade-off by modulating policy
entropy, yet existing methods rely on static values or heuristic adaptations that are decoupled from task-level rewards. We propose Introspective LLM, a hierarchical reinforcement learning framework that learns to control sampling temperature during generation. At each decoding step, the model selects a temperature based on its hidden state and samples the next
token from the resulting distribution. Temperature and
token policies are jointly optimized from downstream rewards using a coordinate ascent scheme. Experiments on mathematical
reasoning benchmarks show that learned temperature policies outperform fixed and heuristic baselines, while exhibiting interpretable exploration behaviors aligned with
reasoning uncertainty.
Read the full abstract ▾
All authors Yixiao Zhou, Yang Li, Dongzhou Cheng, Hehe Fan, Yu Cheng
Computation and Language · Cross-listed from cs.CY
Buy versus Build an LLM: A Decision Framework for Governments
Jiahao Lu, Ziwei Xu, William Tjhi et al. (+4 more)
Large
Language Models (LLMs) represent a new frontier of digital infrastructure that can support a wide range of public-sector applications, from general purpose citizen services to specialized and sensitive state functions. When expanding AI access, governments face a set of strategic choices over whether to buy existing services, build domestic capabilities, or adopt hybrid approaches across different domains and use cases. These are critical decisions especially when leading model providers are often foreign corporations, and LLM outputs are increasingly treated as trusted inputs to public decision-making and public discourse. In practice, these decisions are not intended to mandate a single approach across all domains; instead, national AI strategies are typically pluralistic, with sovereign, commercial and open-source models coexisting to serve different purposes. Governments may rely on commercial models for non-sensitive or commodity tasks, while pursuing greater control for critical, high-risk or strategically important applications. This paper provides a strategic framework for making this decision by evaluating these options across dimensions including sovereignty, safety, cost, resource capability, cultural fit, and sustainability. Importantly, "building" does not imply that governments must act alone: domestic capabilities may be developed through public research institutions, universities, state-owned enterprises, joint ventures, or broader national ecosystems. By detailing the technical requirements and practical challenges of each pathway, this work aims to serve as a reference for policy-makers to determine whether a buy or build approach best aligns with their specific national needs and societal goals.
Read the full abstract ▾
All authors Jiahao Lu, Ziwei Xu, William Tjhi, Junnan Li, Antoine Bosselut, Pang Wei Koh, Mohan Kankanhalli
Computation and Language · Cross-listed from cs.CV
Human-Aligned MLLM Judges for Fine-Grained Image Editing Evaluation: A Benchmark, Framework, and Analysis
Runzhou Liu, Hailey Weingord, Sejal Mittal et al. (+18 more)
Evaluating image editing models remains challenging due to the coarse granularity and limited interpretability of traditional metrics, which often fail to capture aspects important to human perception and intent. Such metrics frequently reward visually plausible outputs while overlooking controllability, edit localization, and faithfulness to user instructions. In this work, we introduce a fine-grained
Multimodal Large
Language Model (MLLM)-as-a-Judge framework for image editing that decomposes common evaluation notions into twelve fine-grained interpretable factors spanning image preservation, edit quality, and instruction fidelity. Building on this formulation, we present a new human-validated
benchmark that integrates human judgments, MLLM-based evaluations, model outputs, and traditional metrics across diverse image editing tasks. Through extensive human studies, we show that the proposed MLLM judges align closely with human evaluations at a fine granularity, supporting their use as reliable and scalable evaluators. We further demonstrate that traditional image editing metrics are often poor proxies for these factors, failing to distinguish over-edited or semantically imprecise outputs, whereas our judges provide more intuitive and informative assessments in both offline and online settings. Together, this work introduces a
benchmark, a principled factorization, and empirical evidence positioning fine-grained MLLM judges as a practical foundation for studying, comparing, and improving image editing approaches.
Read the full abstract ▾
All authors Runzhou Liu, Hailey Weingord, Sejal Mittal, Prakhar Dungarwal, Anusha Nandula, Bo Ni, Samyadeep Basu, Hongjie Chen, Nesreen K. Ahmed, Li Li, Jiayi Zhang, Koustava Goswami, Subhojyoti Mukherjee, Branislav Kveton, Puneet Mathur, Franck Dernoncourt, Yue Zhao, Yu Wang, Ryan A. Rossi, Zhengzhong Tu, Hongru Du
Computation and Language
Know More, Know Clearer: A Meta-Cognitive Framework for Knowledge Augmentation in Large Language Models
Hao Chen, Ye He, Yuchun Fan et al. (+5 more)
Knowledge augmentation has significantly enhanced the performance of Large
Language Models (LLMs) in knowledge-intensive tasks. However, existing methods typically operate on the simplistic premise that model performance equates with internal knowledge, overlooking the knowledge-confidence gaps that lead to overconfident errors or uncertain truths. To bridge this gap, we propose a novel meta-cognitive framework for reliable knowledge augmentation via differentiated intervention and
alignment. Our approach leverages internal cognitive signals to partition the knowledge space into mastered, confused, and missing regions, guiding targeted knowledge expansion. Furthermore, we introduce a cognitive consistency mechanism to synchronize subjective certainty with objective accuracy, ensuring calibrated knowledge boundaries. Extensive experiments demonstrate the our framework consistently outperforms strong baselines, validating its rationality in not only enhancing knowledge capabilities but also fostering cognitive behaviors that better distinguish knowns from unknowns. All codes are available at https://github.com/AI9
Stars/Know-More-Know-Clearer.
Read the full abstract ▾
All authors Hao Chen, Ye He, Yuchun Fan, Yukun Yan, Zhenghao Liu, Qingfu Zhu, Maosong Sun, Wanxiang Che
Computation and Language
Evaluating the Homogeneity of Keyphrase Prediction Models
Maël Houbre, Florian Boudin and Beatrice Daille
Keyphrases which are useful in several NLP and IR applications are either extracted from text or predicted by generative models. Contrarily to keyphrase extraction approaches, keyphrase generation models can predict keyphrases that do not appear in a document's text called `absent keyphrases`. This ability means that keyphrase generation models can associate a document to a notion that is not explicitly mentioned in its text. Intuitively, this suggests that for two documents treating the same subjects, a keyphrase generation model is more likely to be homogeneous in their indexing i.e. predict the same keyphrase for both documents, regardless of those keyphrases appearing in their respective text or not; something a keyphrase extraction model would fail to do. Yet, homogeneity of keyphrase prediction models is not covered by current
benchmarks. In this work, we introduce a method to evaluate the homogeneity of keyphrase prediction models and study if absent keyphrase generation capabilities actually help the model to be more homogeneous. To our surprise, we show that keyphrase extraction methods are competitive with generative models, and that the ability to generate absent keyphrases can actually have a negative impact on homogeneity. Our data, code and
prompts are available on huggingface and github.
Read the full abstract ▾
Computation and Language
SciAgentGym: Benchmarking Multi-Step Scientific Tool-use in LLM Agents
Yujiong Shen, Yajie Yang, Zhiheng Xi et al. (+17 more)
Scientific
reasoning inherently demands integrating sophisticated toolkits to navigate domain-specific knowledge. Yet, current
benchmarks largely overlook
agents' ability to orchestrate tools for such rigorous workflows. To bridge this gap, we introduce SciAgentGym, a scalable interactive environment featuring 1,780 domain-specific tools across four natural science disciplines, supported by a robust execution infrastructure. Complementing this, we present SciAgentBench, a tiered evaluation suite designed to stress-test agentic capabilities from elementary actions to long-horizon workflows. Our evaluation identifies a critical bottleneck: state-of-the-art models still struggle with complex scientific tool-use, and their performance degrades substantially as interaction horizons extend. To address this, we propose SciForge, a data synthesis method that models the tool action space as a dependency graph to generate logic-aware training trajectories. By fine-tuning on these trajectories, our SciAgent-8B outperforms the significantly larger Qwen3-VL-235B-Instruct while exhibiting positive cross-domain transfer of scientific tool-use capabilities. These results underscore the promising potential of next-generation autonomous scientific
agents.
Read the full abstract ▾
All authors Yujiong Shen, Yajie Yang, Zhiheng Xi, Binze Hu, Huayu Sha, Jiazheng Zhang, Qiyuan Peng, Junlin Shang, Jixuan Huang, Yutao Fan, Jingqi Tong, Shihan Dou, Ming Zhang, Lei Bai, Zhenfei Yin, Tao Gui, Xingjun Ma, Qi Zhang, Xuanjing Huang, Yu-Gang Jiang
Computation and Language · Cross-listed from cs.IR
RGAlign-Rec: Ranking-Guided Alignment for Latent Query Reasoning in Recommendation Systems
Junhua Liu, Yang Jihao, Cheng Chang et al. (+3 more)
Proactive intent prediction is a critical capability in modern e-commerce chatbots, enabling "zero-query" recommendations by anticipating user needs from behavioral and contextual signals. However, existing industrial systems face two fundamental challenges: (1) the semantic gap between discrete user features and the semantic intents within the chatbot's Knowledge Base, and (2) the objective misalignment between general-purpose LLM outputs and task-specific ranking utilities. To address these issues, we propose RGAlign-Rec, a closed-loop
alignment framework that integrates an LLM-based semantic reasoner with a Query-Enhanced (QE) ranking model. We also introduce Ranking-Guided
Alignment (RGA), a multi-stage training paradigm that utilizes downstream ranking signals as feedback to refine the LLM's latent
reasoning. Extensive experiments on a large-scale industrial
dataset from Shopee demonstrate that RGAlign-Rec achieves a 0.12% gain in GAUC, leading to a significant 3.52% relative reduction in error rate, and a 0.56% improvement in Recall@3. Online A/B testing further validates the cumulative effectiveness of our framework: the Query-Enhanced model (QE-Rec) initially yields a 0.98% improvement in CTR, while the subsequent Ranking-Guided
Alignment stage contributes an additional 0.13% gain. These results indicate that ranking-aware
alignment effectively synchronizes semantic
reasoning with ranking objectives, significantly enhancing both prediction accuracy and service quality in real-world proactive recommendation systems.
Read the full abstract ▾
All authors Junhua Liu, Yang Jihao, Cheng Chang, Kunrong LI, Bin Fu, Kwan Hui Lim
Computation and Language
ProbeLLM: Automating Principled Diagnosis of LLM Failures
Yue Huang, Zhengzhe Jiang, Yuchen Ma et al. (+8 more)
Understanding how and why large
language models (LLMs) fail is becoming a central challenge as models rapidly evolve and static evaluations fall behind. While automated probing has been enabled by dynamic test generation, existing approaches often discover isolated failure cases, lack principled control over exploration, and provide limited insight into the underlying structure of model weaknesses. We propose ProbeLLM, a
benchmark-agnostic automated probing framework that elevates weakness discovery from individual failures to structured failure modes. ProbeLLM formulates probing as a hierarchical Monte Carlo Tree Search, explicitly allocating limited probing budgets between global exploration of new failure regions and local refinement of recurring error patterns. By restricting probing to verifiable test cases and leveraging tool-augmented generation and verification, ProbeLLM grounds failure discovery in reliable evidence. Discovered failures are further consolidated into interpretable failure modes via failure-aware
embeddings and boundary-aware induction. Across diverse
benchmarks and LLMs, ProbeLLM reveals substantially broader, cleaner, and more fine-grained failure landscapes than static
benchmarks and prior automated methods, supporting a shift from case-centric evaluation toward principled weakness discovery.
Read the full abstract ▾
All authors Yue Huang, Zhengzhe Jiang, Yuchen Ma, Yu Jiang, Xiangqi Wang, Yujun Zhou, Yuexing Hao, Kehan Guo, Pin-Yu Chen, Stefan Feuerriegel, Xiangliang Zhang
Computation and Language
When Words Don't Mean What They Say: Figurative Understanding in Bengali Idioms
Adib Sakhawat, Shamim Ara Parveen, Md Ruhul Amin et al. (+3 more)
Figurative
language understanding remains a significant challenge for Large
Language Models (LLMs), especially for low-resource
languages. To address this, we introduce a new idiom
dataset, a large-scale, culturally-grounded corpus of 10,361 Bengali idioms. Each idiom is annotated under a comprehensive 19-field schema, established and refined through a deliberative expert consensus process, that captures its semantic, syntactic, cultural, and religious dimensions, providing a rich, structured resource for computational linguistics. To establish a robust
benchmark for Bangla figurative
language understanding, we evaluate 30 state-of-the-art
multilingual and instruction-tuned LLMs on the task of inferring figurative meaning. Our results reveal a critical performance gap, with no model surpassing 50% accuracy, a stark contrast to significantly higher human performance (83.4%). This underscores the limitations of existing models in cross-linguistic and cultural
reasoning. By releasing the new idiom
dataset and
benchmark, we provide foundational infrastructure for advancing figurative
language understanding and cultural grounding in LLMs for Bengali and other low-resource
languages.
Read the full abstract ▾
All authors Adib Sakhawat, Shamim Ara Parveen, Md Ruhul Amin, Shamim Al Mahmud, Md Saiful Islam, Tahera Khatun
Computation and Language
ViMedCSS: A Vietnamese Medical Code-Switching Speech Dataset & Benchmark
Tung X. Nguyen, Nhu Vo, Giang-Son Nguyen et al. (+6 more)
Code-switching (CS), which is when Vietnamese
speech uses English words like drug names or procedures, is a common phenomenon in Vietnamese medical communication. This creates challenges for Automatic
Speech Recognition (ASR) systems, especially in low-resource
languages like Vietnamese. Current most ASR systems struggle to recognize correctly English medical terms within Vietnamese sentences, and no
benchmark addresses this challenge. In this paper, we construct a 34-hour Vietnamese Medical Code-Switching
Speech dataset (ViMedCSS) containing 16,576 utterances. Each utterance includes at least one English medical term drawn from a curated bilingual lexicon covering five medical topics. Using this
dataset, we evaluate several state-of-the-art ASR models and examine different specific fine-tuning strategies for improving medical term recognition to investigate the best approach to solve in the
dataset. Experimental results show that Vietnamese-optimized models perform better on general segments, while
multilingual pretraining helps capture English insertions. The combination of both approaches yields the best balance between overall and code-switched accuracy. This work provides the first
benchmark for Vietnamese medical code-switching and offers insights into effective domain adaptation for low-resource,
multilingual ASR systems.
Read the full abstract ▾
All authors Tung X. Nguyen, Nhu Vo, Giang-Son Nguyen, Duy Mai Hoang, Chien Dinh Huynh, Inigo Jauregi Unanue, Massimo Piccardi, Wray Buntine, Dung D. Le
Computation and Language · Cross-listed from cs.CV
RADAR: Revealing Asymmetric Development of Abilities in MLLM Pre-training
Yunshuang Nie, Bingqian Lin, Minzhe Niu et al. (+7 more)
Pre-trained Multi-modal Large
Language Models (MLLMs) provide a knowledge-rich foundation for post-training by leveraging their inherent perception and
reasoning capabilities to solve complex tasks. However, the lack of an efficient evaluation framework impedes the diagnosis of their performance bottlenecks. Current evaluation primarily relies on testing after supervised fine-tuning, which introduces laborious additional training and autoregressive decoding costs. Meanwhile, common pre-training metrics cannot quantify a model's perception and
reasoning abilities in a disentangled manner. Furthermore, existing evaluation
benchmarks are typically limited in scale or misaligned with pre-training objectives. Thus, we propose RADAR, an efficient ability-centric evaluation framework for Revealing Asymmetric Development of Abilities in MLLM pRe-training. RADAR involves two key components: (1) Soft Discrimination Score, a novel metric for robustly tracking ability development without fine-tuning, based on quantifying nuanced gradations of the model preference for the correct answer over distractors; and (2) Multi-Modal Mixture
Benchmark, a new 15K+ sample
benchmark for comprehensively evaluating pre-trained MLLMs' perception and
reasoning abilities in a 0-shot manner, where we unify authoritative
benchmark datasets and carefully collect new
datasets, extending the evaluation scope and addressing the critical gaps in current
benchmarks. With RADAR, we comprehensively reveal the asymmetric development of perceptual and
reasoning capabilities in pretrained MLLMs across diverse factors, including data volume, model size, and pretraining strategy. Our RADAR underscores the need for a decomposed perspective on pre-training ability bottlenecks, informing targeted interventions to advance MLLMs efficiently. Our code is publicly available at https://github.com/Nieysh/RADAR.
Read the full abstract ▾
All authors Yunshuang Nie, Bingqian Lin, Minzhe Niu, Kun Xiang, Jianhua Han, Guowei Huang, Xingyue Quan, Hang Xu, Bokui Chen, Xiaodan Liang
Computation and Language
BaziQA-Benchmark: Evaluating Symbolic and Temporally Compositional Reasoning in Large Language Models
Jiangxi Chen and Qian Liu
We present BaziQA-
Benchmark, a standardized
benchmark for evaluating symbolic and temporally compositional
reasoning in large
language models. The
benchmark is derived from 200 professionally curated, multiple-choice problems from the Global Fortune-teller Competition (2021–2025), where each instance requires structured inference over a fixed symbolic chart and interacting temporal conditions. Unlike anecdotal or
prompt-driven evaluations, BaziQA-
Benchmark enables objective scoring and controlled comparison across years, domains, and model families. We evaluate contemporary
language models under a multi-turn setting and analyze performance variation across temporal difficulty,
reasoning domains, and inference protocols.To further probe
reasoning behavior, we introduce a lightweight Structured
Reasoning Protocol that constrains inference order without adding domain knowledge. Results show that models consistently outperform chance but remain far from saturation, exhibiting pronounced sensitivity to temporal composition and
reasoning order, as well as systematic failures on precise temporal localization and multi-condition symbolic judgments.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.SI
Semantic Communities and Boundary-Spanning Lyrics in K-pop: A Graph-Based Unsupervised Analysis
Oktay Karakuş and Can Eyüpoğlu
We examine how lyric representation shapes computational similarity using four 384-dimensional
multilingual MiniLM representations of 7,891 K-pop songs by 1,456 artists from charts spanning 2000 to October 2023. Whole-song truncated
embeddings provide the strongest standardized separation between neighbouring and random song pairs (1.995), compared with 1.931 for chunked documents, although chunked representations show greater adjusted Rand agreement and line-based representations yield higher modularity. Using whole-song
embeddings, we construct a weighted cosine nearest-neighbour graph ($k=30$). A representative Louvain partition identifies 21 operational communities, without a corresponding robust plateau in the Markov Stability analysis. AI-assisted examination of selected lyrics and terms suggests overlapping patterns of
memory, separation, longing, self-reflection, attraction, and performance. Community membership is associated with character-script composition (Cramér's $V=0.294$; permutation $p=0.001$). High-participation songs exhibit broader cross-community connectivity, while participation has weak negative associations with
embedding-relative novelty. These findings show how representation choices and textual characteristics influence the network organization of a cultural corpus, subject to limitations from truncation, graph construction, and the absence of independent human semantic validation.
Read the full abstract ▾
Computation and Language
MentalBench: A DSM-Grounded Benchmark for Evaluating Psychiatric Diagnostic Capability of Large Language Models
Hoyun Song, Migyeong Kang, Jisu Shin et al. (+7 more)
Large
language models (LLMs) have attracted growing interest as supportive tools for psychiatric assessment and clinical decision support. However, existing mental health
benchmarks largely rely on social media data or supportive dialogue settings, limiting their ability to assess whether models can apply formal diagnostic criteria and differential diagnostic rules. In this paper, we introduce MentalBench, a
benchmark for evaluating whether LLMs can make DSM-grounded psychiatric diagnostic decisions under varying levels of clinical ambiguity. At the core of MentalBench is MentalKG, a psychiatrist-built and validated knowledge graph encoding DSM-5 diagnostic criteria and differential diagnostic rules for 23 psychiatric disorders. Using MentalKG as an expert-curated logical backbone, we generate 24,750 synthetic clinical cases that systematically vary in information completeness and diagnostic complexity, enabling DSM-grounded evaluation. Our experiments show that although state-of-the-art LLMs perform well on noise-free queries that probe DSM-5 knowledge, they struggle to calibrate their confidence when distinguishing between disorders with overlapping symptoms. These findings raise concerns about the reliability of LLMs as psychiatric decision-support tools and highlight the need for more evaluation that reflects the diverse challenges in real-world psychiatric diagnosis.
Read the full abstract ▾
All authors Hoyun Song, Migyeong Kang, Jisu Shin, Jihyun Kim, Chanbi Park, Hangyeol Yoo, Jihyun An, Alice Oh, Jinyoung Han, KyungTae Lim