Computation and Language
LLM BiasScope: A Real-Time Bias Analysis Platform for Comparative LLM Evaluation
Himel Ghosh and Nick Elias Werner
As large
language models (LLMs) are deployed widely, detecting and understanding bias in their outputs is critical. We present LLM BiasScope, a web application for side-by-side comparison of LLM outputs with real-time bias analysis. The system supports multiple providers (Google Gemini, DeepSeek, MiniMax, Mistral, Meituan, Meta Llama) and enables researchers and practitioners to compare models on the same
prompts while analyzing bias patterns. LLM BiasScope uses a two-stage bias detection pipeline: sentence-level bias detection followed by bias type classification for biased sentences. The analysis runs automatically on both user
prompts and model responses, providing statistics, visualizations, and detailed breakdowns of bias types. The interface displays two models side-by-side with synchronized streaming responses, per-model bias summaries, and a comparison view highlighting differences in bias distributions. The system is built on Next.js with React, integrates Hugging Face inference endpoints for bias detection, and uses the Vercel AI SDK for multi-provider LLM access. Features include real-time streaming, export to JSON/PDF, and interactive visualizations (bar charts, radar charts) for bias analysis. LLM BiasScope is available as an open-source web application, providing a practical tool for bias evaluation and comparative analysis of LLM behaviour.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.LG
When LLM Judge Scores Look Good but Best-of-N Decisions Fail
Eddie Landesberg
Large
language models are often used as judges to score candidate responses, then validated with a single global metric such as correlation with reference labels. This can be misleading when the real deployment task is best-of-n selection within a
prompt. In a 5,000-
prompt best-of-4
benchmark from Chatbot Arena, a judge with moderate global correlation (r = 0.47) captures only 21.0% of the improvement that perfect selection would achieve over random choice. The gap arises because global agreement is driven largely by
prompt-level baseline effects, while selection depends on within-
prompt ranking: within-
prompt correlation is only r_within = 0.27, and coarse pointwise scoring creates ties in 67% of pairwise comparisons. In a matched-pair best-of-2 audit, explicit pairwise judging recovers much of this lost signal, raising recovery from 21.1% to 61.2%. For judge-based selection, the relevant audit should report within-
prompt signal, tie rates, and recovery/top-1 accuracy, not global agreement alone.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.RO
Red-Teaming Vision-Language-Action Models via Quality Diversity Prompt Generation for Robust Robot Policies
Siddharth Srikanth, Freddie Liang, Ya-Chuan Hsu et al. (+9 more)
Vision-
Language-Action (VLA) models have significant potential to enable general-purpose robotic systems for a range of
vision-
language tasks. However, the performance of VLA-based
robots is highly sensitive to the precise wording of
language instructions, and it remains difficult to predict when such
robots will fail. We propose Quality Diversity (QD) optimization as a natural framework for red-teaming
embodied models, and present Q-DIG (Quality Diversity for Diverse Instruction Generation), which performs red-teaming by scalably identifying diverse, natural
language task descriptions that induce failures while remaining task-relevant. Q-DIG integrates QD techniques with
Vision-
Language Models (VLMs) to generate a broad spectrum of
adversarial instructions that expose meaningful vulnerabilities in VLA behavior. Our results across multiple simulation
benchmarks show that Q-DIG finds more diverse and meaningful failure modes compared to baseline methods, and that fine-tuning VLAs on the generated instructions improves task success rates. Furthermore, results from a user study highlight that Q-DIG generates
prompts judged to be more natural and human-like than those from baselines. Finally, real-world evaluations of Q-DIG
prompts show results consistent with simulation, and fine-tuning VLAs on the generated
prompts further success rates on unseen instructions. Together, these findings suggest that Q-DIG is a promising approach for identifying vulnerabilities and improving the robustness of VLA-based
robots. Our anonymous project website is at qdigvla.github.io.
Read the full abstract ▾
All authors Siddharth Srikanth, Freddie Liang, Ya-Chuan Hsu, Varun Bhatt, Shihan Zhao, Henry Chen, Bryon Tjanaka, Minjune Hwang, Akanksha Saran, Daniel Seita, Aaquib Tabrez, Stefanos Nikolaidis
Computation and Language
Marked Pedagogies: Examining Linguistic Biases in Personalized Automated Writing Feedback
Mei Tan, Lena Phalen and Dorottya Demszky
Effective personalized feedback is critical to students' literacy development. Though LLM-powered tools now promise to automate such feedback at scale, LLMs are not
language-neutral: they privilege standard academic English and reproduce social stereotypes, raising concerns about how "personalization" shapes the feedback students receive. We examine how four widely used LLMs (GPT-4o, GPT-3.5-turbo, Llama-3.3 70B, Llama-3.1 8B) adapt written feedback in response to student attributes. Using 600 eighth-grade persuasive essays from the PERSUADE
dataset, we generated feedback under
prompt conditions
embedding gender, race/ethnicity, learning needs, achievement, and motivation. We analyze lexical shifts across model outputs by adapting the Marked Words framework. Our results reveal systematic, stereotype-aligned shifts in feedback conditioned on presumed student attributes–even when essay content was identical. Feedback for students marked by race,
language, or disability often exhibited positive feedback bias and feedback withholding bias–overuse of praise, less substantive critique, and assumptions of limited ability. Across attributes, models tailored not only what content was emphasized but also how writing was judged and how students were addressed. We term these instructional orientations Marked Pedagogies and highlight the need for transparency and accountability in automated feedback tools.
Read the full abstract ▾
Computation and Language
Shattering the Shortcut: A Topology-Regularized Benchmark for Multi-hop Medical Reasoning in LLMs
Xing Zi, Xinying Zhou, Jinghao Xiao et al. (+2 more)
While Large
Language Models (LLMs) achieve expert-level performance on standard medical
benchmarks through single-hop factual recall, they severely struggle with the complex, multi-hop diagnostic
reasoning required in real-world clinical settings. A primary obstacle is "shortcut learning", where models exploit highly connected, generic hub nodes (e.g., "inflammation") in knowledge graphs to bypass authentic micro-pathological cascades. To address this, we introduce ShatterMed-QA, a bilingual
benchmark of 10,558 multi-hop clinical questions designed to rigorously evaluate deep diagnostic
reasoning. Our framework constructs a
topology-regularized medical Knowledge Graph using a novel $k$-Shattering
algorithm, which physically prunes generic hubs to explicitly sever logical shortcuts. We synthesize the evaluation vignettes by applying implicit bridge entity masking and
topology-driven hard negative sampling, forcing models to navigate biologically plausible distractors without relying on superficial elimination. Comprehensive evaluations of 21 LLMs reveal massive performance degradation on our multi-hop tasks, particularly among domain-specific models. Crucially, restoring the masked evidence via
Retrieval-Augmented Generation (RAG) triggers near-universal performance recovery, validating ShatterMed-QA's structural fidelity and proving its efficacy in diagnosing the fundamental
reasoning deficits of current medical AI. Explore the
dataset, interactive examples, and full leaderboards at our project website: https://shattermed-qa-web.vercel.app/
Read the full abstract ▾
All authors Xing Zi, Xinying Zhou, Jinghao Xiao, Catarina Moreira, Mukesh Prasad
Computation and Language
CSE-UOI at SemEval-2026 Task 6: A Two-Stage Heterogeneous Ensemble with Deliberative Complexity Gating for Political Evasion Detection
Christos Tzouvaras, Konstantinos Skianis and Athanasios Voulodimos
This paper describes our system for SemEval-2026 Task 6, which classifies clarity of responses in political interviews into three categories: Clear Reply, Ambivalent, and Clear Non-Reply. We propose a heterogeneous dual large
language model (LLM) ensemble via self-consistency (SC) and weighted voting, and a novel post-hoc correction mechanism, Deliberative Complexity Gating (DCG). This mechanism uses cross-model behavioral signals and exploits the finding that an LLM response-length proxy correlates strongly with sample ambiguity. To further examine mechanisms for improving ambiguity detection, we evaluated multi-
agent debate as an alternative strategy for increasing deliberative capacity. Unlike DCG, which adaptively gates
reasoning using cross-model behavioral signals, debate increases
agent count without increasing model diversity. Our solution achieved a Macro-F1 score of 0.85 on the evaluation set, securing 3rd place and tied with the second-best reported score.
Read the full abstract ▾
Computation and Language
Interpreting Negation in GPT-2: Layer- and Head-Level Causal Analysis
Abdullah Al Mofael, Lisa M. Kuhn, Ghassan Alkadi et al. (+1 more)
Negation remains a persistent challenge for modern
language models, often causing reversed meanings or factual errors. In this work, we conduct a causal analysis of how GPT-2 Small internally processes such linguistic transformations. We examine its hidden representations at both the layer and head level. Our analysis is based on a self-curated 12,000-pair
dataset of matched affirmative and negated sentences, covering multiple linguistic templates and forms of negation. To quantify this behavior, we define a metric, the Negation Effect Score (NES), which measures the model's sensitivity in distinguishing between affirmative statements and their negations. We carried out two key interventions to probe causal structure. In activation patching, internal activations from affirmative sentences were inserted into their negated counterparts to see how meaning shifted. In ablation, specific
attention heads were temporarily disabled to observe how logical polarity changed. Together, these steps revealed how negation signals move and evolve through GPT-2's layers. Our findings indicate that this capability is not widespread; instead, it is highly concentrated within a limited number of mid-layer
attention heads, primarily within layers 4 to 6. Ablating these specific components directly disrupts the model's negation sensitivity: on our in-domain, ablation increased NES (indicating weaker negation sensitivity), and re-introducing cached affirmative activations (rescue) increased NES further, confirming that these heads carry affirmative signal rather than restoring baseline behavior. On xNot360, ablation slightly decreased NES and rescue restored performance above baseline. This pattern demonstrates that these causal patterns are consistent across various negation forms and remain detectable on the external xNot360
benchmark, though with smaller magnitude.
Read the full abstract ▾
All authors Abdullah Al Mofael, Lisa M. Kuhn, Ghassan Alkadi, Kuo-Pao Yang
Computation and Language
Not Just the Destination, But the Journey: Reasoning Traces Causally Shape Generalization Behaviors
Pengcheng Wen, Yanxu Zhu, Jiapeng Sun et al. (+5 more)
Chain-of-Thought (CoT) is often viewed as a window into LLM decision-making, yet recent work suggests it may function merely as post-hoc rationalization. This raises a critical
alignment question: Does the
reasoning trace causally shape model generalization independent of the final answer? To isolate
reasoning's causal effect, we design a controlled experiment holding final harmful answers constant while varying
reasoning paths. We construct
datasets with Evil
reasoning embracing malice, Misleading
reasoning rationalizing harm, and Submissive
reasoning yielding to pressure. We train models (0.6B–14B parameters) under multiple paradigms, including question-thinking-answer (QTA), question-thinking (QT), and thinking-only (T-only), and evaluate them in both think and no-think modes. We find that: (1) CoT training could amplify harmful generalization more than standard fine-tuning; (2) distinct
reasoning types induce distinct behavioral patterns aligned with their semantics, despite identical final answers; (3) training on
reasoning without answer supervision (QT or T-only) is sufficient to alter behavior, proving
reasoning carries an independent signal; and (4) these effects persist even when generating answers without
reasoning, indicating deep internalization. Our findings demonstrate that
reasoning content is causally potent, challenging
alignment strategies that supervise only outputs.
Read the full abstract ▾
All authors Pengcheng Wen, Yanxu Zhu, Jiapeng Sun, Han Zhu, Yujin Zhou, Chi-Min Chan, Sirui Han, Yike Guo
Computation and Language · Cross-listed from cs.LG
NeuroLoRA: Context-Aware Neuromodulation for Parameter-Efficient Multi-Task Adaptation
Yuxin Yang, Haoran Zhang, Mingxuan Li et al. (+6 more)
Parameter-Efficient Fine-Tuning (PEFT) techniques, particularly Low-Rank Adaptation (LoRA), have become essential for adapting Large
Language Models (LLMs) to downstream tasks. While the recent FlyLoRA framework successfully leverages bio-inspired sparse random projections to mitigate parameter interference, it relies on a static, magnitude-based routing mechanism that is agnostic to input context. In this paper, we propose NeuroLoRA, a novel Mixture-of-Experts (MoE) based LoRA framework inspired by biological neuromodulation – the dynamic regulation of neuronal excitability based on context. NeuroLoRA retains the computational efficiency of frozen random projections while introducing a lightweight, learnable neuromodulation gate that contextually rescales the projection space prior to expert selection. We further propose a Contrastive Orthogonality Loss to explicitly enforce separation between expert subspaces, enhancing both task decoupling and continual learning capacity. Extensive experiments on MMLU, GSM8K, and ScienceQA demonstrate that NeuroLoRA consistently outperforms FlyLoRA and other strong baselines across single-task adaptation, multi-task model merging, and sequential continual learning scenarios, while maintaining comparable parameter efficiency.
Read the full abstract ▾
All authors Yuxin Yang, Haoran Zhang, Mingxuan Li, Jiachen Xu, Ruoxi Shen, Zhenyu Wang, Tianhao Liu, Siqi Chen, Weilin Huang
Computation and Language · Cross-listed from cs.AI
Efficient Reasoning with Balanced Thinking
Yulin Li, Tengyao Tu, Li Ding et al. (+5 more)
Large
Reasoning Models (LRMs) have shown remarkable
reasoning capabilities, yet they often suffer from overthinking, expending redundant computational steps on simple problems, or underthinking, failing to explore sufficient
reasoning paths despite inherent capabilities. These issues lead to inefficiencies and potential inaccuracies, limiting practical deployment in resource-constrained settings. Existing methods to mitigate overthinking, such as suppressing reflective keywords or adjusting
reasoning length, may inadvertently induce underthinking, compromising accuracy. Therefore, we propose ReBalance, a training-free framework that achieves efficient
reasoning with balanced thinking. ReBalance leverages confidence as a continuous indicator of
reasoning dynamics, identifying overthinking through high confidence variance and underthinking via consistent overconfidence. By aggregating hidden states from a small-scale
dataset into
reasoning mode prototypes, we compute a steering vector to guide LRMs'
reasoning trajectories. A dynamic control function modulates this vector's strength and direction based on real-time confidence,
pruning redundancy during overthinking, and promoting exploration during underthinking. Extensive experiments conducted on four models ranging from 0.5B to 32B, and across nine
benchmarks in math
reasoning, general question answering, and coding tasks demonstrate that ReBalance effectively reduces output redundancy while improving accuracy, offering a general, training-free, and plug-and-play strategy for efficient and robust LRM deployment. Project page and code are available at https://rebalance-ai.github.io .
Read the full abstract ▾
All authors Yulin Li, Tengyao Tu, Li Ding, Junjie Wang, Huiling Zhen, Yixin Chen, Yong Li, Zhuotao Tian
Computation and Language · Cross-listed from cs.IR
Steven Dong, Yubao Tang and Maarten de Rijke
Generative
retrieval (GR) models encode a corpus within model parameters and generate relevant document identifiers directly for a given query. While this paradigm shows promise in
retrieval tasks, existing GR models struggle with complex queries in numerical contexts, such as those involving semantic
reasoning over financial reports, due to limited
reasoning capabilities. This limitation leads to suboptimal
retrieval accuracy and hinders practical applicability. We propose ReasonGR, a framework designed to enhance multi-step semantic
reasoning in numerical contexts within GR. ReasonGR employs a structured prompting strategy combining task-specific instructions with stepwise
reasoning guidance to better address complex
retrieval queries. Additionally, it integrates a
reasoning-focused adaptation module to improve the learning of
reasoning-related parameters. Experiments on the FinQA
dataset, which contains financial queries over complex documents, demonstrate that ReasonGR improves
retrieval accuracy and consistency, indicating its potential for advancing GR models in
reasoning-intensive
retrieval scenarios.
Read the full abstract ▾
Computation and Language
TASTE-Streaming: Towards Streamable Text-Aligned Speech Tokenization and Embedding for Spoken Language Modeling
Liang-Hsuan Tseng and Hung-yi Lee
Text-
speech joint spoken
language modeling (SLM) aims at natural and intelligent
speech-based interactions, but developing such a system may suffer from modality mismatch:
speech unit sequences are much longer than text
tokens. Prior work reduces this gap with text-aligned tokenization and
embedding (TASTE), producing
speech tokens that align in lengths with their textual counterparts. However, the dependence on an external ASR system and the use of a non-causal decoder limits streaming use. To address this limitation, we propose TASTE-S, a streamable extension of TASTE suitable for real-time usage. TASTE-S integrates a CTC-based ASR module into the encoder for instant dual-modality encoding. We also redesign the unit decoder to enable on-the-fly decoding. With joint training, we show that TASTE-S matches TASTE's performance while significantly reducing latency. Further investigations reveal that TASTE-S remains robust to transcriptions and enables long-form encoding and decoding.
Read the full abstract ▾
Computation and Language
LLM-Augmented Therapy Normalization and Aspect-Based Sentiment Analysis for Treatment-Resistant Depression on Reddit
Yuxin Zhu, Sahithi Lakamana, Masoud Rouhizadeh et al. (+3 more)
Treatment-resistant depression (TRD) is a severe form of major depressive disorder in which patients do not achieve remission despite multiple adequate treatment trials. Evidence across pharmacologic options for TRD remains limited, and trials often do not fully capture patient-reported tolerability. Large-scale online peer-support narratives therefore offer a complementary lens on how patients describe and evaluate medications in real-world use. In this study, we curated a corpus of 5,059 Reddit posts explicitly referencing TRD from 3,480 subscribers across 28 mental health-related subreddits from 2010 to 2025. Of these, 3,839 posts mentioned at least one medication, yielding 23,399 mentions of 81 generic-name medications after lexicon-based normalization of brand names, misspellings, and colloquialisms. We developed an aspect-based sentiment classifier by fine-tuning DeBERTa-v3 on the SMM4H 2023 therapy-sentiment Twitter corpus with large
language model based data augmentation, achieving a micro-F1 score of 0.800 on the shared-task test set. Applying this classifier to Reddit, we quantified sentiment toward individual medications across three categories: positive, neutral, and negative, and tracked patterns by drug, subscriber, subreddit, and year. Overall, 72.1% of medication mentions were neutral, 14.8% negative, and 13.1% positive. Conventional antidepressants, especially SSRIs and SNRIs, showed consistently higher negative than positive proportions, whereas ketamine and esketamine showed comparatively more favorable sentiment profiles. These findings show that normalized medication extraction combined with aspect-based sentiment analysis can help characterize patient-perceived treatment experiences in TRD-related Reddit discourse, complementing clinical evidence with large-scale patient-generated perspectives.
Read the full abstract ▾
All authors Yuxin Zhu, Sahithi Lakamana, Masoud Rouhizadeh, Selen Bozkurt, Rachel Hershenberg, Abeed Sarker
Computation and Language · Cross-listed from cs.CV
EndoCoT: Scaling Endogenous Chain-of-Thought Reasoning in Diffusion Models
Xuanlang Dai, Yujie Zhou, Long Xing et al. (+6 more)
Recently,
Multimodal Large
Language Models (MLLMs) have been widely integrated into
diffusion frameworks primarily as text encoders to tackle complex tasks such as spatial
reasoning. However, this paradigm suffers from two critical limitations: (i) MLLMs text encoder exhibits insufficient
reasoning depth. Single-step encoding fails to activate the Chain-of-Thought process, which is essential for MLLMs to provide accurate guidance for complex tasks. (ii) The guidance remains invariant during the decoding process. Invariant guidance during decoding prevents DiT from progressively decomposing complex instructions into actionable denoising steps, even with correct MLLM encodings. To this end, we propose Endogenous Chain-of-Thought (EndoCoT), a novel framework that first activates MLLMs'
reasoning potential by iteratively refining latent thought states through an iterative thought guidance module, and then bridges these states to the DiT's denoising process. Second, a terminal thought grounding module is applied to ensure the
reasoning trajectory remains grounded in textual supervision by aligning the final state with ground-truth answers. With these two components, the MLLM text encoder delivers meticulously reasoned guidance, enabling the DiT to execute it progressively and ultimately solve complex tasks in a step-by-step manner. Extensive evaluations across diverse
benchmarks (e.g., Maze, TSP, VSP, and Sudoku) achieve an average accuracy of 92.1%, outperforming the strongest baseline by 8.3 percentage points. The code and
dataset are publicly available at https://internlm.github.io/EndoCoT/.
Read the full abstract ▾
All authors Xuanlang Dai, Yujie Zhou, Long Xing, Jiazi Bu, Xilin Wei, Yuhong Liu, Beichen Zhang, Kai Chen, Yuhang Zang
Computation and Language
SciMDR: Advancing Scientific Multimodal Document Reasoning
Ziyu Chen, Yilun Zhao, Chengye Wang et al. (+3 more)
Constructing scientific
multimodal document
reasoning datasets for foundation model training involves an inherent trade-off among scale, faithfulness, and realism. To address this challenge, we introduce the synthesize-and-reground framework, a two-stage pipeline comprising: (1) Claim-Centric QA Synthesis, which generates faithful, isolated QA pairs and
reasoning on focused segments, and (2) Document-Scale Regrounding, which programmatically re-embeds these pairs into full-document tasks to ensure realistic complexity. Using this framework, we construct SciMDR, a large-scale training
dataset for cross-modal comprehension, comprising 300K QA pairs with explicit
reasoning chains across 20K scientific papers. We further construct SciMDR-Eval, an expert-annotated
benchmark to evaluate
multimodal comprehension within full-length scientific workflows. Experiments demonstrate that models fine-tuned on SciMDR achieve significant improvements across multiple scientific QA
benchmarks, particularly in those tasks requiring complex document-level
reasoning.
Read the full abstract ▾
All authors Ziyu Chen, Yilun Zhao, Chengye Wang, Rilyn Han, Manasi Patwardhan, Arman Cohan
Computation and Language · Cross-listed from cs.AI
Examining Reasoning LLMs-as-Judges in Non-Verifiable LLM Post-Training
Yixin Liu, Yue Yu, DiJia Su et al. (+7 more)
Reasoning LLMs-as-Judges, which can benefit from inference-time scaling, provide a promising path for extending the success of
reasoning models to non-verifiable domains where the output correctness/quality cannot be directly checked. However, while
reasoning judges have shown better performance on static evaluation
benchmarks, their effectiveness in actual policy training has not been systematically examined. Therefore, we conduct a rigorous study to investigate the actual impact of non-
reasoning and
reasoning judges in reinforcement-learning-based LLM
alignment. Our controlled synthetic setting, where a "gold-standard" judge (gpt-oss-120b) provides preference annotations to train smaller judges, reveals key differences between non-
reasoning and
reasoning judges: non-
reasoning judges lead to reward hacking easily, while
reasoning judges can lead to policies that achieve strong performance when evaluated by the gold-standard judge. Interestingly, we find that the
reasoning-judge-trained policies achieve such strong performance by learning to generate highly effective
adversarial outputs that can also score well on popular
benchmarks such as Arena-Hard by deceiving other LLM-judges. Combined with our further analysis, our study highlights both important findings and room for improvements for applying (
reasoning) LLM-judges in non-verifiable LLM post-training.
Read the full abstract ▾
All authors Yixin Liu, Yue Yu, DiJia Su, Sid Wang, Xuewei Wang, Song Jiang, Bo Liu, Arman Cohan, Yuandong Tian, Zhengxing Chen
Computation and Language
Sparking Scientific Creativity via LLM-Driven Interdisciplinary Inspiration
Priyanka Kargupta, Shuhaib Mehri, Dilek Hakkani-Tur et al. (+1 more)
Despite interdisciplinary research leading to larger and longer-term impact, most work remains confined to single-domain academic silos. Recent AI-based approaches to scientific discovery show promise for interdisciplinary research, but many prioritize rapidly designing experiments and solutions, bypassing the exploratory, collaborative
reasoning processes that drive creative interdisciplinary breakthroughs. As a result, prior efforts largely prioritize automating scientific discovery rather than augmenting the
reasoning processes that underlie scientific disruption. We present Idea-Catalyst, a novel framework that systematically identifies interdisciplinary insights to support creative
reasoning in both humans and large
language models. Starting from an abstract research goal, Idea-Catalyst is designed to assist the brainstorming stage, explicitly avoiding premature anchoring on specific solutions. The framework embodies key metacognitive features of interdisciplinary
reasoning: (a) defining and assessing research goals, (b) awareness of a domain's opportunities and unresolved challenges, and (c) strategic exploration of interdisciplinary ideas based on impact potential. Concretely, Idea-Catalyst decomposes an abstract goal (e.g., improving human-AI collaboration) into core target-domain research questions that guide the analysis of progress and open challenges within that domain. These challenges are reformulated as domain-agnostic conceptual problems, enabling
retrieval from external disciplines (e.g., Psychology, Sociology) that address analogous issues. By synthesizing and recontextualizing insights from these domains back into the target domain, Idea-Catalyst ranks source domains by their interdisciplinary potential. Empirically, this targeted integration improves average novelty by 21% and insightfulness by 16%, while remaining grounded in the original research problem.
Read the full abstract ▾
All authors Priyanka Kargupta, Shuhaib Mehri, Dilek Hakkani-Tur, Jiawei Han
Computation and Language
CLASP: Defending Hybrid Large Language Models Against Hidden State Poisoning Attacks
Alexandre Le Mercier, Thomas Demeester and Chris Develder
State space models (SSMs) like Mamba have gained significant traction as efficient alternatives to
Transformers, achieving linear complexity while maintaining competitive performance. However, Hidden State Poisoning Attacks (HiSPAs), a recently discovered vulnerability that corrupts SSM
memory through
adversarial strings, pose a critical threat to these architectures and their hybrid variants. Framing the HiSPA mitigation task as a binary classification problem at the
token level, we introduce the CLASP model (Classifier Against State Poisoning) to defend against this threat. CLASP exploits distinct patterns in Mamba's block output
embeddings (BOEs) and uses an XGBoost classifier to identify malicious
tokens with minimal computational overhead. We consider a realistic scenario in which both SSMs and HiSPAs are likely to be used: an LLM screening résumés to identify the best candidates for a role. Evaluated on a corpus of 2,483 résumés totaling 9.5M
tokens with controlled injections, CLASP achieves 95.9%
token-level F1 score and 99.3% document-level F1 score on malicious
tokens detection. Crucially, the model generalizes to unseen attack patterns: under leave-one-out cross-validation, performance remains high (96.9% document-level F1), while under clustered cross-validation with structurally novel triggers, it maintains useful detection capability (91.6% average document-level F1). Operating independently of any downstream model, CLASP processes 1,032
tokens per second with under 4GB VRAM consumption, potentially making it suitable for real-world deployment as a lightweight front-line defense for SSM-based and hybrid architectures. All code and detailed results are available at https://anonymous.4open.science/r/hispikes-91C0.
Read the full abstract ▾
Computation and Language
IndexCache: Accelerating Sparse Attention via Cross-Layer Index Reuse
Yushi Bai, Qian Dong, Ting Jiang et al. (+5 more)
Long-context agentic workflows have emerged as a defining use case for large
language models, making
attention efficiency critical for both inference speed and serving cost. Sparse
attention addresses this challenge effectively, and DeepSeek Sparse
Attention (DSA) is a representative production-grade solution: a lightweight lightning indexer selects the top-k most relevant
tokens per query, reducing core
attention from $O(L^2)$ to $O(Lk)$. However, the indexer itself retains $O(L^2)$ complexity and must run independently at every layer, despite the fact that the resulting top-k selections are highly similar across consecutive layers. We present IndexCache, which exploits this cross-layer redundancy by partitioning layers into a small set of Full layers that run their own indexers and a majority of Shared layers that simply reuse the nearest Full layer's top-k indices. We propose two complementary approaches to determine and optimize this configuration. Training-free IndexCache applies a greedy search
algorithm that selects which layers to retain indexers by directly minimizing
language modeling loss on a calibration set, requiring no weight updates. Training-aware IndexCache introduces a multi-layer
distillation loss that trains each retained indexer against the averaged
attention distributions of all layers it serves, enabling even simple interleaved patterns to match full-indexer accuracy. Experimental results on a 30B DSA model show that IndexCache can remove 75% of indexer computations with negligible quality degradation, achieving up to 1.82$\times$ prefill speedup and 1.48$\times$ decode speedup compared to standard DSA. These positive results are further confirmed by our preliminary experiments on the production-scale GLM-5 model (Figure 1).
Read the full abstract ▾
All authors Yushi Bai, Qian Dong, Ting Jiang, Xin Lv, Zhengxiao Du, Aohan Zeng, Jie Tang, Juanzi Li
Computation and Language
Long-Context Encoder Models for Polish Language Understanding
Sławomir Dadas, Rafał Poświata, Marek Kozłowski et al. (+4 more)
While decoder-only Large
Language Models (LLMs) have recently dominated the NLP landscape, encoder-only architectures remain a cost-effective and parameter-efficient standard for discriminative tasks. However, classic encoders like BERT are limited by a short context window, which is insufficient for processing long documents. In this paper, we address this limitation for the Polish by introducing a high-quality Polish model capable of processing sequences of up to 8192
tokens. The model was developed by employing a two-stage training procedure that involves positional
embedding adaptation and full parameter continuous pre-training. Furthermore, we propose compressed model variants trained via knowledge
distillation. The models were evaluated on 25 tasks, including the KLEJ
benchmark, a newly introduced financial task suite (FinBench), and other classification and regression tasks, specifically those requiring long-document understanding. The results demonstrate that our model achieves the best average performance among Polish and
multilingual models, significantly outperforming competitive solutions in long-context tasks while maintaining comparable quality on short texts.
Read the full abstract ▾
All authors Sławomir Dadas, Rafał Poświata, Marek Kozłowski, Małgorzata Grębowiec, Michał Perełkiewicz, Paweł Klimiuk, Przemysław Boruta
Computation and Language
Strategic Navigation or Stochastic Search? How Agents and Humans Reason Over Document Collections
Łukasz Borchmann, Jordy Van Landeghem, Michał Turski et al. (+12 more)
Multimodal agents offer a promising path to automating complex document-intensive workflows. Yet, a critical question remains: do these
agents demonstrate genuine strategic
reasoning, or merely
stochastic trial-and-error search? To address this, we introduce MADQA, a
benchmark of 2,250 human-authored questions grounded in 800 heterogeneous PDF documents. Guided by Classical Test Theory, we design it to maximize discriminative power across varying levels of agentic abilities. To evaluate agentic behaviour, we introduce a novel evaluation protocol measuring the accuracy-effort trade-off. Using this framework, we show that while the best
agents can match human searchers in raw accuracy, they succeed on largely different questions and rely on brute-force search to compensate for weak strategic planning. They fail to close the nearly 20% gap to oracle performance, persisting in unproductive loops. We release the
dataset and evaluation harness to help facilitate the transition from brute-force
retrieval to calibrated, efficient
reasoning.
Read the full abstract ▾
All authors Łukasz Borchmann, Jordy Van Landeghem, Michał Turski, Shreyansh Padarha, Ryan Othniel Kearns, Adam Mahdi, Niels Rogge, Clémentine Fourrier, Siwei Han, Huaxiu Yao, Artemis Llabrés, Yiming Xu, Dimosthenis Karatzas, Hao Zhang, Anupam Datta
Computation and Language
QAQ: Bidirectional Semantic Coherence for Selecting High-Quality Synthetic Code Instructions
Jiayin Lei, Ming Ma, Yunxi Duan et al. (+2 more)
Synthetic data has become essential for training code generation models, yet it introduces significant noise and
hallucinations that are difficult to detect with current metrics. Existing data selection methods like Instruction-Following Difficulty (IFD) typically assess how hard a model generates an answer given a query ($A|Q$). However, this metric is ambiguous on noisy synthetic data, where low probability can distinguish between intrinsic task complexity and model-generated
hallucinations. Here, we propose QAQ, a novel data selection framework that evaluates data quality from the reverse direction: how well can the answer predict the query ($Q|A$)? We define Reverse Mutual Information (RMI) to quantify the information gain about the query conditioned on the answer. Our analyses reveal that both extremes of RMI signal quality issues: low RMI indicates semantic misalignment, while excessively high RMI may contain defect patterns that LLMs easily recognize. Furthermore, we introduce a selection strategy based on the disagreement between strong and weak models to identify samples that are valid yet challenging. Experiments across three
datasets spanning code generation (WarriorCoder, Magpie-Qwen2.5-Coder-Pro-300K) and math
reasoning (OpenR1-Math-220k) demonstrate that selecting just 25% of data using stratified RMI matches full-data performance while being consistently competitive with or better than existing data selection methods. Our approach highlights the importance of bidirectional semantic coherence in synthetic data curation, offering a scalable pathway to reduce computational costs without sacrificing model capability. Code is available at https://github.com/XXSg559/QAQ.
Read the full abstract ▾
All authors Jiayin Lei, Ming Ma, Yunxi Duan, Chenxi Li, Tianming Yang
Computation and Language
LifeSim: Long-Horizon User Life Simulator for Personalized Assistant Evaluation
Feiyu Duan, Xuanjing Huang and Zhongyu Wei
The rapid advancement of large
language models (LLMs) has accelerated progress toward universal AI assistants. However, existing
benchmarks for personalized assistants remain misaligned with real-world user-assistant interactions, failing to capture the complexity of external contexts and users' cognitive states. To bridge this gap, we propose LifeSim, a user simulator that models user cognition through the Belief-Desire-Intention (BDI) model within physical environments for coherent life trajectories generation, and simulates intention-driven user interactive behaviors. Based on LifeSim, we introduce LifeSim-Eval, a comprehensive
benchmark for multi-scenario, long-horizon personalized assistance. LifeSim-Eval covers 8 life domains and 1,200 diverse scenarios, and adopts a multi-turn interactive method to assess models' abilities to complete explicit and implicit intentions, recover user profiles, and produce high-quality responses. Under both single-scenario and long-horizon settings, our experiments reveal that current LLMs face significant limitations in handling implicit intention and long-term user preference modeling.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.CV
Linking Perception, Confidence and Accuracy in MLLMs
Yuetian Du, Yucheng Wang, Rongyu Zhang et al. (+5 more)
Recent advances in Multi-modal Large
Language Models (MLLMs) have predominantly focused on enhancing visual perception to improve accuracy. However, a critical question remains unexplored: Do models know when they do not know? Through a probing experiment, we reveal a severe confidence miscalibration problem in MLLMs. To address this, we propose Confidence-Driven Reinforcement Learning (CDRL), which uses original-noise image pairs and a novel confidence-based reward to enhance perceptual sensitivity and robustly calibrate the model's confidence. Beyond training benefits, calibrated confidence enables more effective test-time scaling as a free lunch. We further propose Confidence-Aware Test-Time Scaling (CA-TTS), which dynamically coordinates Self-Consistency, Self-Reflection, and Visual Self-Check modules guided by confidence signals. An Expert Model acts in multiple roles (e.g., Planner, Critic, Voter) to schedule these modules and provide external verification. Our integrated framework establishes new state-of-the-art results with consistent 8.8% gains across four
benchmarks. More ablation studies demonstrate the effectiveness of each module and scaling superiority.
Read the full abstract ▾
All authors Yuetian Du, Yucheng Wang, Rongyu Zhang, Zhijie Xu, Boyu Yang, Ming Kong, Jie Liu, Qiang Zhu
Computation and Language · Cross-listed from cs.AI
TopoBench: Benchmarking LLMs on Hard Topological Reasoning
Mayug Maniparambil, Nils Hoehing, Janak Kapuriya et al. (+5 more)
Solving topological grid puzzles requires
reasoning over global spatial invariants such as connectivity, loop closure, and region
symmetry and remains challenging for even the most powerful large
language models (LLMs). To study these abilities under controlled settings, we introduce TopoBench, a
benchmark of six puzzle families across three difficulty levels. We evaluate strong
reasoning LLMs on TopoBench and find that even frontier models solve fewer than one quarter of hard instances, with two families nearly unsolved. To investigate whether these failures stem from
reasoning limitations or from difficulty extracting and maintaining spatial constraints, we annotate 750 chain of thought traces with an error taxonomy that surfaces four candidate causal failure modes, then test them with targeted interventions simulating each error type. These interventions show that certain error patterns like premature commitment and constraint forgetting have a direct impact on the ability to solve the puzzle, while repeated
reasoning is a benign effect of search. Finally we study mitigation strategies including
prompt guidance,
cell-aligned grid representations and tool-based constraint checking, finding that the bottleneck lies in extracting constraints from spatial representations and not in
reasoning over them. Code and data are available at github.com/mayug/topobench-
benchmark.
Read the full abstract ▾
All authors Mayug Maniparambil, Nils Hoehing, Janak Kapuriya, Arjun Karuvally, Ellen Rushe, Anthony Ventresque, Noel O'Connor, Fergal Reid
Computation and Language
Cross-Context Review: Improving LLM Output Quality by Separating Production and Review Sessions
Tae-Eun Song
Large
language models struggle to catch errors in their own outputs when the review happens in the same session that produced them. This paper introduces Cross-Context Review (CCR), a straightforward method where the review is conducted in a fresh session with no access to the production conversation history. We ran a controlled experiment: 30 artifacts (code, technical documents, presentation scripts) with 150 injected errors, tested under four review conditions – same-session Self-Review (SR), repeated Self-Review (SR2), context-aware Subagent Review (SA), and Cross-Context Review (CCR). The central result is that a second review helps only when it happens in a fresh session: CCR (F1 28.6%) outperforms a second review in the same session (SR2, 21.7%) robustly, both in the first run (paired t, p<0.001) and in the three-run average (Holm-adjusted p=0.004). This version updates the broader comparisons. Averaged across runs, and excluding one SR run whose records could not be verified, CCR is not significantly ahead of context-aware subagent review (SA, 23.8%; p=0.057) or of a single same-session review (SR, 27.1%; p=0.26); the first version's advantages over these two baselines came from run 1. CCR needs no infrastructure and costs one extra session.
Read the full abstract ▾
Computation and Language
SommBench: Assessing Sommelier Expertise of Language Models
William Brach, Tomas Bedej, Jacob Nielsen et al. (+10 more)
With the rapid advances of large
language models, it becomes increasingly important to systematically evaluate their
multilingual and multicultural capabilities. Previous cultural evaluation
benchmarks focus mainly on basic cultural knowledge that can be encoded in linguistic form. Here, we propose SommBench, a
multilingual benchmark to assess sommelier expertise, a domain deeply grounded in the senses of smell and taste. While
language models learn about sensory properties exclusively through textual descriptions, SommBench tests whether this textual grounding is sufficient to emulate expert-level sensory judgment. SommBench comprises three main tasks: Wine Theory Question Answering (WTQA), Wine Feature Completion (WFC), and Food-Wine Pairing (FWP). SommBench is available in multiple
languages: English, Slovak, Swedish, Finnish, German, Danish, Italian, and Spanish. This helps separate a
language model's wine expertise from its
language skills. The
benchmark datasets were developed in close collaboration with a professional sommelier and native speakers of the respective
languages, resulting in 1,024 wine theory question-answering questions, 1,000 wine feature-completion examples, and 1,000 food-wine pairing examples. We provide results for the most popular
language models, including closed-weights models such as Gemini 2.5, and open-weights models, such as GPT-OSS and Qwen 3. Our results show that the most capable models perform well on wine theory question answering (up to 97% correct with a closed-weights model), yet feature completion (peaking at 65%) and food-wine pairing show (MCC ranging between 0 and 0.39) turn out to be more challenging. These results position SommBench as an interesting and challenging
benchmark for evaluating the sommelier expertise of
language models. The
benchmark is publicly available at https://github.com/sommify/sommbench.
Read the full abstract ▾
All authors William Brach, Tomas Bedej, Jacob Nielsen, Jacob Pichna, Juraj Bedej, Eemeli Saarensilta, Julie Dupouy, Gianluca Barmina, Andrea Blasi Núñez, Peter Schneider-Kamp, Kristian Košťál, Michal Ries, Lukas Galke Poech
Computation and Language
To Words and Beyond: Probing Large Language Models for Sentence-Level Psycholinguistic Norms of Memorability and Reading Times
Thomas Hikaru Clark, Carlos Arriaga, Javier Conde et al. (+2 more)
Large
Language Models (LLMs) have recently been shown to produce estimates of psycholinguistic norms, such as valence, arousal, or concreteness, for words and multiword expressions, that correlate with human judgments. These estimates are obtained by prompting an LLM, in zero-shot fashion, with a question similar to those used in human studies. Meanwhile, for other norms such as lexical decision time or age of acquisition, LLMs require supervised fine-tuning to obtain results that align with ground-truth values. In this paper, we extend this approach to the previously unstudied features of sentence memorability and reading times, which involve the relationship between multiple words in a sentence-level context. Our results show that via fine-tuning, models can provide estimates that correlate with human-derived norms and exceed the predictive power of interpretable baseline predictors, demonstrating that LLMs contain useful information about sentence-level features. At the same time, our results show very mixed zero-shot and few-shot performance, providing further evidence that care is needed when using LLM-prompting as a proxy for human cognitive measures.
Read the full abstract ▾
All authors Thomas Hikaru Clark, Carlos Arriaga, Javier Conde, Gonzalo Martínez, Pedro Reviriego
Computation and Language · Cross-listed from cs.HC
Human-Centred LLM Privacy Audits: Findings and Frictions
Dimitri Staufer, Kirsten Morehouse, David Hartmann et al. (+1 more)
Large
language models (LLMs) learn statistical associations from massive training corpora and user interactions, and deployed systems can surface or infer information about individuals. Yet people lack practical ways to inspect what a model associates with their name. We report interim findings from an ongoing study and introduce LMP2, a browser-based self-audit tool. In two user studies ($N_{total}{=}458$), GPT-4o predicts 11 of 50 features for everyday people with $\ge$60% accuracy, and participants report wanting control over LLM-generated associations despite not considering all outputs
privacy violations. To validate our probing method, we evaluate eight LLMs on public figures and non-existent names, observing clear separation between stable name-conditioned associations and model defaults. Our findings also contribute to exposing a broader generative AI evaluation crisis: when outputs are probabilistic, context-dependent, and user-mediated through elicitation, what model–individual associations even include is under-specified and operationalisation relies on crafting probes and metrics that are hard to validate or compare. To move towards reliable, actionable human-centred LLM
privacy audits, we identify nine frictions that emerged in our study and offer recommendations for future work and the design of human-centred LLM
privacy audits.
Read the full abstract ▾
All authors Dimitri Staufer, Kirsten Morehouse, David Hartmann, Bettina Berendt
Computation and Language · Cross-listed from cs.AI
XSkill: Continual Learning from Experience and Skills in Multimodal Agents
Guanyu Jiang, Zhaochen Su, Xiaoye Qu et al. (+1 more)
Multimodal agents can now tackle complex
reasoning tasks with diverse tools, yet they still suffer from inefficient tool use and inflexible orchestration in open-ended settings. A central challenge is enabling such
agents to continually improve without parameter updates by learning from past trajectories. We identify two complementary forms of reusable knowledge essential for this goal: experiences, providing concise action-level guidance for tool selection and decision making, and skills, providing structured task-level guidance for planning and tool use. To this end, we propose XSkill, a dual-stream framework for continual learning from experience and skills in
multimodal agents. XSkill grounds both knowledge extraction and
retrieval in visual observations. During accumulation, XSkill distills and consolidates experiences and skills from multi-path rollouts via visually grounded summarization and cross-rollout critique. During inference, it retrieves and adapts this knowledge to the current visual context and feeds usage history back into accumulation to form a continual learning loop. Evaluated on five
benchmarks across diverse domains with four backbone models, XSkill consistently and substantially outperforms both tool-only and learning-based baselines. Further analysis reveals that the two knowledge streams play complementary roles in influencing the
reasoning behaviors of
agents and show superior zero-shot generalization.
Read the full abstract ▾
All authors Guanyu Jiang, Zhaochen Su, Xiaoye Qu, Yi R. Fung