Computation and Language · Cross-listed from cs.IR
Quantifying Divergence in Inter-LLM Communication Through API Retrieval and Ranking
Eyhab Al-Masri
Large
language models (LLMs) increasingly operate as autonomous
agents that reason over external APIs to perform complex tasks. However, their reliability and agreement remain poorly characterized. We present a unified benchmarking framework to quantify inter-LLM divergence, defined as the extent to which models differ in API discovery and ranking under identical tasks. Across 15 canonical API domains and 5 major model families, we measure pairwise and group-level agreement using set-, rank-, and consensus-based metrics including Average Overlap, Jaccard similarity, Rank-Biased Overlap, Kendall's tau, Kendall's W, and Cronbach's alpha. Results show moderate overall
alignment (AO about 0.50, tau about 0.45) but strong domain dependence: structured tasks (Weather,
Speech-to-Text) are stable, while open-ended tasks (Sentiment Analysis) exhibit substantially higher divergence.
Volatility and consensus analyses reveal that coherence clusters around data-bound domains and degrades for abstract
reasoning tasks. These insights enable reliability-aware orchestration in multi-
agent systems, where consensus weighting can improve coordination among heterogeneous LLMs. Beyond performance benchmarking, our results reveal systematic failure modes in multi-
agent LLM coordination, where apparent agreement can mask instability in action-relevant rankings. This hidden divergence poses a pre-deployment safety risk and motivates diagnostic
benchmarks for early detection.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.IR
Beyond Static: Related Questions Retrieval Through Conversations in Community Question Answering
Xiao Ao, Jie Zou, Yibiao Wei et al. (+2 more)
In community question answering (cQA) platforms like Stack Overflow, related question
retrieval is recognized as a fundamental task that allows users to retrieve related questions to answer user queries automatically. Although many traditional approaches have been proposed for investigating this research field, they mostly rely on static approaches and neglect the interaction property. We argue that the conversational way can well distinguish the fine-grained representations of questions and has great potential to improve the performance of question
retrieval. In this paper, we propose a related question
retrieval model through conversations, called TeCQR, to locate related questions in cQA. Specifically, we build conversations by utilizing tag-enhanced clarifying questions (CQs). In addition, we design a noise tolerance model that evaluates the semantic similarity between questions and tags, enabling the model to effectively handle noisy feedback. Moreover, the tag-enhanced two-stage offline training is proposed to fully exploit the mutual relationships among user queries, questions, and tags to learn their fine-grained representations. Based on the learned representations and contextual conversations, TeCQR incorporates conversational feedback by learning to ask tag-enhanced clarifying questions to retrieve related questions more effectively. Experimental results demonstrate that our model significantly outperforms state-of-the-art baselines.
Read the full abstract ▾
All authors Xiao Ao, Jie Zou, Yibiao Wei, Peng Wang, Weikang Guo
Computation and Language · Cross-listed from cs.AI
CID-TKG: Collaborative Historical Invariance and Evolutionary Dynamics Learning for Temporal Knowledge Graph Reasoning
Shuai-Long Lei, Xiaobin Zhu, Jiarui Liang et al. (+3 more)
Temporal knowledge graph (TKG)
reasoning aims to infer future facts at unseen timestamps from temporally evolving entities and relations. Despite recent progress, existing approaches still suffer from inherent limitations due to their inductive biases, as they predominantly rely on time-invariant or weakly time-dependent structures and overlook the evolutionary dynamics. To overcome this limitation, we propose a novel collaborative learning framework for TKGR (dubbed CID-TKG) that integrates evolutionary dynamics and historical invariance semantics as an effective inductive bias for
reasoning. Specifically, CID-TKG constructs a historical invariance graph to capture long-term structural regularities and an evolutionary dynamics graph to model short-term temporal transitions. Dedicated encoders are then employed to learn representations from each structure. To alleviate semantic discrepancies across the two structures, we decompose relations into view-specific representations and align view-specific query representations via a contrastive objective, which promotes cross-view consistency while suppressing view-specific noise. Extensive experiments verify that our CID-TKG achieves state-of-the-art performance under extrapolation settings.
Read the full abstract ▾
All authors Shuai-Long Lei, Xiaobin Zhu, Jiarui Liang, Guoxi Sun, Zhiyu Fang, Xu-Cheng Yin
Computation and Language
Can LLMs Perceive Time? An Empirical Investigation
Aniketh Garikaparthi
Large
language models cannot estimate how long their own tasks take. We investigate this limitation through four experiments across 68 tasks and four model families. Pre-task estimates overshoot actual duration by 4–7$\times$ ($p < 0.001$), with models predicting human-scale minutes for tasks completing in seconds. Relative ordering fares no better: on task pairs designed to expose heuristic reliance, models score at or below chance (GPT-5: 18% on counter-intuitive pairs, $p = 0.033$), systematically failing when complexity labels mislead. Post-hoc recall is disconnected from reality – estimates diverge from actuals by an order of magnitude in either direction. These failures persist in multi-step agentic settings, with errors of 5–10$\times$. The models possess propositional knowledge about duration from training but lack experiential grounding in their own inference time, with practical implications for
agent scheduling, planning and time-critical scenarios.
Read the full abstract ▾
Computation and Language
Eyla: Toward an Identity-Anchored LLM Architecture with Integrated Biological Priors – Vision, Implementation Attempt, and Lessons from AI-Assisted Development
Arif Aditto
We present the design rationale, implementation attempt, and failure analysis of Eyla, a proposed identity-anchored LLM architecture that integrates biologically-inspired subsystems – including HiPPO-initialized state-space models, zero-initialized adapters, episodic
memory retrieval, and calibrated uncertainty training – into a unified
agent operating system running on consumer hardware. Unlike existing approaches that optimize models for generic helpfulness, Eyla targets identity consistency: the ability to maintain a coherent self-model under
adversarial pressure, admit uncertainty, and resist manipulation. We propose the Identity Consistency Score (ICS), a novel
benchmark for evaluating this property across LLMs. We then present an honest account of attempting to implement this architecture using AI coding assistants (Claude Code, Cursor) as a non-programmer, documenting a $1,000+ failure that produced a 1.27B parameter model with 86
brain subsystems contributing less than 2% to output. Our analysis identifies five systematic failure modes of AI-assisted development for novel architectures and offers concrete recommendations. To our knowledge, this is the first paper to combine an architectural
vision with a documented first-person failure analysis of AI-assisted LLM development, providing lessons for both the AI systems and AI-assisted software engineering communities.
Read the full abstract ▾
Computation and Language
How Trustworthy Are LLM-as-Judge Ratings for Interpretive Responses? Implications for Qualitative Research Workflows
Songhee Han, Jueun Shin, Jiyoon Han et al. (+2 more)
As qualitative researchers show growing interest in using automated tools to support interpretive analysis, a large
language model (LLM) is often introduced into an analytic workflow as is, without systematic evaluation of interpretive quality or comparison across models. This practice leaves model selection largely unexamined despite its potential influence on interpretive outcomes. To address this gap, this study examines whether LLM-as-judge evaluations meaningfully align with human judgments of interpretive quality and can inform model-level decision making. Using 712 conversational excerpts from semi-structured interviews with K-12 mathematics teachers, we generated one-sentence interpretive responses using five widely adopted inference models: Command R+ (Cohere), Gemini 2.5 Pro (Google), GPT-5.1 (OpenAI), Llama 4 Scout-17B Instruct (Meta), and Qwen 3-32B Dense (Alibaba). Automated evaluations were conducted using AWS Bedrock's LLM-as-judge framework across five metrics, and a stratified subset of responses was independently rated by trained human evaluators on interpretive accuracy, nuance preservation, and interpretive coherence. Results show that LLM-as-judge scores capture broad directional trends in human evaluations at the model level but diverge substantially in score magnitude. Among automated metrics, Coherence showed the strongest
alignment with aggregated human ratings, whereas Faithfulness and Correctness revealed systematic misalignment at the excerpt level, particularly for non-literal and nuanced interpretations. Safety-related metrics were largely irrelevant to interpretive quality. These findings suggest that LLM-as-judge methods are better suited for screening or eliminating underperforming models than for replacing human judgment, offering practical guidance for systematic comparison and selection of LLMs in qualitative research workflows.
Read the full abstract ▾
All authors Songhee Han, Jueun Shin, Jiyoon Han, Bung-Woo Jun, Hilal Ayan Karabatman
Computation and Language
Dynin-Omni: Omnimodal Unified Large Diffusion Language Model
Jaeik Kim, Woojin Kim, Jihwan Hong et al. (+8 more)
We present Dynin-Omni, the first masked-
diffusion-based omnimodal foundation model that unifies text, image, and
speech understanding and generation, together with
video understanding, within a single architecture. Unlike autoregressive unified models that serialize heterogeneous modalities, or compositional unified models that require orchestration with external modality-specific decoders, Dynin-Omni natively formulates omnimodal modeling as masked
diffusion over a shared discrete
token space, enabling iterative refinement under bidirectional context. Dynin-Omni adopts a multi-stage training strategy with model-merging-based modality expansion and omnimodal
alignment. We evaluate Dynin-Omni across 19
multimodal benchmarks spanning
language reasoning, image generation and editing,
video understanding, and
speech recognition and synthesis. Dynin-Omni achieves 87.6 on GSM8K, 1733.6 on MME-P, 61.4 on VideoMME, 0.87 on GenEval, and 2.1 WER on LibriSpeech test-clean, consistently outperforming existing open-source unified models while remaining competitive with strong modality-specific expert systems. These results demonstrate the potential of masked
diffusion as a unified paradigm for any-to-any modeling, providing a flexible foundation for real-time omnimodal systems, unified cross-modal
retrieval and generation, and
embodied multimodal agents.
Read the full abstract ▾
All authors Jaeik Kim, Woojin Kim, Jihwan Hong, Yejoon Lee, Sieun Hyeon, Mintaek Lim, Yunseok Han, Dogeun Kim, Hoeun Lee, Hyunggeun Kim, Jaeyoung Do
Computation and Language
Scalable Identification and Prioritization of Requisition-Specific Personal Competencies Using Large Language Models
Wanxin Li, Denver McNeney, Nivedita Prabhu et al. (+5 more)
AI-powered recruitment tools are increasingly adopted in personnel selection, yet they struggle to capture the requisition (req)-specific personal competencies (PCs) that distinguish successful candidates beyond job categories. We propose a large
language model (LLM)-based approach to identify and prioritize req-specific PCs from reqs. Our approach integrates dynamic few-shot prompting, reflection-based self-improvement, similarity-based filtering, and multi-stage validation. Applied to a
dataset of Program Manager reqs, our approach correctly identifies the highest-priority req-specific PCs with an average accuracy of 0.76, approaching human expert inter-rater reliability, and maintains a low out-of-scope rate of 0.07.
Read the full abstract ▾
All authors Wanxin Li, Denver McNeney, Nivedita Prabhu, Charlene Zhang, Renee Barr, Matthew Kitching, Khanh Dao Duc, Anthony S. Boyce
Computation and Language · Cross-listed from cs.AI
How Emotion Shapes the Behavior of LLMs and Agents: A Mechanistic Study
Moran Sun, Tianlin Li, Yuwei Zheng et al. (+4 more)
Emotion plays an important role in human cognition and performance. Motivated by this, we investigate whether analogous emotional signals can shape the behavior of large
language models (LLMs) and
agents. Existing
emotion-aware studies mainly treat
emotion as a surface-level style factor or a perception target, overlooking its mechanistic role in task processing. To address this limitation, we propose E-STEER, an interpretable
emotion steering framework that enables direct representation-level intervention in LLMs and
agents. It embeds
emotion as a structured, controllable variable in hidden states, and with it, we examine the impact of
emotion on objective
reasoning, subjective generation, safety, and multi-step
agent behaviors. The results reveal non-monotonic
emotion-behavior relations consistent with established psychological theories, and show that specific
emotions not only enhance LLM capability but also improve safety, and systematically shape multi-step
agent behaviors.
Read the full abstract ▾
All authors Moran Sun, Tianlin Li, Yuwei Zheng, Zhenhong Zhou, Aishan Liu, Xianglong Liu, Yang Liu
Computation and Language
Ning Yang, Hengyu Zhong, Wentao Wang et al. (+4 more)
The extension of context windows in Large
Language Models is typically facilitated by scaling positional encodings followed by Continued Pre-Training (CPT). While effective, this paradigm is notoriously data-hungry and computationally expensive, requiring massive long-text corpora to recalibrate the model to the shifted positional distribution. We propose LinearARD, a restoration
distillation method that restores Rotary Position
Embedding (RoPE)-scaled students through
attention-structure consistency with a frozen native-RoPE teacher. Rather than next-
token prediction or opaque hidden-state matching, LinearARD aligns row-wise distributions of dense $Q/Q$, $K/K$, and $V/V$ self-relation
matrices from the final
attention layer to directly supervise
attention dynamics. To remove the quadratic
memory bottleneck of $n \times n$ relation maps, we introduce a linear-activation-
memory kernel that stores only per-
token log-sum-exp statistics and recomputes logits in the backward pass to obtain exact Kullback-Leibler divergence gradients. Across LLaMA2-7B, LLaMA3-8B, and Mistral-7B-v0.1 extended to 32K context, LinearARD recovers 93.1%/94.2%/94.3% of native short-context performance and achieves strong long-context robustness on RULER using only 4.25M
tokens—amounting to just 1.6% (a $\sim$60$\times$ reduction) of the 256M-
token budget required by state-of-the-art baselines. Under this severely constrained budget, CPT and LongReD remain near zero on RULER, demonstrating that relation-level restoration provides a substantially better efficiency-quality tradeoff. Our code is available at https://github.com/gracefulning/LinearARD.
Read the full abstract ▾
All authors Ning Yang, Hengyu Zhong, Wentao Wang, Baoliang Tian, Yuan Zhou, Haijun Zhang, Jun Wang
Computation and Language
Tabular PDF Information Extraction with Local LLMs and Layout-Aware Parsing: A Reliability Evaluation
Muhammad Anis Al Hilmi, Neelansh Khare, Noel Framil Iglesias et al. (+3 more)
Extracting structured information from academic PDF documents is non trivial: a single page typically combines free text metadata with tabular regions, exhibits cross program variation, and is susceptible to Unicode encoding artifacts that interfere with downstream parsing. This study evaluates the reliability of information extraction approaches for tabular PDF documents, using academic course registration documents (Kartu Rencana Studi or KRS) from Indonesian higher education as a case study. Three strategies are compared: LLM only, Hybrid Deterministic - LLM (regex & LLM), and a Camelot based pipeline with LLM fallback. Experiments were conducted on 140 documents for the LLM based test and 860 documents for the Camelot based pipeline evaluation, covering four study programs with varying data in tables and metadata. Three 12 - 14B LLM models (Gemma 3, Phi 4, and Qwen 2.5) were run locally using Ollama and a consumer grade CPU without a GPU. Evaluations used exact match (EM) and Levenshtein similarity (LS) metrics with a threshold of 0.7. Although not applicable to all models, the results show that the hybrid approach can improve efficiency compared to LLM only, especially for deterministic metadata. The Camelot based pipeline with LLM fallback produced the best combination of accuracy (EM and LS up to 0.99 - 1.00) and computational efficiency (less than 1 second per PDF in most cases). The Qwen 2.5:14b model demonstrated the most consistent performance across all scenarios. These findings confirm that integrating deterministic and LLM based methods is a reliable and efficient strategy for information extraction from tabular text based PDF documents in computationally constrained environments.
Read the full abstract ▾
All authors Muhammad Anis Al Hilmi, Neelansh Khare, Noel Framil Iglesias, Kurnia Adi Cahyanto, Azhar Al Afghani, Musfi Yuliadi
Computation and Language
AlpsBench: An LLM Personalization Benchmark for Real-Dialogue Memorization and Preference Alignment
Jianfei Xiao, Xiang Yu, Chengbing Wang et al. (+8 more)
As Large
Language Models (LLMs) evolve into lifelong AI assistants, LLM personalization has become a critical frontier. However, progress is currently bottlenecked by the absence of a gold-standard evaluation
benchmark. Existing
benchmarks either overlook personalized information management that is critical for personalization or rely heavily on synthetic dialogues, which exhibit an inherent distribution gap from real-world dialogue. To bridge this gap, we introduce AlpsBench, An LLM PerSonalization
benchmark derived from real-world human-LLM dialogues. AlpsBench comprises 2,500 long-term interaction sequences curated from WildChat, paired with human-verified structured
memories that encapsulate both explicit and implicit personalization signals. We define four pivotal tasks - personalized information extraction, updating,
retrieval, and utilization - and establish protocols to evaluate the entire lifecycle of
memory management. Our benchmarking of frontier LLMs and
memory-centric systems reveals that: (i) models struggle to reliably extract latent user traits; (ii)
memory updating faces a performance ceiling even in the strongest models; (iii)
retrieval accuracy declines sharply in the presence of large distractor pools; and (iv) while explicit
memory mechanisms improve recall, they do not inherently guarantee more preference-aligned or emotionally resonant responses. AlpsBench aims to provide a comprehensive framework.
Read the full abstract ▾
All authors Jianfei Xiao, Xiang Yu, Chengbing Wang, Wuqiang Zheng, Xinyu Lin, Kaining Liu, Hongxun Ding, Yang Zhang, Wenjie Wang, Fuli Feng, Xiangnan He
Computation and Language · Cross-listed from cs.CR
Tool Receipts, Not Zero-Knowledge Proofs: Practical Hallucination Detection for AI Agents
Abhinaba Basu
AI
agents that execute tasks via tool calls frequently hallucinate results - fabricating tool executions, misstating output counts, or presenting inferences as facts. Recent approaches to verifiable AI inference rely on zero-knowledge proofs, which provide cryptographic guarantees but impose minutes of proving time per query, making them impractical for interactive
agents. We propose NabaOS, a lightweight verification framework inspired by Indian epistemology (Nyaya Shastra), which classifies every claim in an LLM response by its epistemic source (pramana): direct tool output (pratyaksha), inference (anumana), external testimony (shabda), absence (abhava), or ungrounded opinion. Our runtime generates HMAC-signed tool execution receipts that the LLM cannot forge, then cross-references claims against these receipts to detect
hallucinations in real time. We evaluate on NyayaVerifyBench, a new
benchmark of 1,800
agent response scenarios across four
languages with injected
hallucinations of six types. NabaOS detects 94.2% of fabricated tool references, 87.6% of count misstatements, and 91.3% of false absence claims, with <15ms verification overhead per response. For deep delegation (
agents performing multi-step web tasks), our cross-checking protocol catches 78.4% of URL fabrications via independent re-fetching. We compare against five approaches: zkLLM (cryptographic proofs, 180s/query), TOPLOC (locality-sensitive hashing), SPEX (sampling-based proof of execution), tensor commitments, and self-consistency checking. NabaOS achieves the best cost-latency-coverage trade-off for interactive
agents: 94.2% coverage at <15ms versus zkLLM's near-perfect coverage at 180,000ms. For interactive
agents, practical receipt-based verification provides better cost-benefit than cryptographic proofs, and epistemic classification gives users actionable trust signals rather than binary judgments.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.LG
Training Language Models via Neural Cellular Automata
Dan Lee, Seungwook Han, Akarsh Kumar et al. (+1 more)
Pre-training is crucial for large
language models (LLMs), as it is when most representations and capabilities are acquired. However, natural
language pre-training has problems: high-quality text is finite, it contains human biases, and it entangles knowledge with
reasoning. This raises a fundamental question: is natural
language the only path to intelligence? We propose using neural cellular automata (NCA) to generate synthetic, non-linguistic data for pre-pre-training LLMs–training on synthetic-then-natural
language. NCA data exhibits rich spatiotemporal structure and statistics resembling natural
language while being controllable and cheap to generate at scale. We find that pre-pre-training on only 164M NCA
tokens improves downstream
language modeling by up to 6% and accelerates convergence by up to 1.6x. Surprisingly, this even outperforms pre-pre-training on 1.6B
tokens of natural
language from Common Crawl with more compute. These gains also transfer to
reasoning benchmarks, including GSM8K, HumanEval, and BigBench-Lite. Investigating what drives transfer, we find that
attention layers are the most transferable, and that optimal NCA complexity varies by domain: code benefits from simpler dynamics, while math and web text favor more complex ones. These results enable systematic tuning of the synthetic distribution to target domains. More broadly, our work opens a path toward more efficient models with fully synthetic pre-training.
Read the full abstract ▾
All authors Dan Lee, Seungwook Han, Akarsh Kumar, Pulkit Agrawal
Computation and Language
Learning When to Sample: Confidence-Aware Selective Sampling for Efficient Chain-of-Thought Reasoning
Juming Xiong, Kevin Guo, Congning Ni et al. (+7 more)
Large
language models (LLMs) can achieve strong
reasoning performance through chain-of-thought (CoT)
reasoning, yet they often generate unnecessarily long
reasoning paths that incur high inference cost. Self-consistency-based approaches push accuracy higher still, but they require sampling and aggregating multiple
reasoning trajectories, leading to substantial computational overhead. In this paper, we introduce a confidence-aware selective sampling framework that, at inference time, analyzes a single
reasoning trajectory to adaptively determine whether to rely on that trajectory alone or trigger multi-path sampling. The framework uses trajectory-level numeric features and sentence-level linguistic features extracted from
reasoning states to guide selective multi-path
reasoning. We train it on MedQA and evaluate it in-domain on MedQA and under calibration-only transfer on MathQA, MedMCQA, and MMLU, without further fine-tuning. Experimental results show that the proposed framework maintains comparable performance to full and efficient multi-path
reasoning baselines, with accuracy changes of $-0.41 \pm 0.58$ and $-0.31 \pm 0.58$ percentage points, respectively, while reducing
token usage by $71.7 \pm 5.0%$ and $36.6 \pm 9.1%$. These findings demonstrate that
reasoning trajectories contain rich signals for uncertainty estimation, enabling a simple, transferable mechanism to balance accuracy and efficiency in LLM
reasoning.
Read the full abstract ▾
All authors Juming Xiong, Kevin Guo, Congning Ni, Weixin Liu, Chao Yan, Katherine Brown, Avinash Baidya, Xiang Gao, Bradley Malin, Zhijun Yin
Computation and Language
Automated Thematic Analysis for Clinical Qualitative Data: Iterative Codebook Refinement with Full Provenance
Seungjun Yi, Joakim Nguyen, Huimin Xu et al. (+9 more)
Thematic analysis (TA) is widely used in health research to extract patterns from patient interviews, yet manual TA faces challenges in scalability and reproducibility. LLM-based automation can help, but existing approaches produce codebooks with limited generalizability and lack analytic auditability. We present an automated TA framework combining iterative codebook refinement with full provenance tracking. Evaluated on five corpora spanning clinical interviews, social media, and public transcripts, the framework achieves the highest composite quality score on four of five
datasets compared to six baselines. Iterative refinement yields statistically significant improvements on four
datasets with large effect sizes, driven by gains in code reusability and distributional consistency while preserving descriptive quality. On two clinical corpora (pediatric cardiology), generated themes align with expert-annotated themes.
Read the full abstract ▾
All authors Seungjun Yi, Joakim Nguyen, Huimin Xu, Terence Lim, Joseph Skrovan, Mehak Beri, Hitakshi Modi, Andrew Well, Carlos M. Mery, Yan Zhang, Mia K. Markey, Ying Ding
Computation and Language · Cross-listed from cs.AI
A Consensus-Driven Multi-LLM Pipeline for Missing-Person Investigations
Joshua Castillo and Ravi Mukkamala
The first 72 hours of a missing-person investigation are critical for successful recovery. Guardian is an end-to-end system designed to support missing-child investigation and early search planning. This paper presents the Guardian LLM Pipeline, a multi-model system in which LLMs are used for intelligent information extraction and processing related to missing-person search operations. The pipeline coordinates end-to-end execution across task-specialized LLM models and invokes a consensus LLM engine that compares multiple model outputs and resolves disagreements. The pipeline is further strengthened by QLoRA-based fine-tuning, using curated
datasets. The presented design aligns with prior work on weak supervision and LLM-assisted annotation, emphasizing conservative, auditable use of LLMs as structured extractors and labelers rather than unconstrained end-to-end decision makers.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.CV
BiCLIP: Domain Canonicalization via Structured Geometric Transformation
Pranav Mantini and Shishir K. Shah
Recent advances in
vision-
language models (VLMs) have demonstrated remarkable zero-shot capabilities, yet adapting these models to specialized domains remains a significant challenge. Building on recent theoretical insights suggesting that independently trained VLMs are related by a canonical transformation, we extend this understanding to the concept of domains. We hypothesize that image features across disparate domains are related by a canonicalized
geometric transformation that can be recovered using a small set of anchors. Few-shot classification provides a natural setting for this
alignment, as the limited labeled samples serve as the anchors required to estimate this transformation. Motivated by this hypothesis, we introduce BiCLIP, a framework that applies a targeted transformation to
multimodal features to enhance cross-modal
alignment. Our approach is characterized by its extreme simplicity and low parameter footprint. Extensive evaluations across 11 standard
benchmarks, including EuroSAT, DTD, and FGVCAircraft, demonstrate that BiCLIP consistently achieves state-of-the-art results. Furthermore, we provide empirical verification of existing
geometric findings by analyzing the orthogonality and angular distribution of the learned transformations, confirming that structured
alignment is the key to robust domain adaptation. Code is available at https://github.com/QuantitativeImagingLaboratory/BilinearCLIP
Read the full abstract ▾
Computation and Language · Cross-listed from cs.SD
VoxEmo: Benchmarking Speech Emotion Recognition with Speech LLMs
Hezhao Zhang, Huang-Cheng Chou, Shrikanth Narayanan et al. (+1 more)
Speech Large
Language Models (LLMs) show great promise for
speech emotion recognition (SER) via generative interfaces. However, shifting from closed-set classification to open text generation introduces zero-shot stochasticity, making evaluation highly sensitive to
prompts. Additionally, conventional
speech LLMs
benchmarks overlook the inherent ambiguity of human
emotion. Hence, we present VoxEmo, a comprehensive SER
benchmark encompassing 35
emotion corpora across 15
languages for
Speech LLMs. VoxEmo provides a standardized toolkit featuring varying
prompt complexities, from direct classification to paralinguistic
reasoning. To reflect real-world perception/application, we introduce a distribution-aware soft-label protocol and a
prompt-ensemble strategy that emulates annotator disagreement. Experiments reveal that while zero-shot
speech LLMs trail supervised baselines in hard-label accuracy, they uniquely align with human subjective distributions.
Read the full abstract ▾
All authors Hezhao Zhang, Huang-Cheng Chou, Shrikanth Narayanan, Thomas Hain
Computation and Language · Cross-listed from cs.CV
PathoScribe: Transforming Pathology Data into a Living Library with a Unified LLM-Driven Framework for Semantic Retrieval and Clinical Integration
Abdul Rehman Akbar, Samuel Wales-McGrath, Alejadro Levya et al. (+5 more)
Pathology underpins modern diagnosis and cancer care, yet its most valuable asset, the accumulated experience encoded in millions of narrative reports, remains largely inaccessible. Although institutions are rapidly digitizing pathology workflows, storing data without effective mechanisms for
retrieval and
reasoning risks transforming archives into a passive data repository, where institutional knowledge exists but cannot meaningfully inform patient care. True progress requires not only digitization, but the ability for pathologists to interrogate prior similar cases in real time while evaluating a new diagnostic dilemma. We present PathoScribe, a unified
retrieval-augmented large
language model (LLM) framework designed to transform static pathology archives into a searchable,
reasoning-enabled living library. PathoScribe enables natural
language case exploration, automated cohort construction, clinical question answering, immunohistochemistry (IHC) panel recommendation, and
prompt-controlled report transformation within a single architecture. Evaluated on 70,000 multi-institutional
surgical pathology reports, PathoScribe achieved perfect Recall@10 for natural
language case
retrieval and demonstrated high-quality
retrieval-grounded
reasoning (mean reviewer score 4.56/5). Critically, the system operationalized automated cohort construction from free-text eligibility criteria, assembling research-ready cohorts in minutes (mean 9.2 minutes) with 91.3% agreement to human reviewers and no eligible cases incorrectly excluded, representing orders-of-magnitude reductions in time and cost compared to traditional manual chart review. This work establishes a scalable foundation for converting digital pathology archives from passive storage systems into active clinical intelligence platforms.
Read the full abstract ▾
All authors Abdul Rehman Akbar, Samuel Wales-McGrath, Alejadro Levya, Lina Gokhale, Rajendra Singh, Wei Chen, Anil Parwani, Muhammad Khalid Khan Niazi
Computation and Language
SciTaRC: A Plan-Annotated Scientific Tabular QA Benchmark for Language Reasoning and Complex Computation
Hexuan Wang, Yaxuan Ren, Srikar Bommireddypalli et al. (+5 more)
We introduce SciTaRC, an expert-authored
benchmark for question answering over scientific tables that targets composite, multi-step
reasoning. To enable fine-grained diagnostic analysis beyond end-task accuracy, SciTaRC pairs each question with a manually constructed
reasoning plan and explicit complexity metrics. State-of-the-art models fail on at least 23% of these questions, while highly capable open-weight models like Llama-3.3-70B collapse on 65.5% of the
benchmark. Error analysis shows that, in zero-shot settings, failures are driven primarily by question comprehension, where models misinterpret the scientific query and derive the wrong
reasoning objective. To determine whether overcoming this gap is sufficient, we use the structured plans to decouple strategy formulation from execution. Surprisingly, providing oracle step-by-step plans yields only limited gains and fails to eliminate the performance gap. This reveals a substantial execution bottleneck: both natural
language and code-based methods struggle to reliably carry out long-horizon computational chains over structured data. Ultimately, SciTaRC serves as a rigorous diagnostic testbed for studying both planning and execution in scientific table
reasoning.
Read the full abstract ▾
All authors Hexuan Wang, Yaxuan Ren, Srikar Bommireddypalli, Shuxian Chen, Adarsh Prabhudesai, Rongkun Zhou, Elina Baral, Philipp Koehn
Computation and Language
ConFu: Contemplate the Future for Better Speculative Sampling
Zongyue Qin, Raghavv Goel, Mukul Gagrani et al. (+3 more)
Speculative decoding has emerged as a powerful approach to accelerate large
language model (LLM) inference by employing lightweight draft models to propose candidate
tokens that are subsequently verified by the target model. The effectiveness of this paradigm critically depends on the quality of the draft model. While recent advances such as the EAGLE series achieve state-of-the-art speedup, existing draft models remain limited by error accumulation: they condition only on the current prefix, causing their predictions to drift from the target model over steps. In this work, we propose ConFu (Contemplate the Future), a novel speculative decoding framework that enables draft models to anticipate the future direction of generation. ConFu introduces (i) contemplate
tokens and soft
prompts that allow the draft model to leverage future-oriented signals from the target model at negligible cost, (ii) a dynamic contemplate
token mechanism with MoE to enable context-aware future prediction, and (iii) a training framework with anchor
token sampling and future prediction replication that learns robust future prediction. ConFu improves
token acceptance rates and generation speed over EAGLE-3 by 8–11% on Llama-3 3B/8B and by approximately 20% on Qwen-3 4B across downstream tasks. We believe our work is the first to bridge speculative decoding with continuous
reasoning tokens, offering a new direction for accelerating LLM inference.
Read the full abstract ▾
All authors Zongyue Qin, Raghavv Goel, Mukul Gagrani, Risheek Garrepalli, Mingu Lee, Yizhou Sun
Computation and Language · Cross-listed from cond-mat.mtrl-sci
From Word2Vec to Transformers: Text-Derived Composition Embeddings for Filtering Combinatorial Electrocatalysts
Lei Zhang and Markus Stricker
Compositionally complex solid solution electrocatalysts span vast composition spaces, and even one materials system can contain more candidate compositions than can be measured exhaustively. Here we evaluate a label-free screening strategy that represents each composition using
embeddings derived from scientific texts and prioritizes candidates based on similarity to two property concepts. We compare a corpus-trained Word2Vec baseline with
transformer-based
embeddings, where compositions are encoded either by linear element-wise mixing or by short composition
prompts. Similarities to `concept directions', the terms conductivity and dielectric, define a 2-dimensional descriptor space, and a symmetric Pareto-front selection is used to filter candidate subsets without using electrochemical labels. Performance is assessed on 15 materials libraries including noble metal
alloys and multicomponent
oxides. In this setting, the lightweight Word2Vec baseline, which uses a simple linear combination of element
embeddings, often achieves the highest number of reductions of possible candidate compositions while staying close to the best measured performance.
Read the full abstract ▾
Computation and Language
MultiGraSCCo: A Multilingual Anonymization Benchmark with Annotations of Personal Identifiers
Ibrahim Baroud, Christoph Otto, Vera Czehmann et al. (+4 more)
Accessing sensitive patient data for machine learning is challenging due to
privacy concerns.
Datasets with annotations of personally identifiable information are crucial for developing and testing anonymization systems to enable safe data sharing that complies with
privacy regulations. Since accessing real patient data is a bottleneck, synthetic data offers an efficient solution for data scarcity, bypassing
privacy regulations that apply to real data. Moreover, neural machine translation can help to create high-quality data for low-resource
languages by translating validated real or synthetic data from a high-resource
language. In this work, we create a
multilingual anonymization
benchmark in ten
languages, using a machine translation methodology that preserves the original annotations and renders names of cities and people in a culturally and contextually appropriate form in each target
language. Our evaluation study with medical professionals confirms the quality of the translations, both in general and with respect to the translation and adaptation of personal information. Our
benchmark with over 2,500 annotations of personal information can be used in many applications, including training annotators, validating annotations across institutions without legal complications, and helping improve the performance of automatic personal information detection. We make our
benchmark and annotation guidelines available for further research.
Read the full abstract ▾
All authors Ibrahim Baroud, Christoph Otto, Vera Czehmann, Christine Hovhannisyan, Lisa Raithel, Sebastian Möller, Roland Roller
Computation and Language
One Language, Two Scripts: Probing Script-Invariance in LLM Concept Representations
Sripad Karne
Do the features learned by Sparse Autoencoders (SAEs) represent abstract meaning, or are they tied to how text is written? We investigate this question using Serbian digraphia as a controlled testbed: Serbian is written interchangeably in Latin and Cyrillic scripts with a near-perfect character mapping between them, enabling us to vary orthography while holding meaning exactly constant. Crucially, these scripts are tokenized completely differently, sharing no
tokens whatsoever. Analyzing SAE feature activations across the Gemma model family (270M-27B parameters), we find that identical sentences in different Serbian scripts activate highly overlapping features, far exceeding random baselines. Strikingly, changing script causes less representational divergence than paraphrasing within the same script, suggesting SAE features prioritize meaning over orthographic form. Cross-script cross-paraphrase comparisons provide evidence against memorization, as these combinations rarely co-occur in training data yet still exhibit substantial feature overlap. This script invariance strengthens with model scale. Taken together, our findings suggest that SAE features can capture semantics at a level of abstraction above surface tokenization, and we propose Serbian digraphia as a general evaluation paradigm for probing the abstractness of learned representations.
Read the full abstract ▾