Computation and Language · Cross-listed from cs.CV
When Relations Break: Analyzing Relation Hallucination in Vision-Language Model Under Rotation and Noise
Philip Wootaek Shin, Ajay Narayanan Sridhar, Sivani Devarapalli et al. (+3 more)
Vision-
language models (VLMs) achieve strong
multimodal performance but remain prone to relation
hallucination, which requires accurate
reasoning over inter-object interactions. We study the impact of visual perturbations, specifically rotation and noise, and show that even mild distortions significantly degrade relational
reasoning across models and
datasets. We further evaluate
prompt-based augmentation and preprocessing strategies (orientation correction and denoising), finding that while they offer partial improvements, they do not fully resolve
hallucinations. Our results reveal a gap between perceptual robustness and relational understanding, highlighting the need for more robust, geometry-aware VLMs.
Read the full abstract ▾
All authors Philip Wootaek Shin, Ajay Narayanan Sridhar, Sivani Devarapalli, Rui Zhang, Jack Sampson, Vijaykrishnan Narayanan
Computation and Language
Gijs van Dijk
We propose a lightweight and single-pass uncertainty quantification method for detecting
hallucinations in Large
Language Models. The method uses
attention matrices to estimate uncertainty without requiring repeated sampling or external models. Specifically, we measure the Kullback-Leibler divergence between each
attention head's distribution and a uniform reference distribution, and use these features in a logistic regression probe. Across multiple
datasets, task types, and model families,
attention divergence is highly predictive of answer correctness and performs competitively with existing uncertainty estimation methods. We find that this signal is concentrated in middle layers and on factual
tokens such as named entities and numbers, suggesting that
attention dynamics provides an efficient and interpretable white-box signal of model uncertainty.
Read the full abstract ▾
Computation and Language
Better Reward Models Are Not Always Less Biased: Social Alignment Beyond Standard Reward Benchmarks
Gayane Ghazaryan and Esra Dönmez
Reward models are central to LLM
alignment, yet current
benchmarks reveal little about how they represent diverse and sometimes competing social preferences. We extend reward model evaluation to four socially consequential domains: bias, safety, morality, and ethical
reasoning. Our analysis shows that strong
benchmark performance can obscure important differences in social behavior. Models may avoid some stereotypes while reinforcing others, and aggregate bias scores can hide these opposing tendencies. We also find that stronger stereotype avoidance can come at the expense of contextual faithfulness, revealing a trade-off between social bias mitigation and sensitivity to context. Across a broad set of reward models, standard
benchmark performance is only inconsistently related to socially aligned behavior. These results show that stronger reward models are not uniformly better at navigating pluralistic social expectations and motivate more direct evaluation of the social preferences encoded in reward models.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.LG
Self-Induced Outcome Potential: Turn-Level Credit Assignment for Agents without Verifiers
Senkang Hu, Yong Dai, Xudong Han et al. (+4 more)
Long-horizon LLM
agents depend on intermediate information-gathering turns, yet training feedback is usually observed only at the final answer, because process-level rewards require high-quality human annotation. Existing turn-level shaping methods reward turns that increase the likelihood of a gold answer, but they require answer supervision or stable task-specific verifiers. Conversely, label-free RL methods extract self-signals from output distributions, but mainly at the answer or trajectory level and therefore cannot assign credit to intermediate turns. We propose Self-Induced Outcome Potential (SIOP), which treats semantic clusters of final answers as latent future outcome states for potential-based turn-level credit assignment. For each query, SIOP samples multiple rollouts, clusters final answers into semantic outcome modes, and builds a reliability-aware target distribution over these states. It then rewards turns for increasing posterior support for reliable future states using a tractable cluster-level approximation. The objective generalizes information-potential shaping from gold-answer supervision to settings without task-specific gold verifiers while avoiding the broadcasted rollout-level advantages used by standard GRPO. We formalize the framework, characterize its supervised gold-answer limit, and show that SIOP improves average performance over verifier-free outcome-level baselines on seven search-augmented agentic
reasoning benchmarks while approaching a gold-supervised outcome baseline. Code is available at https://github.com/dl-m9/SIOP.git.
Read the full abstract ▾
All authors Senkang Hu, Yong Dai, Xudong Han, Zhengru Fang, Yuzhi Zhao, Sam Tak Wu Kwong, Yuguang Fang
Computation and Language · Cross-listed from cs.LG
Conceptors for Semantic Steering
Ilias Triantafyllopoulos, Young-Min Cho, Ren Tao et al. (+6 more)
Activation-based steering provides control of LLM behavior at inference time, but the dominant paradigm reduces each concept to a single direction whose geometry is left largely unexamined. Rather than selecting a single steering direction, we use conceptors: soft projection
matrices estimated from activations pooled across both poles of a bipolar concept, which preserve the concept's full multidimensional subspace. A
geometric analysis shows the bipolar subspace strictly subsumes the single-vector baseline. We further show that the conceptor quota provides a parameter-free layer-selection diagnostic, predicting concept separability with Pearson correlations up to r=0.96 across three instruction-tuned models and three semantic dimensions. Beyond selection, conceptors admit a closed-form Boolean
algebra (AND, OR, NOT): we evaluate conceptor compositionality on thematically related sub-concepts. Across a systematic five-axis design-space evaluation, conceptors match or outperform additive baselines at layers where concept subspaces are multi-dimensional while producing substantially fewer degenerate outputs. Conceptor steering is a geometrically principled, compositional, and practically safer alternative to single-direction steering from a limited number of contrastive pairs.
Read the full abstract ▾
All authors Ilias Triantafyllopoulos, Young-Min Cho, Ren Tao, Miranda Muqing Miao, Sunny Rai, Lyle Ungar, Sharath Chandra Guntuku, Neville Ryant, João Sedoc
Computation and Language
Why Expert Alignment Is Hard: Evidence from Subjective Evaluation
Tzu-Mi Lin, Wataru Hirota, Tatsuya Ishigaki et al. (+2 more)
Aligning large
language models with expert judgment is especially difficult in subjective evaluation tasks, where experts may disagree, rely on tacit criteria, and change their judgments over time. In this paper, we study expert
alignment as a way to understand this difficulty. Using expert evaluations and follow-up questionnaires, we examine how different forms of expert information affect
alignment and what this reveals about subjective judgment. Our findings show four consistent patterns. First,
alignment difficulty varies substantially across experts, suggesting that expert evaluation styles differ widely in their distance from a model's prior behavior. Second, explicit criteria and
reasoning do not always improve
alignment, indicating that expert judgment is not fully captured by verbalized rules. Third, editing is sensitive to both the number and the identity of examples, with small numbers of edits providing useful but unstable gains. Fourth,
alignment difficulty differs across evaluation dimensions: dimensions grounded more directly in proposal content are easier to align, while dimensions requiring external knowledge or value-based judgment remain harder. Taken together, these results suggest that expert
alignment is difficult not only because of model limitations, but also because subjective evaluation is inherently heterogeneous, partly tacit, dimension-dependent, and temporally unstable.
Read the full abstract ▾
All authors Tzu-Mi Lin, Wataru Hirota, Tatsuya Ishigaki, Lung-Hao Lee, Chung-Chi Chen
Computation and Language · Cross-listed from cs.LG
Why Geometric Continuity Emerges in Deep Neural Networks: Residual Connections and Rotational Symmetry Breaking
Kyungwon Jeong, Won-Gi Paeng and Honggyo Suh
Weight
matrices in deep networks exhibit
geometric continuity—principal singular vectors of adjacent layers point in similar directions. While this property has been widely observed, its origin remains unexplained. Through experiments on toy MLPs and small
transformers, we identify two mechanisms: residual connections create cross-layer gradient coherence that aligns weight updates across layers, and
symmetry-breaking nonlinearities constrain all layers to a shared coordinate frame, preventing the rotation drift that would otherwise destabilize weight structure. Crucially, a nonlinear but rotation-equivariant activation reaches less than half of ReLU's continuity, isolating
symmetry breaking—not nonlinearity alone—as the active ingredient. Activation and normalization play distinct roles: activation concentrates continuity in the leading singular direction, while normalization distributes it across multiple directions. In
transformers, continuity is projection-specific: Q, K, Gate, and Up (which read from the residual stream) develop input-space ($\mathbf{v}_1$) continuity; O and Down (which write to it) develop output-space ($\mathbf{u}_1$) continuity; V alone, lacking an adjacent nonlinearity, develops only low continuity.
Read the full abstract ▾
Computation and Language
TabEmbed: Benchmarking and Learning Generalist Embeddings for Tabular Understanding
Minjie Qiang, Mingming Zhang, Xiaoyi Bao et al. (+5 more)
Foundation models have established unified representations for natural
language processing, yet this paradigm remains largely unexplored for tabular data. Existing methods face fundamental limitations: LLM-based approaches lack
retrieval-compatible vector outputs, whereas text
embedding models often fail to capture tabular structure and numerical semantics. To bridge this gap, we first introduce the Tabular
Embedding Benchmark (TabBench), a comprehensive suite designed to evaluate the tabular understanding capability of
embedding models. We then propose TabEmbed, the first generalist
embedding model that unifies tabular classification and
retrieval within a shared
embedding space. By reformulating diverse tabular tasks as semantic matching problems, TabEmbed leverages large-scale contrastive learning with positive-aware hard negative mining to discern fine-grained structural and numerical nuances. Experimental results on TabBench demonstrate that TabEmbed significantly outperforms state-of-the-art text
embedding models, establishing a new baseline for universal tabular representation learning. Code and
datasets are publicly available at https://github.com/qiangminjie27/TabEmbed and https://huggingface.co/
datasets/qiangminjie27/TabBench.
Read the full abstract ▾
All authors Minjie Qiang, Mingming Zhang, Xiaoyi Bao, Xing Fu, Yu Cheng, Weiqiang Wang, Zhongqing Wang, Ningtao Wang
Computation and Language
Adapting Large Language Models to a Low-Resource Agglutinative Language: A Comparative Study of LoRA and QLoRA for Bashkir
Mullosharaf K. Arabov and Svetlana S. Khaybullina
This paper presents a comparative study of parameter-efficient fine-tuning (PEFT) methods, including LoRA and QLoRA, applied to the task of adapting large
language models to the Bashkir
language, a low-resource agglutinative
language of the Turkic family. Experimental evaluation is conducted on a Bashkir text corpus of 71k documents (46.9M
tokens) using models of various architectures: DistilGPT2, GPT-2 (base, medium), Phi-2, Qwen2.5-7B, DeepSeek-7B, and Mistral-7B. To improve the reliability of results, each configuration was trained with three different random seeds. The lowest perplexity on the test set was obtained for GPT-2 medium with full fine-tuning (3.34). Meanwhile, QLoRA applied to Mistral-7B (3.79) and Phi-2 (3.81) achieved comparable quality with over 40 times fewer trainable parameters. However, we also observed cases of significant quality degradation when using PEFT for certain architectures (e.g., DeepSeek-7B with rank 8, perplexity = 129.55), indicating that the outcome depends critically on the choice of the base model and its tokenizer. Additionally, a qualitative analysis of generated texts based on Bashkir
prompts revealed that models with the best perplexity do not necessarily produce the most coherent outputs: QLoRA-tuned models generated monolingual Bashkir continuations, whereas the fully fine-tuned model with the lowest perplexity frequently switched to English. The results suggest that QLoRA on 7B-scale models offers an effective compromise between quality and computational cost for Bashkir. To ensure reproducibility, open data, code, and trained adapters will be released upon acceptance.
Read the full abstract ▾
Computation and Language
UFAL-CUNI at SemEval-2026 Task 11: An Efficient Modular Neuro-symbolic Method for Syllogistic Reasoning
Ivan Kartáč, Kristýna Onderková, Jan Bronec et al. (+3 more)
This paper describes our system submitted to SemEval-2026 Task 11: Disentangling Content and Formal
Reasoning in Large
Language Models. We present an efficient modular neuro-symbolic approach, combining a symbolic prover with small
reasoning LLMs (4B parameters). The system consists of an LLM-based parser that translates natural
language syllogisms to a first-order logic (FOL) representation, an automated
theorem prover, and two optional modules: machine translation for
multilingual inputs and a symbolic
retrieval component for the identification of relevant premises. The system achieves competitive accuracy and relatively low content effect on most subtasks. Our ablations show that this approach outperforms LLM-based zero-shot baselines in this parameter size range, but also reveal limited
multilingual capabilities of small LLMs. Finally, we include a discussion of the task's main ranking metric and analyze its limitations.
Read the full abstract ▾
All authors Ivan Kartáč, Kristýna Onderková, Jan Bronec, Zdeněk Kasner, Mateusz Lango, Ondřej Dušek
Computation and Language
Unintended Negative Impacts of Promotional Language in Patent Evaluation
Bingkun Zhao, Chenwei Zhang and Hao Peng
Promotional
language has been increasingly used to aid the communication of innovative ideas in science. Yet, less is known about its role in the context of technological innovation. Here, we use a validated and domain-diagnosed lexicon of 135 promotional words to study the association between promotional
language and patent evaluation outcomes among 2.7 million USPTO patent applications. Our large-scale study reveals three unexpected findings. First, in contrast to scientific evaluation, we find that a higher frequency of promotional words is negatively associated with the probability of an application being (i) granted a patent, (ii) transferred ownership, and (iii) successfully appealed. This promotional penalty holds even after accounting for a range of confounding factors and is largely robust across different technological areas. Among matched samples, the difference in the success rate between the lowest and highest promotional density quintile is 5.5, 5.9, and 5.3 percentage points for patentability, transferability, and rejection reversal. Second, contrary to institutional skepticism, we show that promotional
language is not a mask of weak technology, but objectively reflects the degree of combinatorial novelty and future citation impact. Third, digging into the mechanisms, we find that the tolerance to promotional framing is strongly moderated by human factors, with men and experienced examiners showing a higher acceptance of promotional narratives than women and novice examiners. By revealing an emerging paradox in the patent system, our study offers theoretical and practical implications for improving patent evaluation through more objective scrutiny of linguistic patterns in patent filings.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.LG
Reinforcement Learning for Compositional Generalization with Outcome-Level Optimization
Xiyan Fu and Wei Liu
Compositional generalization refers to correctly interpret novel combinations of known primitives, which remains a major challenge. Existing approaches often rely on supervised fine-tuning, which encourages models to imitate target outputs. This
token-level training paradigm fails to capture the global compositional structure required for generalizing to unseen combinations. In this work, we investigate whether compositional generalization can instead be improved through outcome-level reinforcement learning. We adopt Group Relative Policy Optimization to optimize models based on feedback on their final outputs. Within this framework, we explore both a simple binary outcome reward and a composite reward that provides additional composition feedback. Experiments on multiple compositional
benchmarks show that reinforcement learning improves compositional generalization compared to supervised fine-tuning. Further analysis reveals that supervised models tend to overfit frequent training compositions, whereas reinforcement learning improves compositional generalization by reshaping the output distribution, particularly for more complex composition types.
Read the full abstract ▾
Computation and Language
Rethinking Local Learning: A Cheaper and Faster Recipe for LLM Post-Training
Hengyu Shi, Tianyang Han, Peizhe Wang et al. (+3 more)
LLM post-training typically propagates task gradients through the full depth of the model. Although this end-to-end structure is simple and general, it couples task adaptation to full-depth activation storage, long-range backward dependencies and direct task-gradient access to pretrained representations. We argue that this full-depth backward coupling can be unnecessarily expensive and intrusive, particularly when post-training supervision is much narrower than pre-training. To this end, we propose LoPT: Local-Learning Post-Training, a simple post-training strategy that makes gradient reach an explicit design choice. LoPT places a single gradient boundary at the
transformer midpoint: the second-half block learns from the task objective, while the first-half block is updated by a lightweight feature-reconstruction objective to preserve useful representations and maintain interface compatibility. LoPT shortens the task-induced backward path while limiting direct interference from narrow task gradients on early-layer representations. Extensive experiments demonstrate that LoPT achieves competitive performance with lower
memory cost, higher training efficiency and better retention of pretrained capabilities. Our code is available at: https://github.com/HumyuShi/LoPT
Read the full abstract ▾
All authors Hengyu Shi, Tianyang Han, Peizhe Wang, Zhiling Wang, Xu Yang, Junhao Su
Computation and Language
Storage Is Not Memory: A Retrieval-Centered Architecture for Agent Recall
Joshua Adler and Guy Zehavi
Extraction at ingestion is the wrong primitive for
agent memory: content discarded before the query is known cannot be recovered at
retrieval time. We propose True
Memory, a six-layer architecture that shifts the center of the system from a storage schema to a multi-stage
retrieval pipeline operating over events preserved verbatim. The full system runs as a single SQLite file on commodity CPU with no external database, vector index, graph store, or GPU. On LoCoMo (1,540 questions across 10 multi-session conversations), True
Memory Pro reaches 93.0% accuracy (3-run mean) against 61.4% for Mem0, 65.4% for Supermemory, approximately 71% for Zep, and 94.5% for EverMemOS under a matched gpt-4.1-mini answer model. On LongMemEval (500 questions), True
Memory Pro reaches 87.8% (3-run mean). On BEAM-1M (700 questions at the 1-million-
token scale), True
Memory Pro reaches 76.6% (3-run mean), above the prior published result of 73.9% for Hindsight. A 56-configuration ablation shows a 1.3-percentage-point spread within the top-performing configuration family.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.LG
Self-Attention as Transport: Limits of Symmetric Spectral Diagnostics
Dominik Dahlem, Diego Maniloff and Mac Misiura
Every
attention head defines a degree-normalized transport operator, and a growing family of diagnostics reads model behavior (
hallucination among them) from its spectrum. We ask what such diagnostics can and cannot infer. The operator splits orthogonally into a symmetric part governing transport capacity and an antisymmetric part encoding orientation. We prove an identifiability limit: every transpose-invariant spectral diagnostic is orientation-blind (unable to distinguish an operator from its transpose, hence blind to the orientation of information flow), with a transpose-stability bound limiting any Lipschitz diagnostic's transpose sensitivity by the asymmetry coefficient $G$. This bounds what spectral diagnostics of the
attention operator can resolve (e.g.\ LapEigvals and the
attention-spectral branch of LLM-Check). On the surviving axis, a closed-form bipartite-Cheeger landscape shows uniform causal
attention obeys an $n$-independent temporal-cut floor $\phi \ge 1/5$ while window
attention pierces it as $O(w/n)$; the floor is an idealized
benchmark, not an empirical attractor, and the fraction of real heads falling below it is itself an empirically stable architectural descriptor. The two-axis diagnostic ($\phi$ for capacity, $G$ for asymmetry magnitude) yields a falsifiable polarity prediction, borne out in sign under length-controlled, forced-scoring evaluation across decoder-only, encoder-only, and encoder–decoder models (capacity-axis signal 0.62–0.84 LC-AUROC): polarity reverses between HaluEval and MedHallu, directionally as predicted though asymmetric in strength, with decision polarity calibrated per regime.
Read the full abstract ▾
Computation and Language
A Comparative Analysis of Machine Learning and Deep Learning Models for Tweet Sentiment Classification: A Case Study on the Sentiment140 Dataset
Vita Anggraini, Cintya Bella, Bastian et al. (+3 more)
The exponential growth of social media has created an urgent need for automated systems to analyze unstructured public sentiment in real time. This study compares a traditional Logistic Regression model using TF-IDF features with a deep learning Bidirectional Long Short-Term
Memory (BiLSTM) architecture on a 10,000-tweet subset of the Sentiment140
dataset. Experimental results show that Logistic Regression outperformed BiLSTM, achieving an accuracy of 73.5% compared with 69.17%, while the deep learning model exhibited mild overfitting. These findings suggest that for medium-scale informal text data, classical machine learning with robust feature extraction can outperform more complex deep learning approaches. Finally, the trained models were integrated into an interactive web application using Streamlit and deployed on Hugging Face Spaces for public access.
Read the full abstract ▾
All authors Vita Anggraini, Cintya Bella, Bastian, Luluk Muthoharoh, Ardika Satria, Martin C.T. Manullang
Computation and Language
Sentiment Analysis and Customer Satisfaction Prediction on E-Commerce Platforms Based on YouTube Comments Using the XGBoost Algorithm
Ridho Benedictus Togi Manik, Muhammad Aqil Ramadhan, Ihsan Maulana Yusuf et al. (+3 more)
The exponential expansion of digital commerce in Indonesia has significantly shifted consumer interactions toward
video-centric social networks, particularly YouTube. Consequently, the sheer volume of unstructured, multi-contextual comments poses a tremendous challenge for manual sentiment tracking. This study investigates and constructs a predictive model for customer satisfaction leveraging the Extreme Gradient Boosting (XGBoost) architecture coupled with Term Frequency-Inverse Document Frequency (TF-IDF) vectorization. By utilizing a secondary
dataset of YouTube comments retrieved from e-commerce review
videos, the raw text underwent rigorous preprocessing to generate normalized numerical features. The experimental results demonstrate that the PyCaret-optimized machine learning framework delivers superior classification resilience. Beyond standard performance metrics, lexical evaluations and feature-importance mapping uncover a notable phenomenon: e-commerce discourse is heavily infiltrated by socio-political terminologies, which ultimately influence the polarity of audience satisfaction.
Read the full abstract ▾
All authors Ridho Benedictus Togi Manik, Muhammad Aqil Ramadhan, Ihsan Maulana Yusuf, Luluk Muthoharoh, Ardika Satria, Martin Clinton Tosima Manullang
Computation and Language
BenCSSmark: Making the Social Sciences Count in LLM Research
Arnault Chatelain, Étienne Ollion, Qianwen Guan et al. (+7 more)
This position paper argues that the under-representation of social science tasks in contemporary LLM
benchmarks limits advances in both LLM evaluation and social scientific inquiry.
Benchmarks – standardized tools for assessing computational systems – are pivotal in the development of artificial intelligence (AI), including large
language models (LLMs).
Benchmarks do more than measure progress – they actively structure it, shaping reputations, research agendas, and commercial outcomes. Despite this central role, the social sciences are largely absent from mainstream evaluation frameworks, even though scholars in these fields generate dozens of rigorously annotated, context-sensitive
datasets each year. Integrating this work into
benchmark design could significantly improve the generalization and robustness of AI models. In turn, models trained on social scientific tasks would likely yield better performance on classic and contemporary tasks in disciplines as diverse as history, sociology, political science or economics. This is all the more pressing as these disciplines are quickly turning to LLMs for assistance. To address this gap, we introduce BenCSSmark, a
benchmark composed of
datasets annotated by computational social scientists. By integrating social scientific perspectives into benchmarking, BenCSSmark seeks to promote more robust, transparent, and socially relevant AI systems and to foster efficient collaboration.
Read the full abstract ▾
All authors Arnault Chatelain, Étienne Ollion, Qianwen Guan, Diandra Fabre, Lorraine Goeuriot, Emile Chapuis, Abdelkrim Beloued, Marie Candito, Nicolas Hervé, Didier Schwab
Computation and Language
A Comparative Study of PyCaret AutoML and CNN-BiLSTM for Binary Hate Speech Detection in Indonesian Twitter
Tanty Widiyastuti, Mayada, Adisty Syawalda Ariyanto et al. (+3 more)
This paper compares a PyCaret AutoML branch and a CNN-BiLSTM branch for binary hate
speech detection on Indonesian Twitter using the HS label from the corpus of Ibrohim and Budi. Both branches share the same preprocessing pipeline so that the comparison reflects modelling differences rather than inconsistent data preparation. The conventional branch uses TF-IDF with a lexicon-based abusive-word count, whereas the neural branch learns dense
token representations and captures both local phrase patterns and bidirectional context. The
benchmark is built from the released 13,130-row annotation table, whose HS label yields a 58:42 class ratio. On the held-out split, CNN-BiLSTM achieves the best result with 83.8% accuracy, 79.8% precision, 82.7% recall, and 81.2% F1-score. Within the PyCaret branch, Random Forest is the strongest conventional model with 77.2% accuracy and 77.0% F1-score. The neural branch therefore improves accuracy by 6.6 points and F1-score by 4.2 points. Exploratory corpus analysis, learning curves, and confusion
matrices show that the
dataset is short-text, moderately imbalanced, and still difficult because many decisions depend on local lexical cues plus short contextual composition. The study concludes that PyCaret AutoML is an effective conventional benchmarking framework, whereas CNN-BiLSTM is the stronger end model for the reported
benchmark setting.
Read the full abstract ▾
All authors Tanty Widiyastuti, Mayada, Adisty Syawalda Ariyanto, Luluk Muthoharoh, Ardika Satria, Martin Clinton Tosima Manullang
Computation and Language
Anticipating Innovation Using Large Language Models
Enrico Maria Fenoaltea, Filippo Santoro, Giordano De Marzo et al. (+2 more)
Forecasting innovation, intended as the emergence of new technological combinations, is a fundamental challenge for science and policy. We show that forthcoming combinations leave an early trace in the collective
language of patents, with predictive signals detectable even decades in advance. We show that signal is not attributable to any single inventor, but emerges as a collective shift in how technologies are described across thousands of patents. To this end, we introduce TechToken, a
transformer-based model that treats technologies, classified by International Patent Classification codes, as words in its vocabulary, learning the
language of technologies by
embedding these codes during fine-tuning. We define context similarity between code
embeddings as a measure of linguistic convergence and show that it accurately predicts first technological combinations. TechToken also improves general representation quality, outperforming state-of-the-art models across different patent-related tasks.
Read the full abstract ▾
All authors Enrico Maria Fenoaltea, Filippo Santoro, Giordano De Marzo, Segun Taofeek Aroyehun, Andrea Tacchella
Computation and Language · Cross-listed from cs.LG
Uncertainty-Aware Exploratory Direct Preference Optimization for Multimodal Large Language Models
Huatian Zhang, Zhendong Mao, Lei Zhang et al. (+1 more)
Direct Preference Optimization (DPO) has proven to be an effective solution for mitigating
hallucination in
Multimodal Large
Language Models (MLLMs) by learning from preference pairs. One of its key challenges lies in how to transfer the sequence-level preference into fine-grained supervision on visual fidelity. To safeguard
vision-related
tokens that are prone to
hallucination, existing methods typically allocate training emphasis according to the model's self-assessed visual sensitivity signals. However, such sensitivity, estimated by a model still under training, introduces self-referential bias: reinforcing already well-learned visual cues while neglecting hard-to-perceive but critical details, thereby limiting deeper
alignment. In this work, we propose an Uncertainty-aware Exploratory Direct Preference Optimization (UE-DPO) method for MLLMs, which enables the model to uncover its cognitive deficiencies and actively explore for self-correction, guided by
token-level epistemic uncertainty. Specifically, we first quantify the uncertainty from the model's failure to ground
token predictions in the given image. Then, based on an uncertainty-aware exploration intensity, we encourage more learning pressure on visually deficient
tokens in preferred samples, and alleviate the over-penalization of beneficial knowledge in dispreferred samples. Further, we provide a theoretical justification for our method, and extensive experiments demonstrate its effectiveness and robustness.
Read the full abstract ▾
All authors Huatian Zhang, Zhendong Mao, Lei Zhang, Yongdong Zhang
Computation and Language
Measuring Psychological States Through Semantic Projection: A Theory-Driven Approach to Language-Based Assessment
Maria Luongo, Davide Marocco and Nicola Milano
Recent advances in natural
language processing have enabled increasingly accurate estimation of psychological traits from
language. However, most existing approaches rely on supervised models trained to predict questionnaire scores, limiting interpretability and generalizability across contexts. The present study introduces a theory-driven and fully unsupervised framework for measuring psychological states directly from natural
language using semantic projection. Psychological constructs were operationalized as interpretable semantic axes derived from lexical anchors and items from validated clinical scales assessing depression, anxiety, and worry. Participants textual responses were embedded using Sentence-BERT and projected onto these axes to generate continuous psychological scores across multiple response formats, including selected words, generated words, phrases, and free-text responses. Projection scores were evaluated through correlations with standardized clinical measures , split-half reliability analyses, attenuation corrections, distributional similarity using Wasserstein distance, and comparisons with lexicon-based sentiment analysis (VADER). Results showed strong associations between projection scores and clinical measures, particularly for structured formats such as selected words, written words, and phrases. Free-text responses produced weaker results when analyzed as whole texts, but performance improved substantially when sentence-level aggregation strategies were applied. These findings support semantic projection as an interpretable and scalable alternative to supervised
language models for psychological assessment and highlight the importance of response format and text-processing strategies in
language-based mental health measurement.
Read the full abstract ▾
Computation and Language
Assessing Cognitive Effort in L2 Idiomatic Processing: An Eye-Tracking Dataset
Eduardo Santos, Juliana Carvalho and César Rennó-Costa
This paper presents the development and validation of an eye-tracking
dataset designed to investigate how second-
language (L2) learners process idiomatic expressions. While native speakers often rely on direct
retrieval of figurative meanings, L2 speakers frequently adopt a literal-first approach, which incurs measurable cognitive costs. This resource captures these costs through ocular metrics recorded from Portuguese L1 speakers of English across all CEFR proficiency levels (A1-C2). Although the study uses entry-level 60 Hz hardware (Tobii Pro Spark), we demonstrate that this sampling rate provides sufficient data density to detect macro-cognitive events such as fixations and regressions in reading. Preliminary analysis validates the
dataset by revealing a strong inverse correlation between
language proficiency and regressive eye movements. Integrated into the MIA (Modeling Idiomaticity in Human and Artificial
Language Processing) initiative, this
dataset serves as a cognitively grounded
benchmark for evaluating both human processing models and the
alignment of large
language models with human-like figurative understanding.
Read the full abstract ▾
Computation and Language
StoryAlign: Evaluating and Training Reward Models for Story Generation
Haotian Xia, Hao Peng, Yunjia Qi et al. (+4 more)
Story generation aims to automatically produce coherent, structured, and engaging narratives. Although large
language models (LLMs) have significantly advanced text generation, stories generated by LLMs still diverge from human-authored works regarding complex narrative structure and human-aligned preferences. A key reason is the absence of effective modeling of human story preferences, which are inherently subjective and under-explored. In this work, we systematically evaluate the modeling of human story preferences and introduce StoryRMB, the first
benchmark for assessing reward models on story preferences. StoryRMB contains $1,133$ high-quality, human-verified instances, each consisting of a
prompt, one chosen story, and three rejected stories. We find existing reward models struggle to select human-preferred stories, with the best model achieving only $66.3\%$ accuracy. To address this limitation, we construct roughly $100,000$ high-quality story preference pairs across diverse domains and develop StoryReward, an advanced reward model for story preference trained on this
dataset. StoryReward achieves state-of-the-art (SoTA) performance on StoryRMB, outperforming much larger models. We also adopt StoryReward in downstream test-time scaling applications for best-of-n (BoN) story selection and find that it generally chooses stories better aligned with human preferences. We will release our
dataset, model, and code to facilitate future research. Related code and data are available at https://github.com/THU-KEG/StoryReward.
Read the full abstract ▾
All authors Haotian Xia, Hao Peng, Yunjia Qi, Xiaozhi Wang, Bin Xu, Lei Hou, Juanzi Li
Computation and Language
Elicitation Matters: How Prompts and Query Protocols Shape LLM Surrogates under Sparse Observations
Ge Lei and Samuel J. Cooper
Large
language models are increasingly used as surrogate models for low-data optimization, but their optimizer-facing prediction and its uncertainty remain poorly understood. We study the surrogate belief elicited from an LLM under sparse observations, showing that it depends strongly on
prompt text and query protocol. We introduce an uncertainty-
alignment criterion that measures whether model uncertainty tracks residual ambiguity among sample-consistent functions. Across controlled inference tasks and
Bayesian optimization studies, we find that structural
prompts act as effective priors, POINTWISE and JOINT querying induce different beliefs, and sequential evidence leads to non-monotonic, order-sensitive confidence updates. These effects change downstream acquisition decisions and regret, showing that elicitation protocol is part of the LLM surrogate specification, not a formatting detail.
Read the full abstract ▾
Computation and Language
Gyan: An Explainable Neuro-Symbolic Language Model
Venkat Srinivasan, Vishaal Jatav, Anushka Chandrababu et al. (+1 more)
Transformer based pre-trained large
language models have become ubiquitous. There is increasing evidence to suggest that even with large scale pre-training, these models do not capture complete compositional context and certainly not, the full human analogous context. Besides, by the very nature of the architecture, these models hallucinate, are difficult to maintain, are not easily interpretable and require enormous compute resources for training and inference. Here, we describe Gyan, an explainable
language model based on a novel non-
transformer architecture, without any of these limitations. Gyan achieves SOTA performance on 3 widely cited data sets and superior performance on two proprietary data sets. The novel architecture decouples the
language model from knowledge acquisition and representation. The model draws on rhetorical structure theory, semantic role theory and knowledge-based computational linguistics. Gyan's meaning representation structure captures the complete compositional context and attempts to mimic humans by expanding the context to a 'world model'. AI model adoption critically depends on trust and transparency especially in mission critical use cases. Collectively, our results demonstrate that it is possible to create models which are trustable and reliable for mission critical tasks. We believe our work has tremendous potential for guiding the development of transparent and trusted architectures for
language models.
Read the full abstract ▾
All authors Venkat Srinivasan, Vishaal Jatav, Anushka Chandrababu, Geetika Sharma
Computation and Language
Every Step Counts: Step-Level Credit Assignment for Tool-Integrated Text-to-SQL
Yaxun Dai, Baolin Sun, Junying Wang et al. (+6 more)
Tool-integrated Text-to-SQL parsing has emerged as a promising paradigm, framing SQL generation as a sequential decision-making process interleaved with tool execution. However, existing reinforcement learning approaches mainly rely on coarse-grained outcome supervision, resulting in a fundamental credit assignment problem: models receive the same reward for any trajectory that yields the correct answer, even when intermediate steps are redundant, inefficient, or erroneous. Consequently, models are encouraged to explore suboptimal
reasoning spaces, limiting both efficiency and generalization. To address this problem, we propose FineStep, a novel framework for step-level credit assignment in tool-augmented Text-to-SQL. First, we introduce a reward design with independent process rewards to alleviate the signal sparsity of outcome supervision. Next, we present a step-level credit assignment mechanism to precisely quantify the value of each
reasoning step. Finally, we develop a policy optimization method based on step-level advantages for efficient updates. Extensive experiments on BIRD
benchmarks show that FineStep achieves state-of-the-art performance and reduces redundant tool interactions, with a 3.25% average EX gain over GRPO at the 4B scale.
Read the full abstract ▾
All authors Yaxun Dai, Baolin Sun, Junying Wang, Pengfei Wang, Yingqi Gao, Xuemei Dong, Mengdie Chu, Xiang Qi, Pingfu Chao
Computation and Language · Cross-listed from cs.CR
Sparse Tokens Suffice: Jailbreaking Audio Language Models via Token-Aware Gradient Optimization
Zheng Fang, Xiaosen Wang, Shenyi Zhang et al. (+2 more)
Jailbreak attacks on audio
language models (ALMs) optimize audio perturbations to elicit unsafe generations, and they typically update the entire waveform densely throughout optimization. In this work, we investigate the necessity of such dense optimization by analyzing the structure of
token-aligned gradients in ALMs. We find that gradient energy is highly non-uniform across audio
tokens, indicating that only a small subset of
token-aligned audio regions dominates the optimization signal. Motivated by this observation, we propose
Token-Aware Gradient Optimization (TAGO), which enables sparse jailbreak optimization by retaining only waveform gradients aligned with audio
tokens that have high gradient energy, while masking the remaining gradients at each iteration. Across three ALMs, TAGO outperforms baselines, and substantial sparsification preserves strong attack success rates (e.g. on Qwen3-Omni, $\mathrm{ASR}_{l}$ remains at 86% with a
token retention ratio of 0.25, compared to 87% with full
token retention). These results demonstrate that dense waveform updates are largely redundant, and we advocate that future audio jailbreak and safety
alignment research should further leverage this heterogeneous
token-level gradient structure.
Read the full abstract ▾
All authors Zheng Fang, Xiaosen Wang, Shenyi Zhang, Shaokang Wang, Zhijin Ge
Computation and Language
Paraphrase-Induced Output-Mode Collapse: When LLMs Break Character Under Semantically Equivalent Inputs
Aofan Liu and Jingxiang Meng
When the substantive content of a request is rewritten, do large
language models still answer in the format the original task asked for? We find that they often do not, even at temperature zero. On a 150-query evaluation over five compact 2025-era LLMs and four task types, we observe a systematic failure mode we call
prompt-variant output-mode collapse: when a closed-form
prompt asks for a bare label or a single choice
token, content-preserving
prompt variants can push the model into conversational prose, the requested format dissolves, and exact-match evaluation pipelines silently misjudge the result. To make this measurable, we release PARACONSIST, a 900-
prompt benchmark of 150 base queries with five lexical, syntactic, and semantic-expansion
prompt variants each, and a Semantic Consistency Score that decomposes
prompt-variant robustness into answer consistency, sentence-BERT semantic similarity, and length stability. Under a whole-word answer-set match, only ~22% of closed-form variant responses preserve the ground-truth label inside their output, while ~78% drift away from the answer space entirely. In our pool, the dominant predictor of collapse is task structure rather than model identity, with model differentiation jointly carried by answer consistency and length stability. Robustness audits should therefore track response-mode preservation as a first-class reliability target alongside answer accuracy.
Read the full abstract ▾
Computation and Language
CHE-TKG: Collaborative Historical Evidence and Evolutionary Dynamics Learning for Temporal Knowledge Graph Reasoning
Shuai-long Lei, Xiaobin Zhu, Jiarui Liang et al. (+3 more)
Temporal knowledge graph (TKG)
reasoning aims to predict future events from historical facts. A key challenge lies in jointly capturing two sources of predictive information in TKGs: historical evidence and evolutionary dynamics. However, existing methods typically focus on only one of these sources, which limits the ability to fully exploit the complementary predictive signals in TKGs. To address this, we propose CHE-TKG, a novel collaborative dual-view learning framework for TKG
reasoning. CHE-TKG explicitly separates and jointly models historical evidence and evolutionary dynamics, aiming to learn and exploit their complementary predictive signals. Specifically, CHE-TKG constructs a historical evidence graph to capture long-term structural regularities and stable relational constraints, alongside an evolutionary dynamics graph to model temporal transitions and recent changes, with dedicated encoders for each view. We further employ relation decomposition and a contrastive
alignment objective to better capture the predictive signals across the two views. Extensive experiments demonstrate that CHE-TKG achieves state-of-the-art performance on multiple
benchmarks.
Read the full abstract ▾
All authors Shuai-long Lei, Xiaobin Zhu, Jiarui Liang, Guoxi Sun, Zhiyu Fang, Xu-Cheng Yin