Computation and Language
ProFit: Leveraging High-Value Signals in SFT via Probability-Guided Token Selection
Tao Liu, Taiqiang Wu, Runming Yang et al. (+3 more)
Supervised fine-tuning (SFT) is a fundamental post-training strategy to align Large
Language Models (LLMs) with human intent. However, traditional SFT often ignores the one-to-many nature of
language by forcing
alignment with a single reference answer, leading to the model overfitting to non-core expressions. Although our empirical analysis suggests that introducing multiple reference answers can mitigate this issue, the prohibitive data and computational costs necessitate a strategic shift: prioritizing the mitigation of single-reference overfitting over the costly pursuit of answer diversity. To achieve this, we reveal the intrinsic connection between
token probability and semantic importance: high-probability
tokens carry the core logical framework, while low-probability
tokens are mostly replaceable expressions. Based on this insight, we propose ProFit, which selectively masks low-probability
tokens to prevent surface-level overfitting. Extensive experiments confirm that ProFit consistently outperforms traditional SFT baselines on general
reasoning and mathematical
benchmarks.
Read the full abstract ▾
All authors Tao Liu, Taiqiang Wu, Runming Yang, Shaoning Sun, Junjie Wang, Yujiu Yang
Computation and Language
OrthoGeoLoRA: Geometric Parameter-Efficient Fine-Tuning for Structured Social Science Concept Retrieval on theWeb
Zeqiang Wang, Xinyue Wu, Chenxi Li et al. (+4 more)
Large
language models and text encoders increasingly power web-based information systems in the social sciences, including digital libraries, data catalogues, and search interfaces used by researchers, policymakers, and civil society. Full fine-tuning is often computationally and energy intensive, which can be prohibitive for smaller institutions and non-profit organizations in the Web4Good ecosystem. Parameter-Efficient Fine-Tuning (PEFT), especially Low-Rank Adaptation (LoRA), reduces this cost by updating only a small number of parameters. We show that the standard LoRA update $\Delta W = BA^\top$ has
geometric drawbacks: gauge freedom, scale ambiguity, and a tendency toward rank collapse. We introduce OrthoGeoLoRA, which enforces an SVD-like form $\Delta W = B\Sigma A^\top$ by constraining the low-rank factors to be orthogonal (Stiefel
manifold). A
geometric reparameterization implements this constraint while remaining compatible with standard optimizers such as Adam and existing fine-tuning pipelines. We also propose a
benchmark for hierarchical concept
retrieval over the European
Language Social Science Thesaurus (ELSST), widely used to organize social science resources in digital repositories. Experiments with a
multilingual sentence encoder show that OrthoGeoLoRA outperforms standard LoRA and several strong PEFT variants on ranking metrics under the same low-rank budget, offering a more compute- and parameter-efficient path to adapt foundation models in resource-constrained settings.
Read the full abstract ▾
All authors Zeqiang Wang, Xinyue Wu, Chenxi Li, Zixi Chen, Nishanth Sastry, Jon Johnson, Suparna De
Computation and Language · Cross-listed from cs.LG
Geometric Stability: The Missing Axis of Representations
Prashant C. Raju
Representational similarity methods compare the geometries of neural representations, but they do not measure how consistently the geometry of a single representation is recovered from subsets of its feature coordinates. We call this property
geometric stability and introduce Shesha, which estimates it by correlating representational dissimilarity
matrices from complementary random feature subsets. Shesha is not invariant to orthogonal rotations: representations with identical Gram
matrices, and therefore identical linear CKA, can have different
geometric stability. Controlled transformations further separate the quantities. Across $2{,}463$ encoder configurations spanning seven domains, similarity and stability are positively associated across non-PCA transformations ($\rho=+0.75$) but negatively associated under PCA-coordinate compression ($\rho=-0.47$). We further evaluate 170 pretrained
vision models across six
datasets. DINOv2 combines strong transfer performance with bottom-quartile stability on five of six
datasets, showing that transferability and feature-split stability need not coincide. Across random feature subsets, the marginal relationship between Shesha and linear-probe variability is
dataset-dependent; after controlling for task
alignment with LogME, higher Shesha is associated with lower variability on five of six
datasets. These results identify
geometric stability as a basis-dependent property that complements representational similarity and task
alignment.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.LG
EvasionBench: A Large-Scale Benchmark for Detecting Managerial Evasion in Earnings Call Q&A
Shijian Ma, Yan Lin and Yi Yang
We present EvasionBench, a comprehensive
benchmark for detecting evasive responses in corporate earnings call question-and-answer sessions. Drawing from 22.7 million Q&A pairs extracted from S&P Capital IQ transcripts, we construct a rigorously filtered
dataset and introduce a three-level evasion taxonomy: direct, intermediate, and fully evasive. Our annotation pipeline employs a Multi-Model Consensus (MMC) framework, combining dual frontier LLM annotation with a three-judge majority voting mechanism for ambiguous cases, achieving a Cohen's Kappa of 0.835 on human inter-annotator agreement. We release: (1) a balanced 84K training set, (2) a 1K gold-standard evaluation set with expert human labels, and (3) [Eva-4B], a 4-billion parameter classifier fine-tuned from Qwen3-4B that achieves 84.9% Macro-F1, outperforming Claude 4.5, GPT-5.2, and Gemini 3 Flash. Our ablation studies demonstrate the effectiveness of multi-model consensus labeling over single-model annotation. EvasionBench fills a critical gap in financial NLP by providing the first large-scale
benchmark specifically targeting managerial communication evasion.
Read the full abstract ▾
Computation and Language
Identity-Robust Language Model Generation via Content Integrity Preservation
Miao Zhang, Kelly Chen, Md Mehrab Tanjim et al. (+1 more)
Large
Language Model (LLM) outputs often vary across user sociodemographic attributes, leading to disparities in factual accuracy, utility, and safety, even for objective questions where demographic information is irrelevant. Unlike prior work on stereotypical or representational bias, this paper studies identity-dependent degradation of core response quality. We show empirically that such degradation arises from biased generation behavior, despite factual knowledge being robustly encoded across identities. Motivated by this mismatch, we propose a lightweight, training-free framework for identity-robust generation that selectively neutralizes non-critical identity information while preserving semantically essential attributes, thus maintaining output content integrity. Experiments across four
benchmarks and 18 sociodemographic identities demonstrate an average 77% reduction in identity-dependent bias compared to vanilla prompting and a 45% reduction relative to
prompt-based defenses. Our work addresses a critical gap in mitigating the impact of user identity cues in
prompts on core generation quality.
Read the full abstract ▾
All authors Miao Zhang, Kelly Chen, Md Mehrab Tanjim, Rumi Chunara
Computation and Language
Adaptive Multi-Stage Patent Claim Generation with Unified Quality Assessment
Chen-Wei Liang, Bin Guo, Zhen-Yuan Wei et al. (+1 more)
Current patent claim generation systems face three fundamental limitations: poor cross-jurisdictional generalization, inadequate semantic relationship modeling between claims and prior art, and unreliable quality assessment. We introduce a novel three-stage framework that addresses these challenges through relationship-aware similarity analysis, domain-adaptive claim generation, and unified quality assessment. Our approach employs multi-head
attention with eight specialized heads for explicit relationship modeling, integrates curriculum learning with dynamic LoRA adapter selection across five patent domains, and implements cross-
attention mechanisms between evaluation aspects for comprehensive quality assessment. Extensive experiments on USPTO HUPD
dataset, EPO patent collections, and Patent-CE
benchmark demonstrate substantial improvements: 7.6-point ROUGE-L gain over GPT-4o, 8.3% BERTScore enhancement over Llama-3.1-8B, and 0.847 correlation with human experts compared to 0.623 for separate evaluation models. Our method maintains 89.4% cross-jurisdictional performance retention versus 76.2% for baselines, establishing a comprehensive solution for automated patent prosecution workflows.
Read the full abstract ▾
All authors Chen-Wei Liang, Bin Guo, Zhen-Yuan Wei, Mu-Jiang-Shan Wang
Computation and Language
Contrastive Bi-Encoder Models for Multi-Label Skill Extraction: Enhancing ESCO Ontology Matching with BERT and Attention Mechanisms
Yongming Sun
Fine-grained labor
market analysis increasingly relies on mapping unstructured job advertisements to standardized skill taxonomies such as ESCO. This mapping is naturally formulated as an Extreme Multi-Label Classification (XMLC) problem, but supervised solutions are constrained by the scarcity and cost of large-scale, taxonomy-aligned annotations–especially in non-English settings where job-ad
language diverges substantially from formal skill definitions. We propose a zero-shot skill extraction framework that eliminates the need for manually labeled job-ad training data. The framework uses a Large
Language Model (LLM) to synthesize training instances from ESCO definitions, and introduces hierarchically constrained multi-skill generation based on ESCO Level-2 categories to improve semantic coherence in multi-label contexts. On top of the synthetic corpus, we train a contrastive bi-encoder that aligns job-ad sentences with ESCO skill descriptions in a shared
embedding space; the encoder augments a BERT backbone with BiLSTM and
attention pooling to better model long, information-dense requirement statements. An upstream RoBERTa-based binary filter removes non-skill sentences to improve end-to-end precision. Experiments show that (i) hierarchy-conditioned generation improves both fluency and discriminability relative to unconstrained pairing, and (ii) the resulting multi-label model transfers effectively to real-world Chinese job advertisements, achieving strong zero-shot
retrieval performance (F1@5 = 0.72) and outperforming TF–IDF and standard BERT baselines. Overall, the proposed pipeline provides a scalable, data-efficient pathway for automated skill coding in labor economics and workforce analytics.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.AI
AviationLMM: A Large Multimodal Foundation Model for Civil Aviation
Wenbin Li, Jingling Wu, Xiaoyong Lin.Jing Chen et al. (+1 more)
Civil aviation is a cornerstone of global transportation and commerce, and ensuring its safety, efficiency and customer satisfaction is paramount. Yet conventional Artificial Intelligence (AI) solutions in aviation remain siloed and narrow, focusing on isolated tasks or single modalities. They struggle to integrate heterogeneous data such as voice communications, radar tracks, sensor streams and textual reports, which limits situational awareness, adaptability, and real-time decision support. This paper introduces the
vision of AviationLMM, a Large
Multimodal foundation Model for civil aviation, designed to unify the heterogeneous data streams of civil aviation and enable understanding,
reasoning, generation and agentic applications. We firstly identify the gaps between existing AI solutions and requirements. Secondly, we describe the model architecture that ingests
multimodal inputs such as air-ground voice, surveillance, on-board telemetry,
video and structured texts, and performs cross-modal
alignment and fusion, and produces flexible outputs ranging from situation summaries and risk alerts to predictive diagnostics and
multimodal incident reconstructions. In order to fully realize this
vision, we identify key research opportunities to address, including data acquisition,
alignment and fusion, pretraining,
reasoning, trustworthiness,
privacy, robustness to missing modalities, and synthetic scenario generation. By articulating the design and challenges of AviationLMM, we aim to boost the civil aviation foundation model progress and catalyze coordinated research efforts toward an integrated, trustworthy and
privacy-preserving aviation AI ecosystem.
Read the full abstract ▾
All authors Wenbin Li, Jingling Wu, Xiaoyong Lin.Jing Chen, Cong Chen
Computation and Language
SubTokenTest: A Practical Benchmark for Real-World Sub-token Understanding
Shuyang Hou, Yi Hu and Muhan Zhang
Recent advancements in large
language models (LLMs) have significantly enhanced their
reasoning capabilities. However, they continue to struggle with basic character-level tasks, such as counting letters in words, a problem rooted in their tokenization process. While existing
benchmarks have highlighted this weakness through basic character operations, such failures are often dismissed due to lacking practical relevance. Yet, many real-world applications, such as navigating text-based maps or interpreting structured tables, rely heavily on precise sub-
token understanding. In this regard, we introduce SubTokenTest, a comprehensive
benchmark that assesses sub-
token understanding through practical, utility-driven tasks. Our
benchmark includes ten tasks across four domains and isolates tokenization-related failures by decoupling performance from complex
reasoning. We provide a comprehensive evaluation of nine advanced LLMs. Additionally, we investigate the impact of test-time scaling on sub-
token reasoning and explore how character-level information is encoded within the hidden states.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.LG
Distribution-Aligned Sequence Distillation for Superior Long-CoT Reasoning
Shaotian Yan, Kaiyuan Liu, Chen Shen et al. (+6 more)
In this report, we introduce DASD-4B-Thinking, a lightweight yet highly capable, fully open-source
reasoning model. It achieves SOTA performance among open-source models of comparable scale across challenging
benchmarks in mathematics, scientific
reasoning, and code generation – even outperforming several larger models. We begin by critically reexamining a widely adopted
distillation paradigm in the community: SFT on teacher-generated responses, also known as sequence-level
distillation. Although a series of recent works following this scheme have demonstrated remarkable efficiency and strong empirical performance, they are primarily grounded in the SFT perspective. Consequently, these approaches focus predominantly on designing heuristic rules for SFT data filtering, while largely overlooking the core principle of
distillation itself – enabling the student model to learn the teacher's full output distribution so as to inherit its generalization capability. Specifically, we identify three critical limitations in current practice: i) Inadequate representation of the teacher's sequence-level distribution; ii) Misalignment between the teacher's output distribution and the student's learning capacity; and iii) Exposure bias arising from teacher-forced training versus autoregressive inference. In summary, these shortcomings reflect a systemic absence of explicit teacher-student interaction throughout the
distillation process, leaving the essence of
distillation underexploited. To address these issues, we propose several methodological innovations that collectively form an enhanced sequence-level
distillation training pipeline. Remarkably, DASD-4B-Thinking obtains competitive results using only 448K training samples – an order of magnitude fewer than those employed by most existing open-source efforts. To support community research, we publicly release our models and the training
dataset.
Read the full abstract ▾
All authors Shaotian Yan, Kaiyuan Liu, Chen Shen, Bing Wang, Sinan Fan, Jun Zhang, Yue Wu, Zheng Wang, Jieping Ye
Computation and Language · Cross-listed from cs.LG
MMR-GRPO: Accelerating GRPO-Style Training through Diversity-Aware Reward Reweighting
Kangda Wei and Ruihong Huang
Group Relative Policy Optimization (GRPO) has become a standard approach for training mathematical
reasoning models; however, its reliance on multiple completions per
prompt makes training computationally expensive. Although recent work has reduced the number of training steps required to reach peak performance, the overall wall-clock training time often remains unchanged or even increases due to higher per-step cost. We propose MMR-GRPO, which integrates Maximal Marginal Relevance to reweigh rewards based on completion diversity. Our key insight is that semantically redundant completions contribute limited marginal learning signal; prioritizing diverse solutions yields more informative updates and accelerates convergence. Extensive evaluations across three model sizes (1.5B, 7B, 8B), three GRPO variants, and five mathematical
reasoning benchmarks show that MMR-GRPO achieves comparable peak performance while requiring on average 47.9% fewer training steps and 70.2% less wall-clock time. These gains are consistent across models, methods, and
benchmarks. Our code is released at: https://github.com/WeiKangda/MMR-GRPO.
Read the full abstract ▾
Computation and Language
How Many Human Judgments Are Enough? Feasibility Limits of Human Preference Evaluation
Wilson Y. Lee
Human preference evaluations are widely used to compare generative models, yet it remains unclear how many judgments are required to reliably detect small improvements. We show that when preference signal is diffuse across
prompts (i.e., all
prompt types are similarly informative), proportional allocation is minimax-optimal: no allocation strategy substantially improves detectability. Empirical analysis of large-scale human preference
datasets shows that most comparisons fall into this diffuse regime, exhibiting small preference margins that require far more judgments than typically collected, even in well-sampled comparisons. These limits persist across evaluation protocols and modalities, including chat, image generation, and code generation with execution feedback. In contrast, curated
benchmarks that reduce
prompt induced variability systematically induce larger margins and improve detectability through a $1.5\times$ reduction in
prompt-level variance. Our results show that inconclusive or negative human evaluation outcomes frequently reflect underpowered evaluation rather than model equivalence, underscoring the need to account explicitly for effect size, budget, and protocol design.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.AI
Human-AI Co-design for Clinical Prediction Models
Jean Feng, Avni Kothari, Patrick Vossler et al. (+6 more)
Developing safe, effective, and practically useful clinical prediction models (CPMs) traditionally requires iterative collaboration between clinical experts, data scientists, and informaticists. This process refines the often small but critical details of the model building process, such as which features/patients to include and how clinical categories should be defined. However, this traditional collaboration process is extremely time- and resource-intensive, resulting in only a small fraction of CPMs reaching clinical practice. This challenge intensifies when teams attempt to incorporate unstructured clinical notes, which can contain an enormous number of concepts. To address this challenge, we introduce HACHI, an iterative human-in-the-loop framework that uses AI
agents to accelerate the development of fully interpretable CPMs by enabling the exploration of concepts in clinical notes. HACHI alternates between (i) an AI
agent rapidly exploring and evaluating candidate concepts in clinical notes and (ii) clinical and domain experts providing feedback to improve the CPM learning process. HACHI defines concepts as simple yes-no questions that are used in linear models, allowing the clinical AI team to transparently review, refine, and validate the CPM learned in each round. In two real-world prediction tasks (acute kidney injury and traumatic
brain injury), HACHI outperforms existing approaches, surfaces new clinically relevant concepts not included in commonly-used CPMs, and improves model generalizability across clinical sites and time periods. Furthermore, HACHI reveals the critical role of the clinical AI team, such as directing the AI
agent to explore concepts that it had not previously considered, adjusting the granularity of concepts it considers, changing the objective function to better align with the clinical objectives, and identifying issues of data bias and leakage.
Read the full abstract ▾
All authors Jean Feng, Avni Kothari, Patrick Vossler, Andrew Bishara, Lucas Zier, Newton Addo, Aaron Kornblith, Yan Shuo Tan, Chandan Singh
Computation and Language
From Symbolic to Natural-Language Relations: Rethinking Knowledge Graph Construction in the Era of Large Language Models
Kanyao Han and Yushang Lai
Knowledge graphs (KGs) have commonly been constructed using predefined symbolic relation schemas, typically implemented as categorical relation labels. This design has notable shortcomings: real-world relations are often contextual, nuanced, and sometimes uncertain, and compressing it into discrete relation labels abstracts away critical semantic detail. Nevertheless, symbolic-relation KGs remain widely used because they have been operationally effective and broadly compatible with pre-LLM downstream models and
algorithms, in which KG knowledge could be retrieved or encoded into quantified features and
embeddings at scale. The emergence of LLMs has reshaped how knowledge is created and consumed. LLMs support scalable synthesis of domain facts directly in concise natural
language, and prompting-based inference favors context-rich free-form text over quantified representations. This position paper argues that these changes call for rethinking the representation of relations themselves rather than merely using LLMs to populate conventional schemas more efficiently. We therefore advocate moving from symbolic to natural-
language relation descriptions, and we propose hybrid design principles that preserve a minimal structural backbone while enabling more flexible and context-sensitive relational representations.
Read the full abstract ▾
Computation and Language
Mi:dm 2.0 Korea-centric Bilingual Language Models
Donghoon Shin, Sejung Lee, Soonmin Bae et al. (+63 more)
We introduce Mi:dm 2.0, a bilingual large
language model (LLM) specifically engineered to advance Korea-centric AI. This model goes beyond Korean text processing by integrating the values,
reasoning patterns, and commonsense knowledge inherent to Korean society, enabling nuanced understanding of cultural contexts, emotional subtleties, and real-world scenarios to generate reliable and culturally appropriate responses. To address limitations of existing LLMs, often caused by insufficient or low-quality Korean data and lack of cultural
alignment, Mi:dm 2.0 emphasizes robust data quality through a comprehensive pipeline that includes proprietary data cleansing, high-quality synthetic data generation, strategic data mixing with curriculum learning, and a custom Korean-optimized tokenizer to improve efficiency and coverage. To realize this
vision, we offer two complementary configurations: Mi:dm 2.0 Base (11.5B parameters), built with a depth-up scaling strategy for general-purpose use, and Mi:dm 2.0 Mini (2.3B parameters), optimized for resource-constrained environments and specialized tasks. Mi:dm 2.0 achieves state-of-the-art performance on Korean-specific
benchmarks, with top-tier zero-shot results on KMMLU and strong internal evaluation results across
language, humanities, and social science tasks. The Mi:dm 2.0 lineup is released under the MIT license to support extensive research and commercial use. By offering accessible and high-performance Korea-centric LLMs, KT aims to accelerate AI adoption across Korean industries, public services, and education, strengthen the Korean AI developer community, and lay the groundwork for the broader
vision of K-intelligence. Our models are available at https://huggingface.co/K-intelligence. For technical inquiries, please contact midm-llm@kt.com.
Read the full abstract ▾
All authors Donghoon Shin, Sejung Lee, Soonmin Bae, Hwijung Ryu, Changwon Ok, Hoyoun Jung, Hyesung Ji, Jeehyun Lim, Jehoon Lee, Ji-Eun Han, Jisoo Baik, Mihyeon Kim, Riwoo Chung, Seongmin Lee, Wonjae Park, Yoonseok Heo, Youngkyung Seo, Seyoun Won, Boeun Kim, Cheolhun Heo, Eunkyeong Lee, Honghee Lee, Hyeongju Ju, Hyeontae Seo, Jeongyong Shim, Jisoo Lee, Junseok Koh, Junwoo Kim, Minho Lee, Minji Kang, Minju Kim, Sangha Nam, Seongheum Park, Taehyeong Kim, Euijai Ahn, Hong Seok Jeung, Jisu Shin, Jiyeon Kim, Seonyeong Song, Seung Hyun Kong, Sukjin Hong, Taeyang Yun, Yu-Seon Kim, A-Hyun Lee, Chae-Jeong Lee, Hye-Won Yu, Ji-Hyun Ahn, Song-Yeon Kim, Sun-Woo Jung, Eunju Kim, Eunji Ha, Jinwoo Baek, Yun-ji Lee, Wanjin Park, Jeong Yeop Kim, Eun Mi Kim, Hyoung Jun Park, Jung Won Yoon, Min Sung Noh, Myung Gyo Oh, Wongyoung Lee, Yun Jin Park, Young S. Kwon, Hyun Keun Kim, Jieun Lee, YeoJoo Park
Computation and Language
Beyond Consensus: Perspectivist Modeling and Evaluation of Annotator Disagreement in NLP
Yinuo Xu and David Jurgens
Annotator disagreement is widespread in NLP, particularly for subjective and ambiguous tasks such as toxicity detection and stance analysis. While early approaches treated disagreement as noise to be removed, recent work increasingly models it as a meaningful signal reflecting variation in interpretation and perspective. This survey provides a unified view of disagreement-aware NLP methods. We first present a domain-agnostic taxonomy of the sources of disagreement spanning data, task, and annotator factors. We then synthesize modeling approaches using a common framework defined by prediction targets and pooling structure, highlighting a shift from consensus learning toward explicitly modeling disagreement, and toward capturing structured relationships among annotators. We review evaluation metrics for both predictive performance and annotator behavior, and noting that most
fairness evaluations remain descriptive rather than normative. We conclude by identifying open challenges and future directions, including integrating multiple sources of variation, developing disagreement-aware interpretability frameworks, and grappling with the practical tradeoffs of perspectivist modeling.
Read the full abstract ▾
Computation and Language
Efficient Multilingual Dialogue Processing via Translation Pipelines and Distilled Language Models
Santiago Martínez Novoa, Nicolás Rozo Fajardo, Diego Alejandro González Vargas et al. (+1 more)
This paper presents team Kl33n3x's
multilingual dialogue summarization and question answering system developed for the NLPAI4Health 2025 shared task. The approach employs a three-stage pipeline: forward translation from Indic
languages to English, multitask text generation using a 2.55B parameter distilled
language model, and reverse translation back to source
languages. By leveraging knowledge
distillation techniques, this work demonstrates that compact models can achieve highly competitive performance across nine
languages. The system achieved strong win rates across the competition's tasks, with particularly robust performance on Marathi (86.7% QnA), Tamil (86.7% QnA), and Hindi (80.0% QnA), demonstrating the effectiveness of translation-based approaches for low-resource
language processing without task-specific fine-tuning.
Read the full abstract ▾
All authors Santiago Martínez Novoa, Nicolás Rozo Fajardo, Diego Alejandro González Vargas, Nicolás Bedoya Figueroa
Computation and Language · Cross-listed from cs.CR
StegoStylo: Squelching Stylometric Scrutiny through Steganographic Stitching
Robert Dilworth
Stylometry – the identification of an author through analysis of a text's style (i.e., authorship attribution) – serves many constructive purposes: it supports copyright and plagiarism investigations, aids detection of harmful content, offers exploratory cues for certain medical conditions (e.g., early signs of dementia or depression), provides historical context for literary works, and helps uncover misinformation and disinformation. In contrast, when stylometry is employed as a tool for authorship verification – confirming whether a text truly originates from a claimed author – it can also be weaponized for malicious purposes. Techniques such as de-anonymization, re-identification, tracking, profiling, and downstream effects like censorship illustrate the
privacy threats that stylometric analysis can enable. Building on these concerns, this paper further explores how
adversarial stylometry combined with steganography can counteract stylometric analysis. We first present enhancements to our
adversarial attack, $\textit{TraceTarnish}$, providing stronger evidence of its capacity to confound stylometric systems and reduce their attribution and verification accuracy. Next, we examine how steganographic
embedding can be fine-tuned to mask an author's stylistic fingerprint, quantifying the level of authorship obfuscation achievable as a function of the proportion of words altered with zero-width Unicode characters. Based on our findings, steganographic coverage of 33% or higher seemingly ensures authorship obfuscation. Finally, we reflect on the ways stylometry can be used to undermine
privacy and argue for the necessity of defensive tools like $\textit{TraceTarnish}$.
Read the full abstract ▾
Computation and Language
SITA: Learning Speaker-Invariant and Tone-Aware Speech Representations for Low-Resource Tonal Languages
Tianyi Xu, Xuan Ouyang, Binwei Yao et al. (+4 more)
Tonal low-resource
languages are widely spoken but remain underserved by modern
speech technologies. A central challenge is learning
speech representations that are robust to nuisance variation, such as speaker gender, while preserving lexical tone, which carries word meaning. We propose SITA, a lightweight adaptation recipe for pretrained wav2vec-style self-supervised
speech encoders. Rather than designing a new backbone or objective, SITA combines existing objectives in a staged optimization framework to reduce tone collapse while preserving ASR capability. Stage 1 improves speaker invariance without erasing tonal contrasts by combining a cross-gender contrastive loss with a tone-repulsive loss that separates same-word, different-tone realizations. Stage 2 restores recognition-oriented linguistic information through CTC fine-tuning and knowledge
distillation on upper encoder layers. We evaluate SITA primarily on Hmong, a tonal
language with limited digital resources and a small speaker pool. Against
multilingual, speaker-
adversarial, label-aware, and semi-supervised baselines, SITA achieves the best trade-off between cross-gender lexical
retrieval and tone separation, while maintaining ASR accuracy close to an ASR-adapted XLS-R teacher. Results on Mandarin show consistent gains, suggesting that SITA is a general plug-in recipe for tonal
speech representation learning.
Read the full abstract ▾
All authors Tianyi Xu, Xuan Ouyang, Binwei Yao, Shoua Xiong, Sara Misurelli, Maichou Lor, Junjie Hu
Computation and Language
Is Grokking Worthwhile? Functional Analysis and Transferability of Generalization Circuits in Transformers
Kaiyu He, Zhang Mian, Peilin Wu et al. (+2 more)
While Large
Language Models (LLMs) excel at factual
retrieval, they often struggle with the "curse of two-hop
reasoning" in compositional tasks. Recent research suggests that parameter-sharing
transformers can bridge this gap by forming a "Generalization
Circuit" during a prolonged "grokking" phase. A fundamental question arises: Is a grokked model superior to its non-grokked counterparts on downstream tasks? Furthermore, is the extensive computational cost of waiting for the grokking phase worthwhile? In this work, we conduct a mechanistic study to evaluate the Generalization
Circuit's role in knowledge assimilation and transfer. We demonstrate that: (i) The inference paths established by non-grokked and grokked models for in-distribution compositional queries are identical. This suggests that the "Generalization
Circuit" does not represent the sudden acquisition of a new
reasoning paradigm. Instead, we argue that grokking is the process of integrating memorized atomic facts into an naturally established
reasoning path. (ii) Achieving high accuracy on unseen cases after prolonged training and the formation of a certain
reasoning path are not bound; they can occur independently under specific data regimes. (iii) Even a mature
circuit exhibits limited transferability when integrating new knowledge, suggesting that "grokked"
Transformers do not achieve a full mastery of compositional logic.
Read the full abstract ▾
All authors Kaiyu He, Zhang Mian, Peilin Wu, Xinya Du, Zhiyu Chen
Computation and Language
Representational Similarity Alignment of Figurative Language in Humans and Large Language Models
Samhita Bollepally, Aurora Sloman-Moll, Takashi Yamauchi et al. (+1 more)
Large
language models (LLMs) often produce judgments similar to those of humans, but their interpretation of figurative
language remains unclear. We compared human participants and four instruction-tuned LLMs (GPT-4, Gemma-2-9B, Llama-3.2, and Mistral-7B) on their ratings of 240 dialogue-based sentences spanning six linguistic traits: conventionality, sarcasm, funny,
emotion, idiomacy, and Gen Z slang. While humans and LLMs were strongly aligned at the surface level, differences emerged in representational similarity patterns, particularly in idiomacy and Gen Z slang. GPT-4 showed the closest
alignment to humans, though all models differed at the representational level, suggesting that similar outputs may arise from different underlying
language representations. We also examined the relationship between model scale and representational similarity using the Gemma model series (Gemma-2-9B-IT, Gemma-3-27B-IT, and Gemma-4-31B-IT), and found that representational similarity patterns were largely unaffected by model size.
Read the full abstract ▾
All authors Samhita Bollepally, Aurora Sloman-Moll, Takashi Yamauchi, Neeka Motamedrasa
Computation and Language
SpectraQuery: A Hybrid Retrieval-Augmented Conversational Assistant for Battery Science
Sreya Vangara, Jagjit Nanda, Yan-Kai Tzeng et al. (+1 more)
Scientific
reasoning increasingly requires linking structured experimental data with the unstructured literature that explains it, yet most large
language model (LLM) assistants cannot reason jointly across these modalities. We introduce SpectraQuery, a hybrid natural-
language query framework that integrates a relational Raman spectroscopy database with a vector-indexed scientific literature corpus using a Structured and Unstructured Query
Language (SUQL)-inspired design. By combining semantic parsing with
retrieval-augmented generation, SpectraQuery translates open-ended questions into coordinated SQL and literature
retrieval operations, producing cited answers that unify numerical evidence with mechanistic explanation. Across SQL correctness, answer groundedness,
retrieval effectiveness, and expert evaluation, SpectraQuery demonstrates strong performance: approximately 80 percent of generated SQL queries are fully correct, synthesized answers reach 93-97 percent groundedness with 10-15 retrieved passages, and battery scientists rate responses highly across accuracy, relevance, grounding, and clarity (4.1-4.6/5). These results show that hybrid
retrieval architectures can meaningfully support scientific workflows by bridging data and discourse for high-volume experimental
datasets.
Read the full abstract ▾
All authors Sreya Vangara, Jagjit Nanda, Yan-Kai Tzeng, Eric Darve