Computation and Language · Cross-listed from cs.DB
CAPER: Clause-Aligned Process Supervision for Text-to-SQL
Lujie Ban, Jiasheng Shi, Jinyang Li et al. (+3 more)
Text-to-SQL systems are typically evaluated by query-level execution correctness, but this terminal signal provides little guidance about which intermediate SQL decision caused success or failure.
Token-level dense supervision is also ill-suited: SQL
tokens do not align with complete semantic decisions, can penalize execution-equivalent queries, and are difficult to label reliably at scale. We therefore propose CAPER, which automatically derives clause-level supervision via counterfactual intervention on the SQL abstract syntax tree, enabling root-cause error localization for reward modeling; the resulting data is used to train CAPER-9B, a lightweight Clause-PRM that provides clause-boundary feedback for policy optimization and candidate verification. Experiments on BIRD and Spider show that clause-aligned supervision not only improves execution accuracy, achieving up to a 15.3% relative EX improvement over GPT-5.4, but also strengthens failure-localization capability, reaching 84.53% accuracy and 90.60% MRR on held-out failures. Our project page is at https://github.com/banrichard/RL-NL2SQL.
Read the full abstract ▾
All authors Lujie Ban, Jiasheng Shi, Jinyang Li, Xiaolin Han, Tsz Nam Chan, Chenhao Ma
Computation and Language
Beyond Ideal Instruction: A Comprehensive Framework for Evaluating LLMs in Realistic Interactions
Xuan Yang, Hao Xu, Tingfeng Hui et al. (+4 more)
Despite great advances in tool-use capabilities of large
language models (LLMs), existing evaluation
benchmarks struggle to fully align with real-world scenarios. Such
benchmarks mostly rely on simulated idealized user assumptions and lacks experience-oriented evaluation. These limitations fail to account for the ambiguity, uncooperative behaviors, and shifting intentions characteristic of real-world users. To fill this gap, we propose RUT-Bench, a dedicated
benchmark designed to assess LLMs under diverse Real-world User Tool calling scenarios. RUT-Bench supports high-fidelity simulations covering both ideal rational patterns and heterogeneous non-ideal behaviors across single-turn and multi-turn dialogues. We conduct comprehensive evaluations on 19 widely adopted open-source and proprietary LLMs using our
benchmark. Experimental results reveal that no tested LLMs achieve an overall success rate above 40%, and nearly all of them experience noticeable performance drops when facing more complicated non-ideal user inputs. Our code and data is available at https://github.com/Miaow-Lab/RUT-Bench.
Read the full abstract ▾
All authors Xuan Yang, Hao Xu, Tingfeng Hui, Hongsheng Xin, Kaike Zhang, Chunxiao Liu, Ning Miao
Computation and Language
From Script to Semantics: Prompting Strategies for African NLI
Anuj Tiwari, Terry Oko-odion and Hannah Nwokocha
Large
language models (LLMs) are increasingly evaluated in
multilingual settings, yet their inference behavior in low-resource African
languages remains underexplored especially under pure prompting without fine-tuning. We present a systematic study of prompting strategies for Natural
Language Inference (NLI) in Swahili, Yoruba, and Hausa using the AfriXNLI
benchmark. We evaluate five prompting strategies Baseline (zero-shot), Script-Aware,
Language Specific, Contrastive, and Native-Label Self-Translation (NL-STP) across two mid-sized open weight models (Llama3.2-3B and Gemma3-4B). To isolate the effect of
prompt design, the effect of few-shot examples and Chain-of-Thought
reasoning is eliminated in our study. We find a significant difference in performance of class wise across strategies with highly neutral class collapse and high prediction skew in some configurations. Contrastive prompting proves to be the most reliable and steadily improving strategy over
language and model and has better balance of class behavior and balance of overall accuracy gains. Notably, well-constructed
prompts are sufficient to beat more powerful baselines that are provided with few-shot
prompts and Chain-of-Thought
prompts. We have found that
prompt formulation is essential to
multilingual NLI with low-resource
languages and that
language aware decision structuring can be used to meaningfully enhance robustness in resource challenged settings.
Read the full abstract ▾
Computation and Language
SagaQA: A Multi-hop Reasoning Benchmark for Long-form Narrative Understanding in TV Series
Galann Pennec, Zhengyuan Liu, Nicholas Asher et al. (+2 more)
We introduce SagaQA, a long-form
video benchmark for multi-hop
reasoning over full-length TV series. Existing
video reasoning benchmarks often emphasize local understanding of adjacent frames or clips. SagaQA addresses this gap by requiring high-level comprehension of extended
multimodal narratives in entire TV shows. A distinguishing feature of SagaQA is the granularity of its
reasoning steps. Our
dataset necessitates long-range
reasoning hops to connect information across completely different episodes. This requires models to reason over entire events and actions, demanding a deep understanding of the show's narration and progression at a
multimodal level. Motivated by recent progress in agentic methods, we further study how different planning strategies handle such complex
reasoning. We categorize these approaches into three classes-Parallel, Sequential, and Hybrid planners-and evaluate their ability to generate coherent and complete
reasoning plans. Our results on SagaQA suggest that hybrid planners consistently produce higher-quality plans and exhibit stronger capabilities for complex, high-level narrative understanding in TV shows.
Read the full abstract ▾
All authors Galann Pennec, Zhengyuan Liu, Nicholas Asher, Philippe Muller, Nancy F. Chen
Computation and Language
Multilingual Unlearning in LLMs: Transfer, Dynamics, and Reversibility
Chaoyi Xiang, Olga Ohrimenko, Benjamin I. P. Rubinstein et al. (+1 more)
Large
language models (LLMs) can memorize sensitive facts, motivating unlearning methods that remove targeted knowledge without costly retraining. However, unlearning research remains heavily English-centric. We study
multilingual unlearning by extending the TOFU
benchmark to five
languages, and fine-tune, unlearn, and query our models with different permutations of
languages. We find that unlearning transfer, the ability of an unlearned model to "forget" facts in
languages other than the unlearning
language, is highly variable: e.g., it is strongest between
languages sharing scripts and families, and we show that the unlearning
language predicts which query
languages are most likely to yield the strongest transfer. Layer-wise analysis reveals that unlearning leaves the shared cross-lingual latent space largely intact in early layers, instead operating primarily in later decoding layers. This suggests that unlearning does not truly erase knowledge, but rather induces superficial suppression. Exploiting this structure, a single inference-time steering direction reverses much of this suppression across
languages, recovering 50% (Qwen) and 90% (Gemma) of the unlearned knowledge.
Read the full abstract ▾
All authors Chaoyi Xiang, Olga Ohrimenko, Benjamin I. P. Rubinstein, Lea Frermann
Computation and Language
SEA-NLI: Natural Language Inference as a Lens into Southeast Asian Cultural Understanding
Peerawat Chomphooyod, Jian Gang Ngui, Yosephine Susanto et al. (+5 more)
Frontier LLMs perform well in Western contexts, but remain poorly tested on underrepresented cultures such as those in Southeast Asia (SEA). Existing NLI
benchmarks are largely Western-centric, translation-derived, or monolingual, limiting their ability to measure culturally grounded
reasoning. We introduce SEA-NLI, a native, culturally grounded NLI
benchmark covering eight SEA countries in English and native regional
languages, verified by native speakers. Across 17 encoder and decoder models, we observe a low performance from all models, especially for knowledge-intensive categories such as
Languages and Science and Technology. Our analysis shows that failure cases mainly stem from missing SEA cultural knowledge: SEA-adapted models and culture-aware prompting improve performance, while CoT prompting offers limited gains.
Read the full abstract ▾
All authors Peerawat Chomphooyod, Jian Gang Ngui, Yosephine Susanto, Attapol T. Rutherford, Alham Fikri Aji, Sarana Nutanong, Can Udomcharoenchaikit, Peerat Limkonchotiwat
Computation and Language · Cross-listed from cs.CV
VistaHop: Benchmarking Long-Horizon Visual DeepSearch
Hang He, Chuhuai Yue, Chengqi Dong et al. (+6 more)
Visual DeepSearch tasks require
multimodal large
language models (MLLMs) to resolve complex visual queries by repeatedly inspecting image regions, grounding
reasoning in visual evidence, and connecting fine-grained clues across multiple steps. However, existing
benchmarks primarily evaluate single-step visual understanding or isolated visual-query response generation. They have limited difficulty, limited search horizons, and single-pass image inspection, and thus fail to evaluate models' ability to iteratively revisit visual evidence and reason across multiple steps. In this work, we introduce VistaHop, a
benchmark designed specifically to evaluate Visual DeepSearch. It evaluates repeated image inspection, visual-anchor grounding, and long-horizon evidence traversal across different visual regions. VistaHop comprises 600 images, 25 visual search scenarios, and 600 Visual DeepSearch tasks. We also propose VistaArena, a unified evaluation framework that supports tool-based interactions, including visual
retrieval, image inspection, and evidence-grounded
reasoning. Experiments show that even state-of-the-art MLRMs remain far from solving VistaHop, with the best-performing model, SenseNova-MARS-32B, achieving only 26.33% Pass@1. These findings highlight the importance of specialized
benchmarks and improved agentic methods for Visual DeepSearch.
Read the full abstract ▾
All authors Hang He, Chuhuai Yue, Chengqi Dong, Chengcheng Wan, Ting Su, Haiying Sun, Jiajun Chai, Xiaohan Wang, Guojun Yin
Computation and Language
Beyond "To whom it may concern": Tailoring Machine Translation to Audience and Intent
Raphael Merx, Ekaterina Vylomova and Trevor Cohn
Translation quality depends on purpose: the same source text demands different translations depending on audience, tone, and communicative intent. Yet MT models and metrics treat translation as a fixed mapping from source to target. LLMs enable users to explicitly specify purpose alongside source text, yet this capability has not been evaluated at scale. We introduce a systematic evaluation of purpose-driven MT across 50
languages, 5 model sizes and 8 text domains. We find that (1) explicit instructions substantially improve translation adaptedness, with larger gains on informal domains (conversation, social media), for larger model sizes and for higher-resource
languages; (2) instructions outperform semantically-matched few-shot examples and paragraph-level context; (3) traditional MT metrics fail to capture adaptation quality, often penalizing adapted translations; (4) when curated instructions are unavailable, models can self-generate them from surrounding document context, closing up to 80% of the adaptedness gap to curated instructions. Our results establish that purpose-adapted MT is a viable and measurable capability of LLMs, while highlighting the need for purpose-aware metrics.
Read the full abstract ▾
Computation and Language
The Word and the Way: Strategies for Domain-Specific BERT Pre-Training in German Medical NLP
Henry He, Johann Frei and Raphael Schmitt
Digital healthcare generates vast amounts of clinical text that can support AI-assisted applications, yet German biomedical
language models remain limited by older architectures or restricted training data. We present ChristBERT (Clinical- and Healthcare-Related Issues and Subjects Tuned BERT), a family of domain-specific German RoBERTa-based
language models trained on a 13.5GB corpus of scientific publications, clinical texts, health-related web content, and translated clinical resources. To investigate the impact of domain adaptation strategies in German clinical NLP, we compare continued pre-training, training from scratch, and domain-specific vocabulary adaptation. The resulting models are evaluated on three medical named entity recognition tasks and two text classification tasks. ChristBERT consistently outperforms existing general-purpose and medical German
language models on four of five
benchmarks and establishes a new state of the art for German clinical
language modeling. Our results show that the optimal adaptation strategy is task-dependent: in our evaluation, training from scratch is particularly effective for highly specialized clinical texts, whereas continued pre-training performs well on more commonly written medical texts. All models are publicly released to support future research and applications in German medical NLP.
Read the full abstract ▾
Computation and Language
Structures Facilitate Retrieve, Rerank, and Generate
Yeqin Zhang, Haomin Fu, Xujie Zhang et al. (+1 more)
Document-grounded dialogue systems (DGDS) utilize knowledge from external documents to answer domain-specific user questions. Existing solutions typically divide documents into independent passages for
retrieval and response generation. This approach, however, neither makes good use of structural information within documents nor provides enough (document) context for knowledge selection and responses. This paper proposes SF-Re2G to address such issues systematically. Firstly, we seek to improve a passage representation by contrasting it with others of the same section, thus improving the
retrieval performance. Secondly, a structure-enhanced reranker is built, leveraging the fact that multiple grounding passages of one dialog turn tend to be in the same neighborhood. Specifically, candidates from the
retrieval are grouped into subgraphs according to the document structure. The reranker will rescore the candidate integrating its group information. Finally, the chosen passages are used for responses, taking into account the subgraph context for better generation. Experimental results on two DGDS
datasets validate our method for both Chinese and English.
Read the full abstract ▾
All authors Yeqin Zhang, Haomin Fu, Xujie Zhang, Cam-Tu Nguyen
Computation and Language
When Does Complexity Conditioning Help a Frozen Sentence Embedding? A Controlled Study of Per-Sentence and Pair-Level Difficulty Adaptation
Suhwan Hwang
A common intuition is that sentence
embeddings should adapt to the difficulty of the input. We test this intuition in a controlled, multi-seed setting: a lightweight post-encoder adapter attaches to a frozen Qwen3-
Embedding-0.6B encoder, accessing only its final pooled
embedding, and is evaluated on four paraphrase and semantic-similarity tasks (PAWS, MRPC, QQP, STS-B). The naive form of the idea fails: surface-based per-sentence complexity is nearly uncorrelated with frozen-baseline error (Pearson approximately 0.05) and provides no advantage over constant or shuffled controls, while degrading a saturated baseline. Even when the target is aligned to a non-circular pair-difficulty signal, the per-sentence gate still cannot reliably capture difficulty because difficulty is primarily a property of the pair, not the individual sentence. In contrast, a small pair-level residual gated by a held-out cross-encoder difficulty signal yields consistent gains on the larger and graded tasks, including +0.022 Spearman on STS-B and +0.037 on QQP, while remaining anchored to the frozen baseline across all seeds. Because this useful form operates on sentence pairs rather than individual sentences, the resulting model is best understood as a lightweight re-ranker over cached frozen
embeddings, not a replacement single-vector
embedding; we make no state-of-the-art claim. Our contribution is a controlled account of when difficulty-aware adaptation helps and when it fails, together with a pre-training diagnostic that predicts the available headroom.
Read the full abstract ▾
Computation and Language
Benchmarking Speech-to-Speech Translation Models
Alkis Koudounas, Hayato Futami, Quentin Jodelet et al. (+3 more)
Speech-to-
speech translation (S2ST) has advanced rapidly, but offline evaluation lacks a unified protocol: studies report non-overlapping metric subsets, preventing direct comparisons. We introduce COMPASS, a unified and reproducible benchmarking framework integrating 46 metrics across eight dimensions, and deploy it on 1,248 model-
language configurations from FLEURS and CVSS, spanning cascaded and end-to-end architectures over ten
language pairs. Architectures exhibit complementary strengths: best-vs-worst gaps exceed 30% on naturalness and speaker preservation but remain within a few points on translation quality, so single-metric rankings systematically misrepresent system quality. Correlation filtering reduces 46 metrics to 10 per direction, with three axes requiring different metrics across X$\to$EN and EN$\to$X (e.g., TER/UTMOS vs. ChrF++/NISQA-MOS); these subsets preserve rankings (Spearman's $\rho>0.80$) while cutting evaluation time by $\approx 2.5\times$. Human validation across dubbing, podcasts, and medical domains shows standalone MOS predictors fail to predict listener preference, while top domain-specific metrics correlate with human judgment ($\rho \geq 0.90$). We release COMPASS as a foundation for domain-aware S2ST evaluation.
Read the full abstract ▾
All authors Alkis Koudounas, Hayato Futami, Quentin Jodelet, Osamu Take, Shinji Watanabe, Emiru Tsunoo
Computation and Language
ARBOR: Online Process Rewards via a Reusable Rubric Buffer for Search Agents
Zheng Liu, Longxiang Zhang, Xintong Wang et al. (+8 more)
LLM-based search
agents are trained predominantly with outcome-only reward, leaving the search process itself unsupervised. This signal degenerates on outcome-homogeneous groups where all sampled trajectories share the same correctness, yielding zero within-group advantage and no gradient. Existing process supervision either trains a costly verifier or generates per-query rubrics that are inconsistent across queries and discarded after one use. We propose ARBOR (Adaptive Rubric Buffer for Online Reward), a reusable process-reward framework that maintains a rubric
memory shared across queries. Query-local drafts induced from contrastive trajectories are admitted, consolidated into cross-query common rubrics, and retired as the policy evolves. A small active subset of common rubrics scores trajectories via sparse pairwise judging, and the resulting scores are added to the base reward, providing process-level gradient even when outcome reward is uniform. ARBOR consistently outperforms GRPO and DAPO baselines on four multi-hop QA
benchmarks, raising average LLM-judge accuracy by up to 4.2 points and converting up to 42% of otherwise-zero-gradient training groups into informative ones.
Read the full abstract ▾
All authors Zheng Liu, Longxiang Zhang, Xintong Wang, Zhiang Xu, Shaoxiong Zhan, Xin Shan, Wen Huang, Tao Dai, Shu-Tao Xia, Chengfu Huo, Liang Ding
Computation and Language · Cross-listed from cs.AI
Solipsistic Superintelligence is Unlikely to be Cooperative
Rakshit S Trivedi, Natasha Jaques, Logan Cross et al. (+2 more)
AI's central challenge is shifting from capability to coexistence. The dominant paradigm in AI research focuses on developing powerful
agents that treat the world as an exogenous and stationary source of feedback. We contend that superintelligence, an extremely capable task solver, born out of such a solipsistic approach to AI design, is unlikely to be cooperative. Deploying AI systems induces endogenous non-stationarity, resulting in a train-test-deploy gap where historical distributions diverge from the deployment context. We refer to this as the self-undermining property of unilateral optimization. Closing this gap requires AI that participates in cooperation: the
equilibrium-selection process through which multiple actors navigate their interdependence. We call for a non-solipsistic research paradigm that treats this interdependence as a core design principle rather than approaching cooperation as a task to solve. This entails building dynamic evaluation testbeds involving adaptive counterparties, treating institutions as design primitives, and preserving human agency as a structural feature of the systems we build.
Read the full abstract ▾
All authors Rakshit S Trivedi, Natasha Jaques, Logan Cross, Alexander Sasha Vezhnevets, Joel Z Leibo
Computation and Language
WebRISE: Requirement-Induced State Evaluation for MLLM-Generated Web Artifacts
Yuxin Meng, Yuhan Suo, Junjie Wang et al. (+9 more)
Existing
benchmarks for MLLM-generated web artifacts assess interaction through local evidence and miss the requirement-induced states and transitions that determine whether a page works. We introduce WebRISE, which compiles task requirements into Interaction Contract Graphs (ICGs) of observable states, user-intent transitions, and DOM/visual assertions for implementation-agnostic browser execution. WebRISE spans 442 tasks across five input modalities (Text, Markdown, Sketch, Image,
Video), with 5,495 transitions and 5,271 requirement checks that separate user-stated functions from implicit product-level constraints. Across 14 MLLMs, even the strongest model reaches only 65.6% transition validity and 66.3% requirement coverage, and visual quality is no proxy for behavior (Qwen3.6-35B-A3B on Markdown: V=80.8 yet T=15.5).
Video gives the strongest interaction signal (+10.6 pp implicit coverage over Text), while implicit constraints persist; defect injection shows ICG-based scoring detects state errors at 2-16x the rate of checkpoint-style evaluation.
Read the full abstract ▾
All authors Yuxin Meng, Yuhan Suo, Junjie Wang, Yuhan Sun, Yiyao Yu, Ruixu Zhang, Ruining Hu, Yubin Wang, Shouwei Ruan, Bin Wang, Yuxiang Zhang, Yujiu Yang
Computation and Language
Sample-Size Scaling of the African Languages NLI Evaluation
Anuj Tiwari, Oluwapelumi Ogunremu, Terry Oko-odion et al. (+2 more)
African
languages have very little labelled data, and it is unclear if augmenting the quantity of annotation data reliably enhances downstream performance. The study is a systematic sample-size scaling study of natural
language inference (NLI) on 16 African
languages based on the AfriXNLI
benchmark. Under controlled conditions, two
multilingual transformer models with roughly 0.6B parameters XLM-R Large fine-tuned on XNLI and AfroXLM-R Large are tested on sample sizes of between 50 and 500 labeled examples and average their results across random subsampling runs. As opposed to the usual belief of monotonic increase with increased data, we find a strongly
language sensitive and often non-monotonic scaling behavior. Some
languages show early saturation or decrease in performance with sample size as well as high variance in low resource regimes. These results indicate that the volume of data is not enough to guarantee stable profits to African NLI, creating the necessity of
language sensitive
datasets creation and stronger multi-lingual modelling strategies.
Read the full abstract ▾
All authors Anuj Tiwari, Oluwapelumi Ogunremu, Terry Oko-odion, Jesujuwon Egbewale, Hannah Nwokocha
Computation and Language
AI Rater Discrimination Depends on Scoring Protocol in Complex Clinical Decision-Making
Sangwon Baek, Kyu Yeon Hur and Kyunga Kim
Clinical AI evaluation increasingly delegates scoring to large
language models (LLMs) acting as AI raters, yet their scoring behavior across evaluation conditions has not been quantitatively characterized. We address this gap through a factorial study of AI rater behavior in adult type 2 diabetes (T2D) pharmacotherapy at 12-month outpatient follow-up, a clinical task involving complex decision-making operationalized across seven evaluation questions. Four open-source LLMs served simultaneously as clinical decision support system (CDSS) models and AI raters. Each CDSS output was scored under two scoring protocols: a rubric-anchored Gold Rubric (GR) protocol incorporating a patient-specific rubric, and a rubric-free Non Gold Rubric (Non-GR) protocol. Linear mixed effects models crossed the scoring protocol factor with five design factors – CDSS model, CDSS
prompt configuration (document-referenced generation [DRG] vs.\ Baseline), rater model,
prompt character, and
prompt type – and estimated main effects together with their protocol interactions. Across all questions, AI raters yielded consistently higher scores within a very narrow range (74–78 points on average) under Non-GR compared to those under GR (7.69 to 49.64 points lower mean scores; 1.68 to 3.67 times wider interquartile ranges). Within each question, GR amplified the AI rater's discrimination between DRG and Baseline CDSS outputs by factors of 1.76 to 5.10, while also revealing substantial behavioral variation across rater models that Non-GR suppressed. These findings support rubric anchoring as the scoring protocol that preserves discriminative power in clinical AI evaluation; rubric-free scoring cannot substitute when questions require patient-specific or jurisdiction-specific criteria that rater models cannot infer from parametric knowledge alone.
Read the full abstract ▾
Computation and Language
MemTrain: Self-Supervised Context Memory Training
Ziheng Li, Xingrun Xing, Haoqing Wang et al. (+2 more)
Memory is an indispensable capability for long-horizon LLM
agents, enabling them to preserve and utilize information accumulated across extended interactions. Existing
memory-
agent approaches are typically trained end-to-end with reinforcement learning on downstream tasks. However, collecting high-quality annotated problems for
memory-intensive scenarios is costly, and the resulting training data often lack sufficient diversity to cover general
memory behaviors. In this work, we propose MemTrain, a self-supervised training framework for generally enhancing the context-
memory capability of LLM
agents for more effective downstream post-training. MemTrain introduces two coupled proxy tasks over unlabeled Wikipedia corpora: (1) an end-to-end masked reconstruction objective, which requires the model to recover masked entities after multiple rounds of
memory updates, thereby encouraging
memory maintenance from the final outcome perspective; and (2) an intermediate
memory recall objective, which requires the model to reconstruct masked historical information using intermediate
memory states, encouraging faithful compression and
memory completeness throughout the interaction process. The two objectives are jointly optimized using GRPO. Extensive experiments on long-text QA and search-based QA
benchmarks demonstrate that MemTrain consistently improves downstream
memory-intensive
reasoning performance across different models, achieving gains of up to 17.67 points over direct task-specific post-training.
Read the full abstract ▾
All authors Ziheng Li, Xingrun Xing, Haoqing Wang, Zhi-Hong Deng, Yehui Tang
Computation and Language
SenseJudge: Human-Centric Preference-Driven Judgment Framework
Rui Li, Junfeng Liu, Xiangwen Kong et al. (+2 more)
Large
Language Models (LLMs) as judges across various scenarios such as assessing model responses is becoming an increasingly accepted paradigm. However, existing judgment approaches often rely on trained judgers using fixed preference data, which tend to overlook diverse user preferences and struggle to adapt to real-world human-AI dialogue scenarios. To address these limitations, we propose SenseJudge, a customizable judgment framework driven by human preferences and SenseBench, a diverse and challenging instruction-following
benchmark derived from real-world multi-turn interactions. We applied the automatic judgment framework and
benchmark to two tasks: (1) LLMs as personalized judges, and (2) model ranking. We conducted extensive experiments, and the results demonstrate that the SenseJudge framework surpasses other judgment methods and models in the LLMs-as-personalized-judges task and achieves model ranking that aligns with real human sense. Additionally, we conducted analyses on position bias and consistency, alongside ablation studies, which affirmed the robustness of SenseJudge.
Read the full abstract ▾
All authors Rui Li, Junfeng Liu, Xiangwen Kong, Linhai Xu, Zhifang Sui
Computation and Language · Cross-listed from cs.CV
GLINT: Sparsely Gated Vision-Language Alignment for Fine-Grained Radiology Representations
Jonggwon Park, Seongeun Lee, Junhyun Park et al. (+6 more)
Vision-
language models (VLMs) for radiology have emerged as a scalable paradigm by leveraging image-report pairs naturally produced in clinical workflows. However, this pairing reveals a mismatch in scale: each finding occupies only a small region of the image, yet supervision is provided only at the global image-report level. This poses a central challenge: prior approaches spread weight densely across all patches rather than concentrating on the sparse subset relevant to a given query. To address this, we present GLINT (Gated
Language-Image
alignmeNT), a framework that explicitly models this sparse correspondence. On the
alignment side, we introduce Sparsely Gated
Alignment, a novel architecture in which a sigmoid gate over a separate gate
embedding space activates only the patches relevant to each textual query, enforcing explicit sparsity. On the representation side, we add Dense Feature Regularization, which anchors the trainable encoder's intermediate features to a frozen self-supervised learning (SSL) teacher, preserving the fine-grained patch features that the gate relies on. The same recipe applies to both 2D chest X-ray (CXR) and 3D chest computed
tomography (CT), built with DINOv3 and V-JEPA 2.1, respectively. GLINT enables zero-shot classification, grounding, and
segmentation from free-text queries, and to our knowledge is the first to demonstrate zero-shot
segmentation on 3D CT volumes without mask supervision. Notably, the most pronounced gains arise on zero-shot grounding and
segmentation, where sparse, query-specific localization is required, consistent with our design intent. In downstream evaluation, GLINT outperforms both SSL encoders and medical VLMs on classification, report generation, and
segmentation.
Read the full abstract ▾
All authors Jonggwon Park, Seongeun Lee, Junhyun Park, Hannah Yun, Hyunwoong Kim, Sohyun Jeong, Hyewon Kang, Byungmu Yoon, Kyoyun Choi
Computation and Language
HyperPatch: Sequential Knowledge Editing Under n-ary Structural Drift
Yu-Kai Chan, Wen-Sheng Lien, Dong-Ting Yao et al. (+4 more)
Large
Language Models (LLMs) rely on Knowledge Editing (KE) to maintain temporal validity, yet real-world knowledge is inherently n-ary. We demonstrate that in non-stationary environments, sequential updates to complex relations induce N-ary Structural Drift, a phenomenon where the binary reification of n-ary events into triples fractures relational atomicity. This precipitates Structure-Conditioned Knowledge Transfer Failure, a systematic mis-grounding of the retriever frequently misdiagnosed as parametric
hallucination. To tackle this, we propose HyperPatch, a parameter-preserving framework that reformulates sequential KE as a stability problem over hypergraph
manifolds. HyperPatch preserves event integrity through three phases: (i) Structural Prior Initialization, establishing a
topology-aware
embedding space via contrastive learning on a Hypergraph Neural Network (HGNN) to capture high-order correlations; (ii) Sequential
Topology Editing, utilizing a dual-stage mechanism that employs SimHash-based Topological
Alignment for rapid conflict resolution and Topological LoRA Adaptation to track drift without backbone retraining; and (iii) Structure-Conditioned
Reasoning, which integrates globally consistent evidence from fused linguistic and structural
manifolds. On the MQuAKE-CF and MQuAKE-T
benchmarks, HyperPatch achieves relative gains in Hop-wise Accuracy (H-Acc) of 96.24% and 21.06% over the strongest baseline, respectively. Further ablations demonstrate superior reliability under continuous n-ary update streams, whereas the standard KG-based variant suffers H-Acc collapses of up to 88.3% due to structural misalignment.
Read the full abstract ▾
All authors Yu-Kai Chan, Wen-Sheng Lien, Dong-Ting Yao, Bo-Kai Ruan, Kwan-Yeung Lin, Hong-Han Shuai, Meng-Fen Chiang
Computation and Language
Fully Automated Identification of Lexical Alignment and Preference-Stage Shifts in Large Language Models
Thomas Stephan Juzek, Xiaoyang Ming and Jose A. Hernandez
The
language used by digital chat assistants such as ChatGPT can diverge from human expectations (misalignment). Research, mostly on Scientific English, has described both what divergences occur and, to some extent, why, linking them to the training stage of human preference learning. Yet, existing approaches rely on manual curation. This paper introduces two curation-free, assumption-light evaluation metrics: the Lexical
Alignment Score, which identifies lexical overuse, and the Triangulated Preference Shift, which quantifies how much of such shifts can be attributed to human preference learning. Using PubMed abstracts, continuations were generated and measured using windowed document prevalence across six model families (Falcon, Gemma, Llama, Mistral, OLMo, Yi). The procedure identifies, without manual intervention, overused items such as 'suggest', 'additionally', and 'strategy', and estimates their link to preference learning. Our findings replicate prior work and remain stable across parameter settings, random seeds, and evaluation on further data. The approach scales readily and enables systematic study of lexical (mis)
alignment beyond Scientific English and across
languages, and as such, the metrics have the potential to contribute to improved
alignment for future models and understanding of its origins.
Read the full abstract ▾
Computation and Language
A cross-domain tropical species dataset with Chinese vernacular names and CITES source links
Jeff Wang
We describe a versioned cross-domain
dataset of 410,499 active tropical species (working snapshot 2026-04-20) spanning three applied subdomains – tropical_plants, tropical_aquatic, and tropical_pets – that share a commercial and regulatory life cycle but are distributed across kingdom-organised biodiversity infrastructures. The resource joins taxonomic identifiers from GBIF, Plants of the World Online, iNaturalist, NCBI Taxonomy, the Catalogue of Life and the Encyclopedia of Life, and adds three original layers: a cross-domain ontology that re-segments taxa along trade and husbandry contexts; a Chinese vernacular layer with explicit per-name provenance under a typology that excludes unverified machine-generated proposals; and a CITES source-linkage layer connecting each taxon to its Species+ entry. Chinese vernacular coverage – the proportion of taxa carrying a CJK Chinese name distinct from the scientific binomial – reaches 99.50 percent (408,456 of 410,499; full-population count). Coverage characterises completeness, not name-translation accuracy; the latter is bounded by the four-level provenance typology and is the subject of a preliminary internal review reported here, with a blind external audit identified as the principal open item. Upstream content is referenced by stable identifier only for the original-contribution layers, supporting CC-BY 4.0 reuse. The
dataset is deposited on Zenodo (10.5281/zenodo.20377811). This preprint is the canonical v1.0 description of the
dataset's current state; future Data Descriptor submission is anticipated but is contingent on the validation and release-engineering items listed in the Limitations.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.LG
FederatedSkill: Federated Learning for Agentic Skill Evolution
Jingbo Yang, Guanyu Yao, Yang Zhang et al. (+3 more)
Modern LLM
agents increasingly rely on skill libraries to handle complex tasks, making skill
evolution a primary driver of self-improvement. However, isolated single-user task streams lack the diversity required to build comprehensive skills. While cross-user collaboration can overcome this data bottleneck, current trajectory-sharing approaches compromise user
privacy and impose a uniform global library that fails to accommodate client heterogeneity. We introduce FederatedSkill, a
privacy-preserving framework for collaborative
agent evolution. Moving beyond raw trajectory sharing, FederatedSkill utilizes semantic skill diffs, structured patches over local libraries, as the fundamental unit of communication. On the server side, an
evolution agent aggregates these patches to dynamically model client-specific capability boundaries, facilitating strictly personalized skill
evolution rather than a suboptimal global average. Evaluated across 20 distinct
agent task families, FederatedSkill demonstrates substantial gains over self-evolving baselines, achieving up to a 44.4% increase in success rate and a 37.5% reduction in computational cost.
Read the full abstract ▾
All authors Jingbo Yang, Guanyu Yao, Yang Zhang, Ramana Rao Kompella, Gaowen Liu, Shiyu Chang
Computation and Language · Cross-listed from cs.CR
PsychoPass: Geometric Profiling of Multi-Turn Adversarial LLM Conversations
Muberra Ozmen and Subhabrata Majumdar
Multi-turn jailbreak attacks on large
language models (LLMs) reveal a mismatch in current guardrails: they operate on individual turns, while attacks unfold as trajectories across conversations. We propose a shift from content to dynamics, modeling conversations as paths in representation space and asking whether
adversarial intent is encoded early in their geometry. We introduce PsychoPass, a framework that extracts
geometric features from conversation trajectories in
embedding space to predict a potential attack before harmful content is produced. These features achieve near-perfect performance in naïve classifiers, which is largely explained by the inclusion of number of turns as a feature. After removing this confound, a smaller but consistent
geometric signal remains, with classification performance that does not depend meaningfully on encoder choice. Crucially, this signal appears early in the conversation: attack outcomes remain above chance from short prefixes alone, more reliably than baseline guardrails. A supporting theoretical analysis explains these findings via a decomposition of length and shape, a detection bound based on prefix length, and encoder invariance. Together, these results show that
adversarial conversations leave an early, representation-robust
geometric fingerprint suitable for online monitoring.
Read the full abstract ▾
Computation and Language
DMT-CBT: Longitudinal Therapeutic State Modeling for CBT Counseling
Chang Liu, Shuyi Zhang, Changsheng Ma et al. (+3 more)
Large
language models (LLMs) have shown growing potential for Cognitive Behavioral Therapy (CBT) counseling. However, most existing approaches still formulate counseling as a local response generation problem, focusing on empathetic replies within short, text-only, or single-session interactions. We argue that this formulation fundamentally mismatches the nature of real psychotherapy. In clinical CBT, therapy is a longitudinal process in which therapists continuously infer, update, and intervene on evolving therapeutic states across sessions. Realistic CBT further involves
multimodal inference and delayed cross-session intervention effects, requiring models to capture longitudinal therapeutic state
evolution under partial observability. We propose DMT-CBT, a framework for Dynamic Modeling of evolving Therapeutic states in CBT counseling. DMT-CBT maintains structured therapeutic states across sessions while incorporating
multimodal behavioral grounding and tool-augmented intervention to support adaptive therapeutic
reasoning. Based on this framework, we construct DMTCorpus, a synthetic multi-session
multimodal CBT counseling
dataset featuring evolving therapeutic states, image-grounded client behaviors, and cross-session intervention continuity. Experimental results show that DMT-CBT improves counseling fidelity and therapeutic alliance, produces more favorable longitudinal affective trajectories, and preserves therapeutic states more faithfully than post-hoc extraction approaches.
Read the full abstract ▾
All authors Chang Liu, Shuyi Zhang, Changsheng Ma, Yongfeng Tao, Minqiang Yang, Bin Hu
Computation and Language · Cross-listed from cs.CR
Decoupled Smart Contract Audits: Lightweight LLM Framework via Distillation and Aggregation
Bagus Rakadyanto Oktavianto Putra, Muhamad Risqi Utama Saputra, Widyawan et al. (+1 more)
Smart contracts face critical security challenges that require thorough
auditing in decentralized web services. While Large
Language Models (LLMs) have shown promise in automated vulnerability detection, existing approaches lack severity evaluations with actionable remediation and demand unnecessarily massive computational overhead. In this study, we introduce an efficient end-to-end smart contract security audit framework utilizing lightweight, highly optimized open-source LLMs (0.6B-4B parameters). Our framework decouples comprehensive audit tasks into four interconnected components: vulnerability detection, explanation, severity classification, and remediation recommendation. To maintain high accuracy without massive parameters, we implement Rank-Stabilized Low-Rank Adapters (rsLoRA), knowledge
distillation, and a custom Chain-of-Verification (CoVe) aggregation strategy to systematically screen and consolidate multiple draft responses from the model into a highly accurate audit report. Experimental results demonstrate that our lightweight pipeline consistently outperforms state-of-the-art open-source coder dense LLMs (7B to 34B parameters), achieving 98.25% accuracy in vulnerability detection and an
alignment score of 0.4375 in generative explanation tasks. Furthermore, our extensive ablation studies empirically validate the superiority of our decoupled audit processes over unified prompting and uncover a novel severity centrality bias, establishing a critical
benchmark for future research in LLM-assisted
auditing.
Read the full abstract ▾
All authors Bagus Rakadyanto Oktavianto Putra, Muhamad Risqi Utama Saputra, Widyawan, Guntur Dharma Putra
Computation and Language
Experience-Driven Dynamic Exits for LLMs with Reinforcement Learning
Yanyu Zhu, Hoilam Pao, Niu Hu et al. (+6 more)
Large
Language Models suffer from slow autoregressive inference. While self-speculative decoding accelerates this process, its efficiency is hampered by static configurations like fixed exit layers and speculation lengths. We reframe this optimization as a Markov Decision Process and propose LEDE, a framework that uses offline reinforcement learning. LEDE learns a policy to dynamically select the optimal exit layer and speculation length based on the local context of the generated sequence at each step, balancing computational cost and draft quality. Comprehensive evaluations on Llama-2 and Llama-3 models show LEDE achieves up to a $2.0\times$$\sim$$2.7\times$ speedup over autoregressive decoding and and provides an additional 17% speedup over the static speculative baselines.
Read the full abstract ▾
All authors Yanyu Zhu, Hoilam Pao, Niu Hu, Wei Guo, Shaoxiong Zhan, Boyu Lai, Zitai Wang, Yongqin Zeng, Hai-Tao Zheng
Computation and Language
Coherence Maximization Improves Pluralistic Alignment
Taslim Mahbub, Yiding Pei and Shi Feng
Aligning AI systems with diverse human values requires value specifications grounded in concrete examples, but generating such examples without extensive human supervision remains an open challenge. We investigate what makes these examples effective, using Internal Coherence Maximization (ICM) – which infers labels by maximizing their mutual predictability – to generate persona-specific examples that steer a model toward a target group's values, without human supervision. Across four
benchmarks spanning classification, preference, and open-ended generation, ICM-inferred in-context examples match the performance of gold labels. Crucially, coherence matters beyond individual label accuracy: with accuracy held constant, more coherent examples generalize substantially better than incoherent ones. For personas underrepresented in pretraining data, targeted human feedback on the questions where the model is least certain about a persona's values yields better generalization than the same number of labels on arbitrary questions. These results identify coherence as a key design principle for scalable value specification, leveraging the diverse human perspectives already encoded in pretrained
language models.
Read the full abstract ▾
Computation and Language
Small RL Controller, Large Language Model: RL-Guided Adaptive Sampling for Test-Time Scaling
Runpeng Dai, Tong Zheng, Rui Liu et al. (+2 more)
Test-time scaling improves the
reasoning performance of large
language models but incurs substantial cost in both total computation and latency. Existing adaptive sampling methods partially mitigate this issue by dynamically deciding when to stop sampling, yet they typically rely on heuristic rules or rely on distribution assumptions. In this work, we formulate adaptive sampling as a Markov decision process (MDP). We train a lightweight sampling controller with reinforcement learning (RL) to jointly balance answer correctness, latency, and computation cost. At each round, the controller decides to stop sampling or to acquire additional samples. Our method is lightweight which only relies on statistics of final answers, and can be trained and deployed on CPU. We further show that the resulting framework admits an interpretation as the Lagrangian relaxation of a constrained optimization problem with explicit budget constraints. Experiments against strong baselines such as ASC and ESC show that our method achieves improved trade-offs among answer correctness, sampling rounds, and total samples required.
Read the full abstract ▾
All authors Runpeng Dai, Tong Zheng, Rui Liu, Chengsong Huang, Hongtu Zhu