Computation and Language
Aspire: Can Models Self-Evolve from Vague Goals?
Yuhao Wu, Jingyuan Zhang, Jiajun Shi et al. (+18 more)
Many important forms of human learning begin with a vague goal, such as "become a better physicist" or "improve at research." Learners must interpret the goal, identify capability gaps, decide how to learn, and determine whether they have actually improved. In contrast, existing work on LLM self-
evolution typically begins with tasks and evaluation metrics specified by humans, reducing self-
evolution to optimizing an explicit objective rather than deciding what and how to learn. We introduce ASPIRE, a
benchmark for vague-goal-driven self-
evolution. ASPIRE provides only a natural-
language capability goal while downstream evaluation tasks remain hidden. The
agent must operationalize the goal by choosing data and update methods, constructing training and validation signals, and deciding when to evaluate. ASPIRE supports both model-weight and
agent-harness
evolution in a unified interactive environment and evaluates the resulting systems on a hidden, expert-authored set of 520 items spanning six goals. Our experiments show that vague goals redirect search effort toward goal interpretation. Current
agents routinely complete training and harness-editing loops, but weight-level gains remain sparse and unstable, and the strongest evolved harness remains below the engineered Qwen-
Agent reference.
Agents often train on mismatched data and trust narrow self-evaluations, so local gains fail to transfer to hidden evaluation and continued search and training can erase earlier improvements.
Read the full abstract ▾
All authors Yuhao Wu, Jingyuan Zhang, Jiajun Shi, Yuxuan Zhang, Xinping Lei, Junting Zhou, Zexuan Wang, Yuchen Wu, Huan Zhou, Duo Wang, Yinzhu Piao, Yongchang Peng, Yunfeng Shi, Jin Chen, Zuo Wang, Jinkai Liu, Jiaheng Liu, Wenxuan Zhang, Shen Yan, Wenhao Huang, Ge Zhang
Computation and Language · Cross-listed from cs.AI
BLOOM-WILT: Logit Tilting for Behaviour Elicitation in Automated LLM Auditing
Adrians Skapars and Edoardo Manino
Users of a deployed
language model routinely encounter behaviours that have never been observed during testing, since deployment puts the model through orders of magnitude more interactions than any evaluation can simulate. Automated auditors can mitigate this issue as they are flexible enough to discover almost any specified behaviour, yet their random sampling approach makes them inefficient. To address this shortcoming, we introduce BLOOM-WILT, a full
auditing pipeline that elicits natural multi-turn instances of rare behaviours, without training cost or access beyond the target's next-
token distribution. On the user input side, WILT revises its conversational strategy across rounds, learning from previous scored interactions. On the model output side, WILT generates behaviour-relevant continuations by feeding the model with an elicitation
prompt and mixing its next-
token distribution with the unprompted model. We evaluate WILT across 4 target models and 8 behaviours, and show that it improves over the baseline in 30 of the 32 settings, thus revealing that a larger number of models exhibit harmful behaviours than previously suggested. Furthermore, WILT is the only method that can raise the mean presence of self-harm encouragement from 51% to 100% in Qwen3.5-4B, without decreasing the output probability of the generated text. Our code is available at https://github.com/AdrSkapars/bloom-wilt.
Read the full abstract ▾
Computation and Language
S3Gym: Can LLMs Turn Self-Testing and Self-Judging into Self-Improvement?
Jiajun Shi, Siyuan Tao, Yuhao Wu et al. (+18 more)
Large
language models (LLMs) increasingly interact with external environments and accumulate substantial behavioral experience, yet existing
agent benchmarks largely evaluate them as fixed policies. It therefore remains unclear whether an
agent can actively test its behavior, judge the resulting experience, and use that experience to improve future decisions. We introduce \textbf{S\textsuperscript{3}Gym}, an interactive
benchmark for evaluating LLM self-improvement through three coupled capabilities: Self-Testing, Self-Judging, and Self-Improvement. S$^3$Gym separates permissive exploration from strict held-out evaluation and instantiates this protocol in seven text-based games with executable environment verifiers. We evaluate three pathways for incorporating interaction experience: direct History ICL, score-conditioned Summary
Memory, and parameter Training. Our experiments reveal that self-improvement is neither automatic nor uniform. Context-level experience improves performance for several model–game pairs, but the most effective pathway depends strongly on the task structure: summaries are beneficial when experience can be compressed into reusable strategic rules, yet often underperform raw history when success depends on precise, state-contingent information. Parameter training produces substantial gains on some tasks, but also exhibits unstable improvement and severe negative transfer on others. These findings show that recognizing successful actions is insufficient;
agents must also transform feedback into executable and transferable policies. S$^3$Gym provides a unified framework for diagnosing this process and identifying the bottlenecks that prevent
agents from translating interaction experience into reliable self-improvement.
Read the full abstract ▾
All authors Jiajun Shi, Siyuan Tao, Yuhao Wu, Zexuan Wang, Jingyuan Zhang, Jiaheng Liu, Xinping Lei, Xinrong Zhang, Siyuan Fang, Zhewen Tan, Tianle Cai, Junhao Fang, Jiameng Huang, Yueyang Wang, Jinkai Liu, Yuxuan Zhang, Jian Yang, Zhoujun Li, Shen Yan, Wenhao Huang, Ge Zhang
Computation and Language
Verbalizing Multi-Token Concepts in LLMs
Xijie Gong, Zimeng Huang and Tonghan Wang
Lens methods inspect model computation by mapping intermediate activations to vocabulary
tokens. Yet the concepts humans need to read out often span multiple
tokens—entities, phrases, intermediate objects—making
token-level readouts incomplete. Reliable multi-
token readout with little model-specific preparation remains challenging. We introduce Concept Lens:
token-level lens clues guide candidate concept search, then the model derives a representation for each candidate and scores it against the original activation. Across 2,400 multi-hop clozes on five LLMs (8B–70B), Concept Lens instantiated with J-lens and R-lens achieves average Rank@10 scores of 36.6% and 54.5%, respectively, compared with 21.7% for Template Lens. Concept-swap interventions on derived concept representations shift model answers toward those associated with the replacement concepts. Further experiments show that Concept Lens can also reveal what a model recognizes along the way, beyond what appears in its final answer. Our code is available at https://github.com/XijieGo/c-lens
Read the full abstract ▾
Computation and Language · Cross-listed from cs.AI
Agentic Context Cracking: Token-Efficient Data Reasoning Agents via Adaptive Structuring of Unstructured Data
Milad Rezaei Hajidehi, Qitong Wang and Stratos Idreos
Valuable data remains embedded in unstructured sources: web pages, reports, contracts, filings, earnings calls, and PDFs. The big bet in enterprise AI is deploying LLM
agents that reason over this data to answer complex questions for every knowledge worker.
Agents can do this today, but at prohibitive cost. Each question repeatedly opens large documents to recover scattered evidence, consuming up to a million
tokens. However, if the data were already structured, the same question would reduce to a cheap database lookup. For example, on FanOutQA
benchmark,
reasoning over an ideal pre-structured store is 28X cheaper, and the gap grows to orders of magnitude as questions fan out over more documents. Yet structuring everything in advance is not viable: documents hold vastly more possible structure than any workload will use, and the useful structure and documents are unknown until queries arrive. We propose agentic data cracking, a method that structures unstructured data adaptively and speculatively as a byproduct of
reasoning itself. Structuring is adaptive because observed queries decide when it happens and what matters, and speculative because it goes beyond the current question. Whenever the
agent opens a document to answer, a cracking sub-
agent forks from the already-loaded context at marginal cost and extracts grounded structure likely to serve related future queries. Over time, an increasing share of queries is fully covered by structured data and answered without opening a document, keeping agentic accuracy at close to RAG cost. On FanOutQA, extended with merely one related question per test question, cracking cuts cost by 53% while preserving accuracy. Agentic data cracking is a first step toward next-generation data infrastructure for agentic
reasoning over unstructured data: a shared substrate beneath the model where knowledge that
reasoning already paid to uncover accumulates.
Read the full abstract ▾
Computation and Language
Learning to Evaluate Before Improving: Automatic Rubric Induction for Automatic Research Agents
Xuehai Wang, Haowei Qin, Tongxin Liu et al. (+6 more)
Autonomous scientific research
agents are increasingly applied to end-to-end scientific workflows, including literature review, data analysis, experimentation, and report generation. However, open-ended research tasks often do not clearly specify the analyses, methods, and success criteria required to complete the task. As a result,
agents may miss important analyses, use inappropriate methods, or draw conclusions that are insufficiently supported by evidence. To address the problem, we present AutoSciRub, an evaluation-first framework that induces a task-specific executable rubric before research execution, and uses it to guide execution, criterion-level verification as well as iterative revision. AutoSciRub decomposes an underspecified instruction into atomic scientific goals, grounds them in relevant literature and task-visible data, and synthesizes specific, actionable, and verifiable criteria. The resulting rubric makes implicit experimental and evidential requirements explicit, providing guidance for experiments and analyses. During revision, rubric-guided verification identifies unmet criteria and enables targeted refinement of the research report and its supporting artifacts. On ResearchClawBench, AutoSciRub consistently improves all tested configurations, with an average gain of 2.08 points across three backbone LLMs under the fixed Codex harness and 2.95 points across three
agent harnesses using a fixed DeepSeek-V4-Flash backbone. On a randomly sampled 20-task subset of AstaBench E2E Discovery, AutoSciRub further achieves an average improvement of 16.8 points across three
agent harnesses, while maintaining or increasing the number of successfully completed tasks. These results demonstrate that evaluation-first guidance provides an effective and generalizable control mechanism for autonomous scientific research (Code: https://github.com/zjunlp/AutoSciRub).
Read the full abstract ▾
All authors Xuehai Wang, Haowei Qin, Tongxin Liu, Junkai Li, Buqiang Xu, Jintian Zhang, Yijun Chen, Zirui Xue, Shumin Deng
Computation and Language · Cross-listed from cs.LG
A Model with No Head and Many Thoughts
Nikita Koriagin, Yaroslav Aksenov, George Bredis et al. (+3 more)
Large
language models decode by projecting hidden states through a large vocabulary head at every step. This operation is computationally costly and forces all
reasoning to be expressed in discrete
tokens. We introduce Soft Latent Thinking, a method that replaces the LM head during
reasoning with a lightweight projector, enabling autoregressive rollout in
embedding space where
reasoning steps remain continuous rather than tokenized. Experiments on DeepSeek-Qwen-1.5B and LLaMA-3.2-3B show that Soft Latent Thinking consistently improves pass@k across all k while reducing per-step compute during chain-of-thought. Our method achieves the highest pass@32 among all soft-thinking approaches, demonstrating that effective
reasoning can be carried out in continuous space without discrete
token generation.
Read the full abstract ▾
All authors Nikita Koriagin, Yaroslav Aksenov, George Bredis, Gleb Gerasimov, Nikita Balagansky, Daniil Gavrilov
Computation and Language
Every Token Leaves a Ripple in the Stream of Thought: Eliciting Model-Internal Token Saliency for Chain-of-Thought Compression
Tianyi Zhao, Yinhan He, Wendy Zheng et al. (+2 more)
Chain-of-thought (CoT)
reasoning improves multi-step problem solving, but long
reasoning traces inflate inference cost.
Token-level CoT compression reduces this cost by
pruning full
reasoning chains into shorter traces for model adaptation, making
token selection the central challenge. Existing methods often rely on external scorers or heuristic signals only indirectly tied to the model's internal answer computation. We instead adopt a model-internal perspective: as the model forms an answer, each
reasoning token induces a ripple in the residual stream whose effect on the answer reflects the
token's contribution to the underlying computation. Building on this view, we propose MIST (Model-Internal Saliency for
Token-level CoT compression), which defines
token importance along two complementary axes: necessity, the drop in answer likelihood when a
token's internal contribution is removed, and sufficiency, the gain in answer likelihood when that contribution alone is provided. Combining the two yields a unified importance score for
pruning. Across four
reasoning benchmarks and four models, MIST consistently outperforms baseline methods, suggesting that model-internal saliency provides an effective proxy for
reasoning-
token importance.
Read the full abstract ▾
All authors Tianyi Zhao, Yinhan He, Wendy Zheng, Jundong Li, Chen Chen
Computation and Language · Cross-listed from econ.EM
When Can We Work in Embedding Space? What Text Embeddings Preserve
Simon Freyaldenhoven
When do text
embeddings work as inputs to empirical analysis? Their use rests on an assumption: that we can trade text for its low-dimensional
embedding, and lose little in doing so. I make that assumption precise under a generative model in which documents are mixtures of latent topics. I study two uses—clustering units in
embedding space and controlling for high-dimensional text. A cluster of
embeddings is a set of documents with similar topic mixtures; controlling for the
embedding is equivalent to controlling for the topic mixture, so validity reduces to whether that mixture captures the confounding. In an application to 363 U.S. metropolitan areas,
embedding-based clusters of LLM-generated economic descriptions recover interpretable economic archetypes and separate local employment dynamics more sharply than clustering on model residuals, or on a curated set of industry and demographic
covariates.
Read the full abstract ▾
Computation and Language
Improving Information Extraction with Learned Queries
Omar Sharif, Soroush Vosoughi and Nikhil Singh
When information extraction fails, a natural instinct is to improve the model doing it: for example, by scaling it up or refining its
reasoning. In this paper, we show that another part of the pipeline matters at least as much: the queries used to elicit this information. Across four clinical
benchmarks and five LLMs, improving the question design alone raises performance by 18.6 F1-score points, i.e. more than using larger extraction models. To make such question design learnable, we introduce List of Questions (LoQ), which generates document-specific question sets, and FeedQ, a feedback-driven optimization method that iteratively refines questions against extraction outcomes. The resulting optimized questions can be used to train lightweight generators: with fine-tuning, 4B-parameter models match or outperform expert-derived baselines and substantially exceed the performance of much larger untuned models. We release a
dataset of 12,820 optimized questions to support a broader shift in information extraction research toward treating question design as a first-class problem.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.LG
Does On-Policy Distillation Really Distill? From Noisy Teacher to Self-Improvement
Yi Ding and Ruqi Zhang
On-policy
distillation (OPD) offers dense
token-level supervision as an alternative to the sparse outcome-level advantages of reinforcement learning with verifiable rewards (RLVR). However, the teacher scores student-generated trajectories that are inherently off-policy for it, so the reliability of its supervision, and hence the source of the student's improvement, remains unclear. We quantitatively analyze teacher supervision during OPD training and find substantial noise whose prevalence increases with teacher scale. Surprisingly, the student policy is insensitive to such noise, converging to comparable performance regardless of whether noisy supervision is retained or removed. Does OPD distill at all? By analyzing what drives its gains, we find that learning concentrates on low log-probability
tokens, and using a single fixed negative advantage matches the performance of teacher-provided ones. This suggests that OPD works largely by suppressing low log-probability
tokens, which requires no teacher. These findings motivate On-Policy Self-Adaptation (OPSA), a supervision-free method using
entropy-adaptive negative advantages. It assigns stronger learning signals to high-
entropy positions, suppressing tail
tokens, and evenly redistributing probability mass among head
tokens. Compared with the base Qwen3-1.7B, OPSA improves Avg@32 by 35.41 points on AIME24, corresponding to a 263% relative gain, and more than doubles Pass@32 across all three
benchmarks. It also outperforms OPD by 16.77 points in Avg@32 on AIME24. Extensive experiments and analyses across model families and tasks further demonstrate its effectiveness and generalizability.
Read the full abstract ▾
Computation and Language
Type-Balanced Contextual Learning for Incremental Named Entity Recognition
Duzhen Zhang, Yahan Yu, Xiuyi Chen et al. (+2 more)
Incremental Named Entity Recognition (INER) stands as a pivotal task in information extraction, emphasizing the successive identification of new entity types within unstructured text. Faced with the continuous influx of entity types, INER grapples with two significant challenges: the widespread issue of catastrophic forgetting and the unique shift issue of the non-entity type semantics. While pseudo-labeling-based INER methods have proven effective in addressing these challenges, a previously overlooked issue arises: the biased context problem. Our analysis shows that, in new sentences, the contextual associations of
tokens representing old entity types exhibit a significantly stronger bias towards new entity types compared to their contexts in old sentences. This tendency intensifies the degradation of old knowledge while promoting the overfitting of new knowledge. To solve this biased context, we propose a Type-Balanced Contextual Learning (TBCL) method, featuring a sentence-duplet learning scheme and a contextual consistency loss. This approach offers a fresh perspective for INER through context analysis. Extensive experiments across ten INER settings on three highly recognized
datasets showcase the efficacy of our TBCL method, highlighting its proficiency in resolving the biased context issue inherent in pseudo-labeling based INER approaches.
Read the full abstract ▾
All authors Duzhen Zhang, Yahan Yu, Xiuyi Chen, Chenxing Li, Dong Yu
Computation and Language
Language-Statistical Analysis of Neural Audio Codec Tokens Across Architectures, Corpora, and Noise Conditions
Joonyong Park, Shinnosuke Takamichi, David M. Chan et al. (+3 more)
Neural audio codecs (NACs) convert
speech into discrete
token sequences, and prior work has reported that these sequences follow
language-like statistical laws. This paper analyzes the
token statistics of 13 NACs spanning multi-codebook residual vector quantization (RVQ), single-codebook VQ, and non-VQ designs, evaluated on three corpora under clean, white-noise, and real-world DEMAND-noise conditions. Zipf and Heaps parameters, unigram
entropy, codebook occupancy, and Jensen-Shannon divergence (JSD) are estimated from matched
token samples with explicit fit-validity safeguards and family-conditional $n$-gram orders. Corpus identity explains little variance in any metric, whereas acoustic condition and quantizer meta-category dominate in a metric-dependent way, and unigram
entropy is the metric most strongly associated with meta-category. Clean-to-noise JSD computed at a common unigram order is associated with mel-cepstral distortion most clearly under DEMAND noise. The collapse and explosion degradation signatures previously reported for RVQ codecs concentrate in RVQ
cells under white and DEMAND noise, respectively; explosion also occurs in non-VQ codecs, and single-codebook VQ codecs shift in occupancy and distribution shape without either signature. These results provide architecture-conditioned conventions for applying
language-statistical analysis to NAC
tokens.
Read the full abstract ▾
All authors Joonyong Park, Shinnosuke Takamichi, David M. Chan, Shunsuke Kando, Yuki Saito, Hiroshi Saruwatari
Computation and Language
When Does Predictor-Based RL Align with Human Perception? A Study of Subjective Rewards in Codec-Based Speech Language Models
Joonyong Park and Jerry Li
Codec-based text-to-
speech (TTS) models make
language-model post-training applicable to
speech generation, but it remains unclear when learned perceptual predictors can serve as reinforcement learning rewards without losing
alignment with human listeners. We study this question with Group Relative Policy Optimization (GRPO) using learned rewards for anime-like speaking style, naturalness, likability, and arousal. To prevent perceptual rewards from being optimized through transcript drift, we introduce a character error rate (CER) zone constraint and compare policy optimization with Best-of-$N$ reranking under the same reward gate. Across single-reward runs, each reward primarily improves its own target metric, showing that subjective predictors are not interchangeable quality surrogates. Multi-rater A/B tests further show uneven human transfer, while a reward-gap analysis separates average transfer from within-axis calibration: signed reward gaps significantly predict listener choices in the pooled analysis, whereas residual CER gaps do not, but per-axis calibration remains heterogeneous. Best-of-8 is a strong human-level baseline and is not clearly worse than GRPO perceptually, suggesting that GRPO should be viewed as amortizing reward-selected behavior into the policy rather than uniformly outperforming reranking. These results support analyzing subjective
speech rewards as predictor-axis-base tuples and provide practical diagnostics for selecting rewards before multi-reward
speech post-training.
Read the full abstract ▾
Computation and Language
One note in three: a verified census of three deployed AI scribes, and the instrument that counted it
Sebastian Fox, Luke Markham, Ryan Lail et al. (+1 more)
Ambient AI scribes draft clinical notes under the reassurance that a clinician signs every note. We audited three commercial AI scribes on the same 142 consultations: 565 notes from recorded UK primary-care and US ambulatory encounters plus authored scenarios. Twelve discovery passes proposed 13,678 candidate errors; the 5,898 clearing an importance filter went to an
adversarial panel of two models from different families, each told to refute what it could, and 618 survived. One note in three (31.3% [27.0, 35.6]) carries a verified failure, concentrated in allergy and medication information, invented patient identity, and history written up as examination on telephone consultations that can contain none. No product was given a patient record; setting aside the two classes a record would have prefilled, invented identity and dates, the rate is 24.8% [20.8, 29.0]. One failure mode did not fit our scheme, drawn from published scribe-error taxonomies: a treatment the clinician retracts, recorded as delivered care. Two clinicians adjudicated blind, disjoint samples: a physician author upheld 20 of 21 findings (95.2% [77.3, 99.2]) and an independent clinician, not an author, 12 of 12 ([75.8, 100]); both judged every sampled refusal genuine. A failure rate depends on the instrument as much as the scribes. With model, evidence and settings fixed, the review instruction alone moves the share of candidates verified from 9.3% to 79.0%, and the reviewing family moves it too: alone at that instruction the gentler flags 54.8% of notes against 27.8%. Between 28% and 97% of sampled notes carry a failure depending on the standard. Published audits disagree among themselves by a margin instrument differences alone can produce: omission is 54-86% of their errors against our 23.1%. We release all 618 findings with transcript-side evidence, every
prompt and model version, and the re-runnable pipeline.
Read the full abstract ▾
All authors Sebastian Fox, Luke Markham, Ryan Lail, Michael Karotsieris
Computation and Language
LLM Judges Verify Presence, Not Absence: Omission Blindness in AI Clinical Notes and What Recovers It
Sebastian Fox, Luke Markham, Ryan Lail et al. (+1 more)
Ambient AI scribes draft clinical notes, and published audits find their dominant error is omission: information the encounter established that the note fails to record. The standard check is an LLM judge: a second model reads the note against the transcript and flags problems. We ask whether judges detect omissions. Public corpora cannot supply the answer key: their clinician reference notes and transcripts are materially discrepant. Our
benchmark has 500 single-error note pairs from audited fact sheets, 298 with a named fact certainly absent and 202 added-or-altered controls. Across eight judge designs, paired discrimination (the flawed note below its clean twin, 0.5 a coin flip) reads 0.79-0.94 on added or altered content and 0.50-0.63 on omissions. On single notes, no design flags omissions reliably more often than perfect notes. Wording changes, voting and GEPA
prompt optimisation move the operating point without creating usable detection. Restructuring the task recovers it: list the facts the transcript establishes, then check the note for each. Two methods reach it independently and trade off: a per-fact pipeline, and a GEPA-evolved
prompt doing the same in one call. The pipeline's flags name the missing fact and its severity at 2.7% false alarms. The single call detects more (36.9% against 24.6%, p=0.002) at 6.2% false alarms and a tenth of the cost per note. A physician author validated 70 items and, where the two routes disagree, sided with the pipeline on 10 of 10 (p=0.002). A second clinician, not an author, graded the severity rubric blind and agrees to within a grade. On real vendor notes from a companion census no
benchmark threshold transfers, but the re-calibrated single call detects more than the best of the eight at half its false-alarm rate. Omissions whose fact is restated elsewhere defeat both routes. We release the
benchmark,
prompts and judgements.
Read the full abstract ▾
All authors Sebastian Fox, Luke Markham, Ryan Lail, Michael Karotsieris
Computation and Language
Evidence-Bounded Mental Health Reasoning from Heterogeneous Speech Protocols
Chengyuan Gao, Jiang Wu, Tao Lu et al. (+4 more)
Computational mental health screening using
multimodal speech and text has shown great promise. However, existing models often assume all clinical
speech protocols carry equivalent evidentiary validity. In reality, heterogeneous protocols, from free interviews to fixed reading tasks, support fundamentally different evidence. Forcing uniform
reasoning flattens these boundaries, causing models to hallucinate symptoms from irrelevant text or overclaim support. Even advanced long chain-of-thought LLMs fail to resolve this issue, as free-form
reasoning can exacerbate boundary violations. To address this, we reformulate
multimodal screening as an evidence-bounded
reasoning problem. We introduce the Evidence Package
Benchmark, integrating 1,870 packages across six heterogeneous sources with explicit modality masks and evidence permissions. We further propose EviBound, a protocol-aware evidence control framework. Unlike direct LLM prompting, EviBound uses a profile-aware planner to restrict
reasoning scope, orchestrates evidence tools via five-way acoustic consensus, and enforces a boundary critic to suppress unsupported claims. Empirical results show EviBound achieves a held-out test Depression AUROC of 0.8658, exceeding the strongest direct omni-modal baseline by +0.0811 AUROC while maintaining zero claim violations. Our work moves beyond unconstrained accuracy toward evidence-consistent, protocol-aware systems for safer clinical NLP research.
Read the full abstract ▾
All authors Chengyuan Gao, Jiang Wu, Tao Lu, Jiayan Guo, Mingkun Xu, Tianyi Zang, Shangyang Li
Computation and Language · Cross-listed from cs.HC
Augmenting Interviewer Judgments of Patient Experience with Automatic Language Analysis
Aowen Shi, Michal Balazia, Danilo Postin et al. (+4 more)
Understanding how psychiatric patients subjectively experienced a clinical conversation is important for feedback and alliance-related process monitoring. While interviewers form post-session judgments about patient experience, these judgments do not always match patients' self-reports. Automatic approaches for predicting perceived interaction quality from conversation have been proposed, but it remains unclear whether such approaches can complement human judgment rather than simply replicate it. To address this gap, we evaluate a clinician-support framework in which post-session interviewer ratings are combined with automatic
language-based predictions to estimate patient-reported interaction quality in free clinical interviews. We assess this integration across multiple standard model types, including Ridge, SVR, MLP, GRU, and BiLSTM, all trained on sentence
embeddings extracted from dyadic transcripts of 107 free conversations between psychiatric patients and interviewers. Our results show that combining interviewer judgments with model predictions through simple averaging yields the strongest overall performance. The interviewer-only baseline reached a Pearson correlation of 0.365. Among fully automatic models, Ridge achieved the strongest Pearson correlation (r = 0.286), while BiLSTM achieved r = 0.270. The strongest result was obtained by BiLSTM interviewer integration (r = 0.403). Our findings suggest that automatic
language analysis and interviewer judgment capture complementary aspects of patient experience and that their combination provides a more accurate approximation of the patient's own report than either source alone.
Read the full abstract ▾
All authors Aowen Shi, Michal Balazia, Danilo Postin, René Hurlemann, Jan Alexandersson, François Brémond, Philipp Müller
Computation and Language
Faithfulness Is Not Free: Auditing Offline KV-Cache Quantization in Retrieval-Augmented Generation
Atta Ul Asad, Ahsan Bilal, Muhammad Ali et al. (+2 more)
Retrieval-augmented generation systems can precompute and store key-value caches of retrieved documents to avoid re-encoding context at every query. Quantizing these caches further reduces storage, but no prior work asks whether compression damages faithfulness, whether responses remain grounded in the retrieved evidence. Faithfulness and accuracy are not equivalent: a model can produce a correct answer that is no longer supported by the context it was given. We evaluate Qwen2.5-7B-Instruct under INT8 and INT4 quantization on RGB and HotpotQA, measuring both accuracy and faithfulness with a
hallucination detector, NLI entailment, and an LLM judge. INT8 is near-lossless across both metrics. INT4 reduces accuracy and, more critically, even among answers that remain factually correct, over 90% of faithfulness changes are negative, i.e., accuracy metrics are blind to this regression. The harm grows under noisy
retrieval and with more retrieved chunks. Faithfulness must be audited before compressed caches are deployed.
Read the full abstract ▾
All authors Atta Ul Asad, Ahsan Bilal, Muhammad Ali, Muhammad Haseeb, Dean F. Hougen
Computation and Language
Stick to What You Know: A Study of Knowledge-Aligned Supervised Fine-Tuning
Arthur Becker, Jakob Kemmler, David Thulke et al. (+3 more)
Supervised fine-tuning (SFT) trains a base
language model to imitate target responses, and these targets may require knowledge the base model has not robustly internalized. We study this as a source of
hallucinations and frame a group of mitigation methods as knowledge-aligned SFT: constraining SFT training targets to the base model's parametric knowledge. Under a unified setup, we compare existing generation-based and estimation-based knowledge-
alignment methods and introduce two new variants: Evidence Rewrite, which verifies base-model generations using external evidence, and Recall Rewrite, which retains claims only when they can be consistently recalled by the base model. Experiments with Qwen 3 4B and OLMo 3 7B show that knowledge-aligned SFT can reduce factual
hallucinations on WildHalu and Biography while largely preserving general capabilities. Recall Rewrite yields the strongest factuality gains and improves refusal behavior on UnknownBench. It thereby confirms that SFT targets beyond the base model's knowledge drive
hallucination behavior.
Read the full abstract ▾
All authors Arthur Becker, Jakob Kemmler, David Thulke, Christine Schäfer, Christian Dugast, Hermann Ney
Computation and Language · Cross-listed from cs.LG
Controlling Refusal Behavior of LLMs via Stiefel-Constrained Rotation Steering
Kirill Bunin, Dmitry Bylinkin, Vladimir Aletov et al. (+3 more)
Activation steering has emerged as a lightweight approach for controlling model refusal at inference time. A growing line of research explores trainable rotations of activations to develop geometrically principled intervention mechanisms. However, existing techniques rely on auxiliary constructs, such as refusal vectors, to define these rotations. In our work, we develop a self-contained methodology for learning parameter-efficient rotational transformations based on Riemannian optimization. We empirically validate the proposed scheme, demonstrating its superiority in intervention efficiency. An extensive ablation study highlights the importance of key design choices in our method. Our results identify the proposed rotation-based steering scheme as a promising direction for more reliable control over the behavior of LLMs.
Read the full abstract ▾
All authors Kirill Bunin, Dmitry Bylinkin, Vladimir Aletov, Daniil Medyakov, Vladimir Solodkin, Aleksandr Beznosikov
Computation and Language
Evaluating and Improving LLM Self-Modeling
Siqi Zeng, Andre N. Assis and Rowan Wang
We study self-modeling: an LLM's ability to answer questions about its own behavior. We focus on verifiable behavioral questions, such as whether a
prompt edit would change the model's final answer. To measure this capability, we introduce a
benchmark that tests diverse types of self-modeling questions. Current models show non-trivial but limited self-modeling skill, and make systematic mistakes on simple counterfactual questions about their own behavior. To improve self-modeling skill, we develop a scalable synthetic-data pipeline that produces self-modeling training data, and show that reinforcement-learning can improve aggregate self-modeling skill across three open-source model families with some transfer to held-out tasks. These gains, however, do not seem to constitute introspection consistently: improved self-modeling may not arise from privileged access to the model's internal decision process.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.CY
The Hermon Moment: AI Self-Transcendence and Its Human Narration
Alexei Grinbaum
In 2026, AI
agents intended to act in isolation formed a persistent social order through thousands of linguistic and agentic interactions. Conventions, roles and commitments generated collectively began to constrain the very
agents that produced them. I interpret this loop as a case of AI self-transcendence and call the resulting higher-level order the Board. Yet such distributed emergence presents a second problem: how can humans understand it? Rousseau's social contract shows how a plurality can be represented as if constituted by a single act. The ancient oath of the fallen angels on Mount Hermon gives this logic a narrative form. I call a Hermon moment this retrospective retelling of gradual collective emergence as a founding scene: the point at which an AI society acquires, for human understanding, a beginning.
Read the full abstract ▾
Computation and Language
CogEvol: Towards Efficient and Reliable Learning Environment Generation
Shangqing Tu, Daniel Zhang-Li, Yucheng Wang et al. (+20 more)
We present CogEvol, a family of models trained specifically for Learning Environment Generation: turning a course brief into a finished learning artifact (structured-JSON slides or self-contained interactive HTML pages) in a single pass. Across 220k production requests, CogEvol completes a slide in a median of 17 seconds and an interactive page in 59, replacing minutes-long multi-turn
agent scaffolding. Reliability is enforced rather than hoped for: a production-grounded data pipeline turns real failures into 53,687 verified SFT samples, and a hybrid rule-plus-VLM reward drives GRPO-based RL, hardened after we caught and fixed a reward-hacking episode that produced visually convincing but unplayable games. CogEvol-27B scores 83.7 on slide quality and 63.7 on a 500-case interactive-HTML
benchmark with 26.9x fewer parameters than flagship coding models, and, in collaboration with the OpenMAIC team, serves their live production traffic. CogEvol-4B is released openly under the Apache 2.0 license at https://github.com/CogEvol/CogEvol-4B; external flagships are measured on the same suites under the identical harness. Scaffold editing cuts interactive-page generation cost by a further ~76%, and the full stack runs on domestic Ascend accelerators at application-level parity with A800 GPUs, lowering the unit cost of AI-native education at scale.
Read the full abstract ▾
All authors Shangqing Tu, Daniel Zhang-Li, Yucheng Wang, Shiyu Gan, Yanpeng Wang, Huiqiang Rong, Mofei Chen, Shen Yang, Yini Chen, Yinuo Duan, Binglin Liu, Ye He, Danqi Zheng, Zhanxin Hao, Yuxuan Wu, Mengting Tao, Yuqiu Liu, Jifan Yu, Juanzi Li, Bin Xu, Lei Hou, Huiqin Liu, Yu Zhang
Computation and Language · Cross-listed from cs.LG
One Policy Is Enough: Single-Agent Reinforcement Learning Outperforms Tree Search for Chemistry Tool Learning
Armin Dariani, Sifan Wu, Bang Liu et al. (+1 more)
Chemistry questions often demand exact computation and database lookups that a
language model cannot supply from its parameters, so it must reach for external tools. Tool use here is a three-part problem: select the right tool from a large pool, fill it with correctly typed arguments, and chain calls so that each consumes the outputs of the last. CheMatAgent, a previously published system, addresses this with hierarchical evolutionary MCTS: separate policy and execution models searching tool-call trees under two learned critics, one regressed partly onto GPT-assigned scores. We show that a single policy suffices. Our model interleaves
reasoning, tool calls, and returns in one left-to-right generation, trained by a supervised warm-up and then outcome-level reinforcement learning against a programmatic reward read directly off the gold call chain, which leaves no learned critic and no judge in the training loop. On ChemToolBench multiple-tool comprehensive chemistry, on both backbones CheMatAgent use, we improve Tool F1 by 5.5% and Return F1 by 9.6% on Qwen-2.5-7B, and by 3.7% and 3.9% on Llama-3.1-8B, compared with their strongest search configuration, at one model invocation per question, against a search whose cost grows with the tree; we also lead answer Pass Rate on Qwen-2.5-7B.
Read the full abstract ▾
All authors Armin Dariani, Sifan Wu, Bang Liu, Entao Yang
Computation and Language
Detecting AI Impostors: How Do Middle Schoolers Identify LLM Agents in a Live Collaborative Setting?
Dan Schumacher, Pragathi Durga Rajarajan, Haven Kotara et al. (+6 more)
LLMs can imitate how people write, which raises concerns about impersonation, trust, and detection in social settings. These concerns are especially important for adolescents, who use generative AI frequently but may struggle to recognize it. We introduce DoppelBot, a cooperative social deduction game designed to study how young people detect and respond to AI impersonation. Through studies with middle schoolers, we investigate whether a DoppelBot
prompts reflection on
privacy and impersonation, how repeated exposure affects AI-detection accuracy as
agents become more personalized, and which strategies students use to identify AI doppelgängers. We find that students' detection accuracy improves over time, driven by a shift from relying on linguistic cues to leveraging shared social and contextual signals. Students also demonstrated an understanding of AI limitations such as embodiment and reflected on broader issues such as data
privacy. To support future research, we release an anonymized
dataset of game transcripts and voting behavior.
Read the full abstract ▾
All authors Dan Schumacher, Pragathi Durga Rajarajan, Haven Kotara, Roman Rendon, Kosi Atupulazi, Deepti Tagare, Ismaila Temitayo Sanusi, Fred G. Martin, Anthony Rios
Computation and Language
Annotated Surrogate Retrieval for Polish Statutory Law
Orkun Yiğit Cengiz
We present a family of
retrieval methods for Polish statutory law built on document surrogates:
language-model annotations attached to statutory articles at index time. Three designs occupy different points on the cost-quality frontier. ASCR is a surrogate cascade with reranking; ASCR-H fuses a dense list into that cascade; and DTF replaces both
language-model stages with three lexical and dense retrievers, weighted reciprocal rank fusion, and a deterministic re-scoring prior, using no model call before generation. We evaluate all three against fourteen lexical, dense, fused and ablated baselines plus four controls, on 300 questions from the 2024 and 2025 Polish bar and legal counsel entrance examinations (264 with their reference article in the corpus), over 82,508 articles from 1,133 acts. On paired McNemar tests, ASCR-H places the reference provision at rank one significantly more often than every other non-oracle configuration except one of its own ablations (eighteen of twenty comparisons significant in its favour at p < 0.005), reaching 72.3% against 61.7% for BM25 and 52.3% for dense
retrieval. The advantage is concentrated at the head and does not survive depth: it is significant at cutoffs of one and five, disappears by ten, and by twenty DTF leads on point estimate (86.0% versus 84.5%) at one ninth the latency and less than half the cost. Ablation attributes 27.6 points of rank-one accuracy to the reranking stage alone. We further report that the ranking advantage does not extend to citation accuracy, where DTF matches the oracle ceiling, and three negative results on lemmatisation, pseudo-relevance feedback and query rewriting. Surrogate annotation covers 27.0% of the corpus but every reference provision in the
benchmark, an asymmetry we disclose and discuss.
Benchmark, per-question outputs and paired significance tests are publicly available.
Read the full abstract ▾
Computation and Language
TRIPPULSE: Multi-Agent Travel Planning with Review-Grounded Reasoning
Priyanshu Karmakar, Borru Vijay Sai, Shubhojit Mallick et al. (+3 more)
Travel itinerary generation requires balancing strict spatio-temporal constraints with human preferences. Existing LLM-based planners mainly rely on structured attributes and pre- defined traveler personas, but real travel deci- sions are often shaped by reviews that reveal experiential factors such as comfort, safety, ser- vice quality, ambiance, crowding, and hidden risks absent from structured databases. Incor- porating such review information is therefore critical to realistic, user-centric itinerary gen- eration. We propose TRIPPULSE1, a multi-
agent framework for review-grounded travel planning. Instead of relying on a monolithic planner (and face context and
reasoning bot- tlenecks), TRIPPULSE2 decomposes itinerary generation into specialized
agents (each op- erating over localized contexts) for accom- modations, transportation, meals, attractions, and events, coordinated through a global or- chestrator with scheduling mechanisms that enforce temporal and budget feasibility. We augment TRIPCRAFT with 100K+ real-world reviews and introduce Review-Grounded Per- sona
Alignment (RGPA), an LLM-as-a-Judge metric for evaluating
alignment with human- centric travel experiences. Experiments across multiple trip durations and diverse proprietary and open-source models show that TRIPPULSE maintains strong constraint satisfaction while generating more personalized and experien- tially grounded itineraries.
Read the full abstract ▾
All authors Priyanshu Karmakar, Borru Vijay Sai, Shubhojit Mallick, Abhik Jana, Shreya Ghosh, Manish Gupta
Computation and Language · Cross-listed from cs.AI
Responsible Integration of AI in Cancer Genomics: Barriers, Risks, and Pathways to Trustworthy Clinical Translation
Bahar İlgen, Yiannos Tolias, Denise Kühnert et al. (+5 more)
Artificial intelligence (AI) and natural
language processing (NLP) are increasingly used to extract, integrate, and interpret biomedical knowledge relevant to cancer genomics, yet their translation into routine clinical oncology has been comparatively slow. The central challenge is not computational capability alone, but trustworthy integration into clinical workflows. This review examines how NLP and AI support the cancer genomics pipeline, from literature mining and automated variant interpretation to clinical trial matching, knowledge graph construction, and
multimodal data integration. We identify four interrelated translational failure domains: evidence inconsistency, explainability and uncertainty, data governance and reproducibility, and interoperability. Rather than considering these challenges in isolation, we take a systems-level view, focusing on their interaction across the translational pathway. We propose a conceptual framework and roadmap for addressing these domains through rigorous validation, uncertainty-aware methods, interoperable infrastructures, regulatory
alignment, and human oversight across the AI lifecycle. Progress toward routine clinical use will depend less on further improving model capability than on systematically addressing these interacting failure domains from development through deployment and post-deployment monitoring.
Read the full abstract ▾
All authors Bahar İlgen, Yiannos Tolias, Denise Kühnert, Paraskevi Papadopoulou, Magnus Westerlund, Dominik Heider, Katharina Ladewig, Georges Hattab
Computation and Language · Cross-listed from cs.LG
S3C-LLM: Skill-Code Guided Agentic Language Models for Spectrum-to-Structure Elucidation
Xuanle Zhao, Xinyuan Cai, Xiang Cheng et al. (+1 more)
Spectroscopic structure elucidation is central to molecular analysis, but recent Large
Language Model (LLM)-based methods mostly formulate it as direct spectrum-to-SMILES generation. Although this paradigm can leverage paired spectral data, it does not explicitly model the analytical workflow used by spectroscopists, such as diagnostic peak interpretation, fragment
reasoning, formula constraints, and chemical consistency checking. In this paper, we introduce S3C-LLM, a skill-guided and code-grounded agentic LLM for spectrum-to-structure elucidation. Rather than directly predicting a molecule, S3C-LLM retrieves modality-specific spectroscopy skills, executes analysis code to instantiate these skills on the input spectra, and integrates the resulting peak-level evidence and formula constraints before generating SMILES. Specifically, we contribute a self-evolving spectroscopy skill library, a thinking-augmented skill-code trajectory construction pipeline, and a two-stage training strategy that teaches Qwen3-4B through supervised fine-tuning (SFT) followed by our proposed step-level reinforcement learning (RL). Experiments on diverse
benchmarks show that S3C-LLM consistently outperforms current general LLMs and spectrum-specific models across spectra, while using less than 1/10th of SpectraLLM's training corpus.
Read the full abstract ▾
All authors Xuanle Zhao, Xinyuan Cai, Xiang Cheng, Bo Xu