Computation and Language
Uncertainty-Aware Structured Data Extraction from Full CMR Reports via Distilled LLMs
Yi Yu, Parker Martin, Zhenyu Bu et al. (+5 more)
Converting free-text cardiac
magnetic resonance (CMR) reports into auditable structured data remains a bottleneck for cohort assembly, longitudinal curation, and clinical decision support. We present CMR-EXTR, a lightweight framework that converts free-text CMR reports into structured data and assigns per-field confidence for quality control. A teacher-student
distillation pipeline enables fully offline inference while limiting manual annotation. Uncertainty integrates three complementary principles – distribution plausibility, sampling stability, and cross-field consistency – to triage human review. Experiments show that CMR-EXTR achieves 99.65% variable-level accuracy, demonstrating both reliable extraction and informative confidence scores. To our knowledge, this is the first CMR-specific extraction system with integrated confidence estimation. The code is available at https://github.com/yuyi1005/CMR-EXTR.
Read the full abstract ▾
All authors Yi Yu, Parker Martin, Zhenyu Bu, Yixuan Liu, Yi-Yu Zheng, Orlando Simonetti, Yuchi Han, Yuan Xue
Computation and Language
Julie Kallini, Artidoro Pagnoni, Tomasz Limisiewicz et al. (+5 more)
Recent byte-level
language models (LMs) match the performance of
token-level models without relying on subword vocabularies, yet their utility is limited by slow, byte-by-byte autoregressive generation. We address this bottleneck in the Byte Latent
Transformer (BLT) through new training and generation techniques. First, we introduce BLT
Diffusion (BLT-D), a new model and our fastest BLT variant, trained with an auxiliary block-wise
diffusion objective alongside the standard next-byte prediction loss. This enables an inference procedure that generates multiple bytes in parallel per decoding step, substantially reducing the number of forward passes required to generate a sequence. Second, we propose two extensions inspired by speculative decoding that trade some of this speed for higher generation quality: BLT Self-speculation (BLT-S), in which BLT's local decoder continues generating past its normal patch boundaries to draft bytes, which are then verified with a single full-model forward pass; and BLT
Diffusion+Verification (BLT-DV), which augments BLT-D with an autoregressive verification step after
diffusion-based generation. All methods may achieve an estimated
memory-bandwidth cost over 50% lower than BLT on generation tasks. Each approach offers its own unique advantages, together removing key barriers to the practical use of byte-level LMs.
Read the full abstract ▾
All authors Julie Kallini, Artidoro Pagnoni, Tomasz Limisiewicz, Gargi Ghosh, Luke Zettlemoyer, Christopher Potts, Xiaochuang Han, Srinivasan Iyer
Computation and Language · Cross-listed from cs.LG
Position: Mechanistic Interpretability Must Disclose Identification Assumptions for Causal Claims
Zezheng Lin and Fengming Liu
Mechanistic interpretability papers increasingly use causal vocabulary:
circuits, mediators, causal abstraction, monosemanticity. Such claims require explicit identification assumptions. A purposive audit of 10 papers across four methodological strands finds no dedicated identification-assumptions section and a recurring pattern: validation metrics such as faithfulness, completeness, monosemanticity,
alignment, or ablation effects are reported as causal support without stating the assumptions that make them identifying. A two-human-coder audit on $n=30$ reproduces the direction of the main finding: dedicated identification sections are absent, and validation-metric substitution is common, though exact Dim B/D counts are coding-rule sensitive. The paper proposes a disclosure norm: state whether the claim is causal, name the identification strategy, enumerate assumptions, stress at least one, and explain how conclusions shift if assumptions fail. Validation is not identification.
Read the full abstract ▾
Computation and Language
Tool Calling is Linearly Readable and Steerable in Language Models
Zekun Wu, Ze Wang, Seonglae Cho et al. (+4 more)
Language-model
agents can take real actions by calling tools, so choosing the wrong tool can cause errors that are difficult to undo. Most evaluations only observe the tool choice after the model generates a call. We read this choice from the model before generation, and we steer it. For each tool, we average the model's hidden states from a few example requests to obtain a tool vector. Comparing a new request with these tool vectors predicts which tool it needs. The difference between two tool vectors gives a steering direction that can move the model toward a chosen tool without retraining. Across eight instruction-tuned models from 4B to 27B parameters, steering changes the generated call to a chosen target tool in 56-78% of held-out tool pairs, depending on the model. We also observe steering on real APIs from $\tau$-bench and ToolBench, though less reliably. At the final layer, steering could work simply by raising the score of the target tool name. To test this, we compare the steering direction with a direction that only raises that score, layer by layer. In the middle layers, on the three models we examine in depth, the steering direction switches more calls than the score-raising direction. So the tool vectors capture part of the tool choice before the final layer. The same tool vectors also help identify likely tool-selection errors before generation. Calls are more likely to be wrong when a request is similarly close to two tool vectors. Across four models, this signal achieves an AUROC of 0.61-0.78 and outperforms a first-
token confidence baseline on three of them. Together, these results suggest that the model's hidden state provides a way to read, steer, and check tool choice before a call is made.
Read the full abstract ▾
All authors Zekun Wu, Ze Wang, Seonglae Cho, Yufei Yang, Adriano Koshiyama, Sahan Bulathwela, Maria Perez-Ortiz
Computation and Language
GLiGuard: Schema-Conditioned Classification for LLM Safeguard
Urchade Zaratiana, Mary Newhauser, George Hurn-Maloney et al. (+1 more)
Ensuring safe, policy-compliant outputs from large
language models requires real-time content moderation that can scale across multiple safety dimensions. However, state-of-the-art guardrail models rely on autoregressive decoders with 7B–27B parameters, reformulating what is fundamentally a classification problem as sequential text generation, a design choice that incurs high latency and scales poorly to multi-aspect evaluation. In this work, we introduce GLiGuard, a 0.3B-parameter schema-conditioned bidirectional encoder adapted from GLiNER2 for LLM content moderation. The key idea is to encode task definitions and label semantics directly into the input sequence as structured
token schemas, enabling simultaneous evaluation of
prompt safety, response safety, refusal detection, 14 fine-grained harm categories, and 11 jailbreak strategies in a single non-autoregressive forward pass. This schema-conditioned design lets supported task and label blocks be composed directly in the input schema at inference time. Across nine established safety
benchmarks, GLiGuard achieves F1 scores competitive with 7B–27B decoder-based guards despite being 23–90$\times$ smaller, while delivering up to 16$\times$ higher throughput and 17$\times$ lower latency. These results suggest that compact bidirectional encoders can approach the accuracy of much larger guard models while drastically reducing inference cost. Code and models are available at https://github.com/fastino-ai/GLiGuard.
Read the full abstract ▾
All authors Urchade Zaratiana, Mary Newhauser, George Hurn-Maloney, Ash Lewis
Computation and Language
Ask Early, Ask Late, Ask Right: When Does Clarification Timing Matter for Long-Horizon Agents?
Anmol Gulati, Hariom Gupta, Elias Lumer et al. (+2 more)
Long-horizon AI
agents execute complex workflows spanning hundreds of sequential actions, yet a single wrong assumption early on can cascade into irreversible errors. When instructions are incomplete, the
agent must decide not only whether to ask for clarification but when, and no prior work measures how clarification value changes over the course of execution. We introduce a forced-injection framework that provides ground-truth clarifications at controlled points in the
agent's trajectory across four information dimensions (goal, input, constraint, context), three
agent benchmarks, and four frontier models (three per
benchmark; one on a single
benchmark only; 84 task variants; 6,000+ runs). Counter to the common intuition that "earlier is always better," we find that the value of clarification depends sharply on what information is missing: goal clarification loses nearly all value after 10% of execution (pass@3 drops from 0.78 to baseline), while input clarification retains value through roughly 50%. Deferring any clarification type past mid-trajectory degrades performance below never asking at all. Cross-model Kendall tau correlations (0.78-0.87 among models sharing identical task coverage; 0.34-0.67 across the full 4-model panel) confirm these timing profiles are substantially task-intrinsic. A complementary study of 300 unscripted sessions reveals that no current frontier model asks within the empirically optimal window, with strategies ranging from over-asking (52% of sessions) to never asking at all. These empirical demand curves provide the quantitative foundation that existing theoretical frameworks require but have lacked, and establish concrete design targets for timing-aware clarification policies. Code and data will be publicly released.
Read the full abstract ▾
All authors Anmol Gulati, Hariom Gupta, Elias Lumer, Sahil Sen, Vamse Kumar Subbiah
Computation and Language
How to Train Your Latent Diffusion Language Model Jointly With the Latent Space
Viacheslav Meshchaninov, Alexander Shabalin, Egor Chimbulatov et al. (+4 more)
Latent
diffusion models offer an attractive alternative to discrete
diffusion for non-autoregressive text generation by operating on continuous text representations and denoising entire sequences in parallel. The major challenge in latent
diffusion modeling is constructing a suitable latent space. In this work, we present the Latent
Diffusion Language Model (LDLM), in which the latent encoder,
diffusion model, and decoder are trained jointly. LDLM builds its latent space by reshaping the representations of a pre-trained
language model with a trainable encoder, yielding latents that are easy to both denoise and decode into
tokens. We show that naive joint training produces a low-quality
diffusion model, and propose a simple training recipe consisting of an MSE decoder loss,
diffusion-to-encoder warmup, adaptive timestep sampling, and decoder-input noise. Ablations show that each component substantially impacts generation performance. On OpenWebText and LM1B, LDLM achieves better generation performance than existing discrete and continuous
diffusion language models while being $2{\text -}13\times$ faster, indicating that jointly learning the latent space is a key step toward making latent
diffusion competitive for text generation.
Read the full abstract ▾
All authors Viacheslav Meshchaninov, Alexander Shabalin, Egor Chimbulatov, Nikita Gushchin, Ilya Koziev, Alexander Korotin, Dmitry Vetrov
Computation and Language
How Value Induction Reshapes LLM Behaviour
Arnav Arora, Natalie Schluter, Katherine Metcalf et al. (+1 more)
Conversational Large
Language Models are post-trained on
language that expresses specific behavioural traits, such as curiosity, open-mindedness, and empathy, and values, such as helpfulness, harmlessness, and honesty. This is done to increase utility, ensure safety, and improve the experience of the people interacting with the model. However, values are complex and inter-related – inducing one could modify behaviour on another. Further, inducing certain values can make models more addictive or sycophantic through
language used in the generations, with a potential detrimental effect on the user. We investigate these and other unintended effects of value induction into models. We fine-tune models using curated value subsets of existing preference
datasets, measuring the impact of value induction on expression of other values, model safety, anthropomorphic
language, and various QA
benchmarks. We find that (i) inducing values leads to expression of other related, and sometimes contrastive values, (ii) inducing positive values increases safety, and (iii) all values increase anthropomorphic
language use, making models more validating and sycophantic.
Read the full abstract ▾
All authors Arnav Arora, Natalie Schluter, Katherine Metcalf, Maartje ter Hoeve
Computation and Language · Cross-listed from cs.LG
Trajectory as the Teacher: Few-Step Discrete Flow Matching via Energy-Navigated Distillation
Amin Karimi Monsefi, Dominic Culver, Nikhil Bhendawade et al. (+3 more)
Discrete flow matching generates text by iteratively transforming noise
tokens into coherent
language, but may require hundreds of forward passes.
Distillation uses the multi-step trajectory to train a student to reproduce the process in a few steps. When the student underperforms, the usual explanation is insufficient capacity. We argue the opposite: the trajectory is the bottleneck, not the student. Each training trajectory is built through a chain of blind
stochastic jumps with no evaluation of sequence quality; a single bad decision at an early midpoint propagates through subsequent steps, yet the student must imitate the result. Trajectory-Shaped Discrete Flow Matching (TS-DFM) replaces these blind jumps with guided
navigation: a lightweight energy compass evaluates candidate continuations at each midpoint, selecting the most coherent. All shaping is training-only; inference cost is unchanged. On 170M-parameter
language modeling, the shaped student at 8 steps achieves 32% lower perplexity than the 1,024-step teacher while being 128x faster, with gains consistent across source distributions and three evaluators of increasing scale. TS-DFM achieves the best perplexity of any discrete-generation baseline we compare against, including methods trained on 6x more data or using 5x larger models.
Read the full abstract ▾
All authors Amin Karimi Monsefi, Dominic Culver, Nikhil Bhendawade, Manuel R. Ciosici, Yizhe Zhang, Irina Belousova
Computation and Language
CoCoReviewBench: A Completeness- and Correctness-Oriented Benchmark for AI Reviewers
Hexuan Deng, Xiaopeng Ke, Yichen Li et al. (+6 more)
Despite the rapid development of AI reviewers, evaluating such systems remains challenging: metrics favor overlap with human reviews over correctness. However, since human reviews often cover only a subset of salient issues and sometimes contain mistakes, they are unreliable as gold references. To address this, we build category-specific
benchmark subsets and skip evaluation when the corresponding human reviews are missing to strengthen Completeness. We also leverage reviewer–author–meta-review discussions as expert annotations and filter unreliable reviews accordingly to strengthen Correctness. Finally, we introduce CoCoReviewBench, which curates 3,900 papers from ICLR and NeurIPS to enable reliable and fine-grained evaluation of AI reviewers. Analysis shows that AI reviewers remain limited in correctness and are prone to
hallucinations, and highlights
reasoning models as more effective reviewers, motivating further directions for improving AI reviewers.
Benchmarks and models are available at https://github.com/hexuandeng/CoCoReviewBench.
Read the full abstract ▾
All authors Hexuan Deng, Xiaopeng Ke, Yichen Li, Ruina Hu, Dehao Huang, Derek F. Wong, Yue Wang, Xuebo Liu, Min Zhang
Computation and Language
Beyond "I cannot fulfill this request": Alleviating Rigid Rejection in LLMs via Label Enhancement
Ying Zhang, Congyu Qiao, Xin Geng et al. (+1 more)
Large
Language Models (LLMs) rely on safety
alignment to obey safe requests while refusing harmful ones. However, traditional refusal mechanisms often lead to "rigid rejection," where a general template (e.g., "I cannot fulfill this request") indiscriminately triggers refusals and severely undermines the naturalness of interactions between humans and LLMs. To address this issue, LANCE is proposed in this paper to ensure safe yet flexible and natural responses via label enhancement. Specifically, LANCE employs variational inference to perform label enhancement, predicting a continuous distribution across multiple rejection categories. These fine-grained rejection distributions provide multi-way textual gradients for a refinement model to neutralize the hazardous elements in the
prompt, so that the LLMs could generate safe responses that avoid rigid rejections while preserving the naturalness of interactions. Experiments demonstrate that LANCE significantly alleviates the rigid rejection problem while maintaining high security standards, significantly outperforming existing baseline models in terms of helpfulness and naturalness of responses.
Read the full abstract ▾
All authors Ying Zhang, Congyu Qiao, Xin Geng, Ning Xu
Computation and Language · Cross-listed from cs.LG
KL for a KL: On-Policy Distillation with Control Variate Baseline
Minjae Oh, Sangjun Song, Gyubin Choi et al. (+2 more)
On-Policy
Distillation (OPD) has emerged as a dominant post-training paradigm for Large
Language Models, especially in
reasoning domains. However, OPD remains unstable in practice due to the high gradient variance of its single-sample Monte Carlo
estimator, and recipes for stable training are still immature. We propose vOPD (On-Policy
Distillation with a control variate baseline), which casts OPD as policy-gradient RL and stabilizes it by introducing a control variate baseline—canonically a value function—from the RL literature. We show that the OPD value function admits a closed form: the per-
token negative reverse KL divergence between student and teacher, available directly from the existing forward pass with no additional critic or inference. Existing stabilization methods either compute the full
token-level reverse KL over the entire vocabulary, adding significant overhead, or restrict it to a top-k support, biasing the objective. vOPD instead preserves the lightweight single-sample
estimator and subtracts the value function as a detached baseline to keep the gradient unbiased while reducing variance. Moreover, we show that a top-k approximation of the baseline further lowers cost without compromising performance. Across mathematics, science, and instruction-following
benchmarks, vOPD consistently outperforms base OPD and matches the most expensive full-vocabulary method, offering an efficient stabilization of OPD through principled RL variance reduction.
Read the full abstract ▾
All authors Minjae Oh, Sangjun Song, Gyubin Choi, Yunho Choi, Yohan Jo
Computation and Language
MatryoshkaLoRA: Learning Accurate Hierarchical Low-Rank Representations for LLM Fine-Tuning
Ionut-Vlad Modoranu, Mher Safaryan and Dan Alistarh
With the rise in scale for deep learning models to billions of parameters, the computational cost of fine-tuning remains a significant barrier to deployment. While Low-Rank Adaptation (LoRA) has become the standard for parameter-efficient fine-tuning, the need to set a predefined, static rank $r$ requires exhaustive grid searches to balance efficiency and performance. Existing rank-adaptive solutions such as DyLoRA mitigate this by sampling ranks during the training from a predefined distribution. However, they often yield sub-optimal results at higher ranks due to lack of consistent gradient signals across the full hierarchy of ranks, thus making these methods data-inefficient. In this paper, we propose MatryoshkaLoRA, a general, Matryoshka-inspired training framework for LoRA that learns accurate hierarchical low-rank representations by inserting a fixed, carefully crafted diagonal
matrix $P$ between the existing LoRA adapters to scale their sub-ranks accordingly. By introducing this simple modification, our general framework recovers LoRA and DyLoRA only by changing $P$ and ensures all sub-ranks embed the available gradient information efficiently. Our MatryoshkaLoRA supports dynamic rank selection with minimal degradation in accuracy. We further propose Area Under the Rank Accuracy Curve (AURAC), a metric that consistently evaluates the performance of hierarchical low-rank adapters. Our results demonstrate that MatryoshkaLoRA learns more accurate hierarchical low-rank representations than prior rank-adaptive approaches and achieves superior accuracy-performance trade-offs across ranks on the evaluated
datasets. Our code is available at https://github.com/IST-DASLab/MatryoshkaLoRA.
Read the full abstract ▾
Computation and Language
Measuring and Mitigating the Distributional Gap Between Real and Simulated User Behaviors
Shuhaib Mehri, Philippe Laban, Sumuk Shashidhar et al. (+4 more)
As user simulators are increasingly used for interactive training and evaluation of AI assistants, it is essential that they represent the diverse behaviors of real users. While existing works train user simulators to generate human-like responses, whether they capture the broad and heterogeneous distribution of real user behaviors remains an open question. In this work, we introduce a method to measure the distributional gap between real and simulated user behaviors, validated through a human study and ablations. Given a
dataset of real and simulated conversations, our method extracts representations of user behavior from each conversation, quantizes them into discrete distributions via clustering, then computes divergence metrics. We provide the first systematic evaluation of 24 LLM-based user simulators on coding and writing tasks, and reveal a large distributional gap from real users that varies across model families, scales, and behavioral facets. Pairwise comparisons show that most simulators behave similarly, while a few stand apart. Combining behaviorally complementary simulators brings the resulting distribution closer to real users compared to either simulator on its own. Finally, a TF-IDF analysis of the clusters surfaces interpretable patterns of behaviors that simulators capture, miss, and hallucinate.
Read the full abstract ▾
All authors Shuhaib Mehri, Philippe Laban, Sumuk Shashidhar, Marwa Abdulhai, Sergey Levine, Michel Galley, Dilek Hakkani-Tür
Computation and Language
SCENE: Recognizing Social Norms and Sanctioning in Group Chats
Mateusz Jacniacki and Maksymilian Bilski
Online group chats are social spaces with implicit behavior patterns that, when broken, are often met with social sanctioning from the group. The ability and willingness of LLM-based
agents to recognize and adapt to these norms remains mostly unexplored. We introduce SCENE, a social-interaction
benchmark focused on implicit norms and social sanctioning in multi-party chat. SCENE generates plausible non-roleplay scenarios with scripted personas that follow a hidden norm, create opportunities for the subject
agent to violate it, and sanction breaches when they occur. We further propose behavioral evaluation metrics for two functional adaptation abilities: responsiveness to negative sanctioning, and adapting norm from peers behavior. We evaluate six frontier and open-weight models on SCENE. Our results show that Claude Opus 4.7 and Gemini 3.1 Pro adapt to implicit norms significantly more than the evaluated open-weight models. SCENE contributes one
benchmark in the direction of recent calls for dynamic, interactional evaluation of LLM social capabilities.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.CV
Brown Ebouky, Gabriele Carrino, Niccolo Avogaro et al. (+3 more)
Human visual
reasoning is governed by active
vision, a process where meta-cognitive control drives top-down goal-directed
attention, dynamically routing foveal focus toward task-relevant details while maintaining peripheral awareness of the global scene. In contrast, modern
Vision-
Language Models (VLMs) process visual information passively, relying on the static accumulation of massive
token contexts that dilute the visual evidence as the
reasoning chain grows. Here we propose GazeVLM, a VLM that learns to exert this oversight over its
attention resources by generating gaze actions in its
reasoning chain, in the form of <LOOK> tags specifying the coordinates of the region to inspect. During training, each <LOOK> block triggers a continuous suppression bias on the
attention logits that dampens features outside the regions gazed at so far, and that lifts at the end of the block, restoring the global view. The model is trained with this bias as a teaching signal, first by Supervised Fine-Tuning on curated gaze-
reasoning traces, then through Group Relative Policy Optimization (GRPO) with rewards for correct answers and valid grounding. At deployment, the external suppression is removed and the model, running its unmodified forward pass, steers its own
attention to the regions indicated by the <LOOK> blocks, retaining most of the effect of the bias and emulating top-down control of spatial
attention. This model thus transitions between global spatial awareness and localized focal
reasoning without relying on external agentic tools like cropping, and without adding visual
tokens from re-encoded patches. Applied to 4B-parameter backbones, GazeVLM demonstrates strong high-resolution
multimodal reasoning on HRBench-4k and HRBench-8k, surpassing its base models by about 4 points, and outperforming agentic
multimodal pipelines built around thinking with images by 5 to 12 points.
Read the full abstract ▾
All authors Brown Ebouky, Gabriele Carrino, Niccolo Avogaro, Christoph Studer, Andrea Bartezzaghi, Mattia Rigotti
Computation and Language · Cross-listed from cs.LG
OrScale: Orthogonalised Optimization with Layer-Wise Trust-Ratio Scaling
Yuxuan Lou and Yang You
Muon fixes the direction of every
matrix-valued update at the polar factor of its momentum, while each layer's step magnitude is addressed only by a static shape correction. We derive a dynamic per-layer scalar by adapting the LARS/LAMB trust-ratio principle to the orthogonalized setting, where the standard denominator candidates—the raw momentum norm or the polar-factor norm—either live in the wrong unit space or carry no update-scale information. The resulting method, OrScale, uses the norm of the parameter-space direction actually applied and anchors each layer's ratio at one via a per-layer calibration, so that the Moonlight recipe (tuned for AdamW, shared with
Muon via RMS matching) transfers with no additional sweep; a component ablation confirms each design choice is individually load-bearing. Theoretically, OrScale retains a
nuclear-norm $O(1/\sqrt{T})$ convergence rate for any clipped multiplier and achieves a strict layer-adaptive descent gain $\kappa_{\mathrm{eff}}>1$ under two conditions estimable from standard training diagnostics—a bound that predicts the gain should grow with architectural heterogeneity. Experiments confirm the prediction: with every hyperparameter inherited verbatim from the Moonlight recipe, OrScale matches or beats
Muon+Moonlight across dense 125M–1.1B FineWeb-Edu pre-training, and on a 16B-A3B mixture-of-experts model—where the logged trust ratios separate cleanly by layer class—the gap widens by an order of magnitude to $0.130$ nats ($3.8\%$ relative) at parity wall-clock cost.
Read the full abstract ▾
Computation and Language
A Comparative Analysis of Classical Machine Learning and Deep Learning Approaches for Sentiment Classification on IMDb Movie Reviews
Erma Daniar Safitri, Lia Hana Ichisasmita, Citra Agustin et al. (+3 more)
This paper presents a comparative study of classical machine learning and deep learning methods for sentiment classification on the IMDb movie reviews
dataset. The machine learning pipeline uses TF-IDF features and PyCaret AutoML to evaluate Logistic Regression, Naïve Bayes, and Support Vector Machine, while the deep learning pipeline implements BiLSTM and BiLSTM with an
attention mechanism. Experimental results show that classical machine learning, especially SVM, achieves the best performance with an accuracy of 0.8530, outperforming the deep learning models in this study. The BiLSTM with
Attention model improves over the standard BiLSTM and reaches an accuracy of 0.706, indicating better contextual modeling. The paper concludes that although deep learning can capture sequential dependencies, classical machine learning remains a strong baseline when combined with effective feature engineering such as TF-IDF, particularly under limited data and computational resources.
Read the full abstract ▾
All authors Erma Daniar Safitri, Lia Hana Ichisasmita, Citra Agustin, Luluk Muthoharoh, Ardika Satria, Martin Clinton Tosima Manullang
Computation and Language
Beyond Confidence: Rethinking Self-Assessments for Performance Prediction in LLMs
Sree Bhattacharyya, Samarth Khanna, Leona Chen et al. (+3 more)
Large
Language Models (LLMs) are increasingly used in settings where reliable self-assessment is critical. Assessing model reliability has evolved from using probabilistic correctness estimates to, more recently, eliciting verbalized confidence. Confidence, however, has been shown to be an inconsistent and overoptimistic predictor of model correctness. Drawing on cognitive appraisal theory, a framework from human psychology that decomposes self-evaluation into multiple components, we propose a multidimensional perspective on model self-assessment. We elicit six appraisal-based dimensions of self-assessment, alongside confidence, and evaluate their utility for predicting model failure across 12 LLMs and 38 tasks spanning eight domains. We find that competence-related appraisal dimensions, particularly effort and ability, consistently match or outperform confidence across most settings. Effort additionally yields less overoptimistic estimates that remain stable across model sizes. In contrast, affective dimensions provide marginally predictive signals. Furthermore, the most informative dimension varies systematically with task characteristics: effort is most predictive for
reasoning-intensive tasks, while ability and confidence dominate on
retrieval-oriented tasks. Broadly, our findings indicate that structured multidimensional self-assessment is a promising approach to improving the reliability and safety of
language model deployment across diverse real-world settings.
Read the full abstract ▾
All authors Sree Bhattacharyya, Samarth Khanna, Leona Chen, Lucas Craig, Tharun Dilliraj, James Z. Wang
Computation and Language
PolySQL: Scaling Text-to-SQL Evaluation Across SQL Dialects via Automated Backend Isomorphism
Yotam Perlitz, Elad Venezian, Corentin Royer et al. (+2 more)
SQL dialects vary in syntax, types, and functions across database engines. Text-to-SQL
benchmarks, however, predominantly support only SQLite. This creates a critical evaluation gap: cross-dialect evaluation reveals weak per-query agreement (Cohen's ), showing that SQLite performance is an unreliable proxy for other dialects. Yet such evaluation remains prohibitively difficult: existing approaches either require expensive manual query transpilation or rely on tools that often fail on complex SQL. To close this gap, we introduce PolySQL, a novel dual-execution method that eliminates the need for query transpilation by comparing normalized execution results. Notably, our approach achieves higher evaluation fidelity than query transpilation with 100% query coverage. PolySQL comprises three
datasets, enabling the first large-scale cross-dialect study. Our study reveals a 10.1% average accuracy drop from SQLite to other dialects and identifies a significant dialect difficulty hierarchy. We find this degradation stems from logical rather than syntactic errors (61% vs. 8%). We release our framework code and leaderboard to enable rigorous dialect-robust evaluation.
Read the full abstract ▾
All authors Yotam Perlitz, Elad Venezian, Corentin Royer, Francesco Fusco, Andrea Giovannini
Computation and Language
Hybrid TF–IDF Logistic Regression and MLP Neural Baseline for Indonesian Three-Class Sentiment Analysis on Social Media Text
Allya Nurul Islami Pasha, Eka Fidiya Putri, Luluk Muthoharoh et al. (+2 more)
This paper presents a compact three-class sentiment analysis study for Indonesian social media text. The task is formulated with positive, negative, and neutral outputs derived from a fine-grained
emotion dataset. The proposed practical baseline combines TF–IDF text features, three lightweight numeric metadata features, and a balanced multinomial Logistic Regression classifier. For comparison, the study also includes a neural baseline using a two-layer multilayer perceptron (MLP) over the same hybrid feature representation. The
dataset originally contains 732 rows and 191 fine-grained
emotion labels; after cleaning, deduplication, and label remapping, 707 samples remain with an imbalanced distribution of 459 positive, 188 negative, and 60 neutral instances. Experimental results show that the Logistic Regression deployment model reaches 0.8028 accuracy, 0.8003 weighted F1, and 0.7276 macro F1, while project documentation reports a higher-accuracy but non-production MLP baseline. These findings indicate that careful preprocessing, interpretable feature engineering, and class balancing remain competitive for small Indonesian sentiment
datasets, whereas the neural baseline is better treated as a comparative experiment than as the default deployment model.
Read the full abstract ▾
All authors Allya Nurul Islami Pasha, Eka Fidiya Putri, Luluk Muthoharoh, Ardika Satria, Martin C.T. Manullang
Computation and Language
Chain-based Distillation for Effective Initialization of Variable-Sized Small Language Models
Boyu Shi, YiCheng Jiang, Chang Liu et al. (+3 more)
Large
language models (LLMs) achieve strong performance but remain costly to deploy in resource-constrained settings. Training small
language models (SLMs) from scratch is computationally expensive, while conventional knowledge
distillation requires repeated access to large teachers for different target sizes, leading to poor scalability. To solve these problems, we propose Chain-based
Distillation (CBD), a scalable paradigm for efficiently initializing variable-sized
language models. A sparse and limited sequence of intermediate models (called anchors) is constructed via stepwise
distillation, forming a
distillation chain that progressively transfers knowledge from the source LLMs. To support heterogeneous settings, we introduce bridge
distillation for cross-architecture and cross-vocabulary transfer. Models of variable sizes are initialized via parameter interpolation between adjacent anchors, eliminating repeated large teacher inference. Experiments show that the proposed method substantially improves efficiency and downstream performance. A 138M-parameter SLM without recovery pre-training, outperforms scratch-trained models on a 10B-
token corpus on the specific task. CBD also demonstrates versatility in heterogeneous settings for initialize models with different architectures and vocabularies.
Read the full abstract ▾
All authors Boyu Shi, YiCheng Jiang, Chang Liu, Qiufeng Wang, Xu Yang, Xin Geng
Computation and Language
CktFormalizer: Autoformalization of Natural Language into Circuit Representations
Jing Xiong, Qi Han, Chenchen Ding et al. (+5 more)
Hardware infrastructure is a critical bottleneck for LLM-driven
circuit design, limiting what
agents can express, compile, and iteratively refine within an agentic loop. To address this bottleneck, we introduce CKTLEAN, a typed hardware infrastructure embedded in Lean. It supports hardware description, compilation to SystemVerilog, and interactive type-checking and proof feedback through a persistent read-eval-print loop (REPL). On this foundation, we build CKTFORMALIZER, an
agent framework for hardware generation, repair, optimization, and source-level equivalence proving. We evaluate structural correctness through compilation, functional correctness through RTL and gate-level simulation, and formal correctness through proofs relative to stated specifications. Across VerilogEval, RTLLM, ResBench, and CVDP, CKTFORMALIZER with CKTLEAN achieves compilation rates of 91.1%-99.4%. Among designs that pass RTL simulation, 95.4%-100.0% jointly complete synthesis and place-and-route and pass design-rule and layout-versus-schematic checks. In a separate evaluation on 30 VerilogEval problems, interactive proof-state feedback raises kernel-accepted equivalence proof completion from 53.3% to 63.3%. Hardware evaluation feedback also guides architecture exploration and iterative power, performance, and area (PPA) optimization, with synthesis-area reductions of up to 58.6% in the optimization loop. These results suggest that typed representations and explicit proof-state feedback help structure the agentic loop: compiler diagnostics guide targeted repairs, while proof-state feedback helps
agents identify what remains to be proved and determine the next proof step. Project Page: https://ckt-formalizer.github.io/
Read the full abstract ▾
All authors Jing Xiong, Qi Han, Chenchen Ding, He Xiao, Zunhai Su, Chaofan Tao, Xiachong Feng, Ngai Wong
Computation and Language · Cross-listed from cs.LG
Nils Grünefeld, Bertram Højer, Philipp Mondorf et al. (+5 more)
Language model (LM) "
reasoning", commonly described as Chain-of-Thought or test-time scaling, often improves
benchmark performance, but the dynamics underlying this process remain poorly understood. We study these dynamics through the lens of uncertainty quantification by treating the "
reasoning" traces, the intermediate
token sequences generated by LMs, as evolving model states. We summarize each trace by an uncertainty trace profile: a small set of features describing the shape of the uncertainty signal over its trace, such as its slope and linearity. We find that across five LMs evaluated on GSM8K and ProntoQA, these profiles predict whether a trace yields a correct final answer with AUROC up to 0.807, improving markedly on recent related work. We reach AUROC 0.801 using only the first few hundred
tokens of full traces, suggesting that errors can be detected early in the generation. A detailed comparison of correct and incorrect traces further reveals qualitatively distinct uncertainty profiles, with correct traces showing a steeper and less linear decline in uncertainty. Together, the results suggest that our method, grounded in decision-making under uncertainty, provides a principled lens for studying the generative process underlying LM "
reasoning".
Read the full abstract ▾
All authors Nils Grünefeld, Bertram Højer, Philipp Mondorf, Barbara Plank, Anna Rogers, Christian Hardmeier, Stefan Heinrich, Jes Frellsen
Computation and Language · Cross-listed from cs.LG
Rethinking State Tracking in Recurrent Models Through Error Control Dynamics
Jiwan Chung, Heechan Choi and Seon Joo Kim
The theory of state tracking in recurrent architectures has predominantly focused on expressive capacity: whether a fixed architecture can theoretically realize a set of symbolic transition rules. We argue that equally important is error control, the dynamics governing hidden-state drift along the directions that distinguish symbolic states. We prove that affine recurrent networks, a class of models encompassing State-Space Models and Linear
Attention, cannot correct errors along state-separating subspaces once they preserve state representations. Consequently, practical affine trackers do not learn robust state tracking; rather, they learn finite-horizon solutions governed by accumulated state-relevant error. We characterize the mechanics of this failure, showing that tracking remains readable only while the accumulating within-class spread remains small relative to the between-class separation. We demonstrate empirically on group state-tracking tasks that this breakdown is predictable: tracking collapses when the distinguishability ratio crosses the readability threshold. Across trained models, the point of this crossing predicts the horizon at which downstream accuracy fails. These results establish that robust state tracking is determined not only by an architecture's theoretical expressivity but crucially by its error control.
Read the full abstract ▾
Computation and Language
TextLDM: Language Modeling with Continuous Latent Diffusion
Jiaxiu Jiang, Jingjing Ren, Wenbo Li et al. (+10 more)
Diffusion Transformers (DiT) trained with flow matching in a VAE latent space have unified visual generation across images and
videos. A natural next step toward a single architecture for both generation (visual synthesis) and understanding (text generation) is to apply this framework to
language modeling. We propose TextLDM, which transfers the visual latent
diffusion recipe to text generation with minimal architectural modification. A
Transformer-based VAE maps discrete
tokens to continuous latents, enhanced by Representation
Alignment (REPA) with a frozen pretrained
language model to produce representations effective for conditional denoising. A standard DiT then performs flow matching in this latent space, identical in architecture to its visual counterpart. The central challenge we address is obtaining high-quality continuous text representations: we find that reconstruction fidelity alone is insufficient, and that aligning latent features with a pretrained
language model via REPA is critical for downstream generation quality. Trained from scratch on OpenWebText2, TextLDM substantially outperforms prior
diffusion language models and matches GPT-2 under the same settings. Our results establish that the visual DiT recipe transfers effectively to
language, taking a concrete step toward unified
diffusion architectures for
multimodal generation and understanding.
Read the full abstract ▾
All authors Jiaxiu Jiang, Jingjing Ren, Wenbo Li, Bo Wang, Haoze Sun, Yijun Yang, Jianhui Liu, Yanbing Zhang, Shenghe Zheng, Yuan Zhang, Haoyang Huang, Nan Duan, Wangmeng Zuo
Computation and Language
Benchmarking EngGPT2-16B-A3B against Comparable Italian and International Open-source LLMs
Andrea Sassella, Andrea Chizzola, Tommaso Bianchi et al. (+2 more)
This report
benchmarks the performance of ENGINEERING Ingegneria Informatica S.p.A.'s EngGPT2MoE-16B-A3B LLM, a 16B parameter Mixture of Experts (MoE) model with 3B active parameters. Performance is investigated across a wide variety of representative
benchmarks, and is compared against comparably-sized open-source MoE and dense models. In comparison with popular Italian models, namely FastwebMIIA-7B, Minerva-7B, Velvet-14B, and LLaMAntino-3-ANITA-8B, EngGPT2MoE-16B-A3B performs as well or better on international
benchmarks: ARC-Challenge, GSM8K, AIME24, AIME25, MMLU, and HumanEval (HE). It achieves the best performance for the longest context setting (32k) of the RULER
benchmark. On the Italian
benchmark dataset ITALIC, the model performs as well or better than the other models except for Velvet-14B, which outperforms it. Compared with popular MoE models of comparable size, the new model reports higher values than DeepSeek-MoE-16B-Chat on all considered
benchmarks. It has higher values than Moonlight-16B-A3B on HE, MMLU, AIME24, AIME25, GSM8K, and the 32k RULER setting, but lower on BFCL and some ARC and ITALIC settings. Finally it has lower values than GPT-OSS-20B on most
benchmarks, including HE, MMLU, AIME24, AIME25, GSM8K, ARC, BFCL, and the RULER 32k. When compared with popular dense models, EngGPT2MoE-16B-A3B reports higher values on AIME24 and AIME25 than Llama-3.1-8B-Instruct, Gemma-3-12b-it, and Ministral-3-8BInstruct-2512-BF16, but lower values on ITALIC, BFCL, and RULER with a 32k context. When performance is aggregated across all
benchmark metrics, EngGPT2MoE-16B-A3B shows higher performance than the Italian models under evaluation while achieving lower results than some of the most performant international models, in particular GPT-5 nano and Qwen3-8B. Taken together, our findings find the new model to be a step forward for native Italian Large
Language Models.
Read the full abstract ▾
All authors Andrea Sassella, Andrea Chizzola, Tommaso Bianchi, Luca Alessandrelli, Mark James Carman
Computation and Language
Qiyong Zhong, Mao Zheng, Mingyang Song et al. (+5 more)
Tool-integrated
reasoning (TIR) is difficult to scale to small
language models due to instability in long-horizon tool interactions and limited model capacity. While reinforcement learning methods like group relative policy optimization provide only sparse outcome-level rewards. Recently, on-policy
distillation (OPD) has gained popularity by supplying dense
token-level supervision from a teacher on student-generated trajectories. However, our experiments indicate that applying OPD to TIR leads to a critical failure mode: erroneous tool calls tend to cascade across subsequent
reasoning steps, progressively amplifying student-teacher divergence and rendering the teacher's
token-level supervision increasingly unreliable. To address this, we propose SOD, a step-wise on-policy
distillation framework for small
language model
agents, which adaptively reweights
distillation strength at each step based on step-level divergence. Therefore, SOD can attenuate potentially misleading teacher signals in high-divergence regions while preserving dense guidance in well-aligned states. Experiments on challenging math, science, and code
benchmarks show that SOD achieves up to 20.86% improvement over the second-best baseline. Notably, our 0.6B student achieves 26.13% on AIME 2025, demonstrating effective transfer of agentic
reasoning to lightweight models. Our code is available at https://github.com/YoungZ365/SOD.
Read the full abstract ▾
All authors Qiyong Zhong, Mao Zheng, Mingyang Song, Xin Lin, Jie Sun, Houcheng Jiang, Xiang Wang, Junfeng Fang
Computation and Language
Victor Conchello Vendrell, Arnau Padres Masdemont, Niccolò Grillo et al. (+3 more)
Recurrent LLM architectures have emerged as a promising approach for improving
reasoning, as they enable multi-step computation in the
embedding space without generating intermediate
tokens. Models such as Ouro perform
reasoning by iteratively updating internal representations while retaining a standard Key-Value (KV) cache across iterations, causing
memory consumption to grow linearly with
reasoning depth. Consequently, increasing the number of
reasoning iterations can lead to prohibitive
memory usage, limiting the practical scalability of such architectures. In this work, we propose
Memory-Efficient Looped
Transformer (MELT), a novel architecture that decouples
reasoning depth from
memory consumption. Instead of using a standard KV cache per layer and loop, MELT maintains a single KV cache per layer that is shared across
reasoning loops. This cache is updated over time via a learnable gating mechanism. To enable stable and efficient training under this architecture, we propose to train MELT using chunk-wise training in a two phase procedure: interpolated transition, followed by
attention-aligned
distillation, both from the LoopLM starting model to MELT. Empirically, we show that MELT models fine-tuned from pretrained Ouro parameters outperform standard LLMs of comparable size, while maintaining a
memory footprint comparable to those models and dramatically smaller than Ouro's. Overall, MELT achieves constant-
memory iterative
reasoning without sacrificing LoopLM performance, using only a lightweight post-training procedure.
Read the full abstract ▾
All authors Victor Conchello Vendrell, Arnau Padres Masdemont, Niccolò Grillo, Jordi Ros-Giralt, Arash Behboodi, Fabio Valerio Massoli
Computation and Language
SimCT: Recovering Lost Supervision for Cross-Tokenizer On-Policy Distillation
Jie Sun, Mao Zheng, Mingyang Song et al. (+6 more)
On-policy
distillation (OPD) is a standard tool for transferring teacher behavior to a smaller student, but it implicitly assumes that teacher and student predictions are comparable
token by
token, an assumption that fails whenever the two models tokenize the same text differently. Under heterogeneous tokenizers, exact shared-
token matching silently discards a large fraction of the teacher signal at precisely the positions where vocabularies disagree. We propose \textbf{\underline{Sim}ple \underline{C}ross-\underline{T}okenizer OPD (SimCT)}, which restores this signal by enlarging the supervision space: alongside shared
tokens, SimCT compares teacher and student over short multi-
token continuations that both tokenizers can realize, leaving the OPD loss form itself unchanged. We show that these units are the finest jointly tokenizable supervision interface, and that coarser alternatives remove teacher-student distinctions that are useful for on-policy learning. Across three heterogeneous teacher-student pairs on mathematical
reasoning and code-generation
benchmarks, SimCT shows consistent gains over shared-vocabulary OPD and representative cross-tokenizer baselines, with ablations confirming that the improvements come from recovering supervision discarded by exact shared-
token matching. Code is available at \href{https://github.com/sunjie279/SimCT-}{https://github.com/sunjie279/SimCT-}.
Read the full abstract ▾
All authors Jie Sun, Mao Zheng, Mingyang Song, Qiyong Zhong, Yilin Cheng, Bichuan Feng, Pengfei Liu, Junfeng Fang, Xiang Wang