Computation and Language · Cross-listed from cs.HC
Generative UI: LLMs are Effective UI Generators
Yaniv Leviathan, Dani Valevski, Matan Kalman et al. (+9 more)
AI models excel at creating content, but typically render it with static, predefined interfaces. Specifically, the output of LLMs is often a markdown "wall of text". Generative UI is a long standing promise, where the model generates not just the content, but the interface itself. Until now, Generative UI was not possible in a robust fashion. We demonstrate that when properly prompted and equipped with the right set of tools, a modern LLM can robustly produce high quality custom UIs for virtually any
prompt. When ignoring generation speed, results generated by our implementation are overwhelmingly preferred by humans over the standard LLM markdown output. In fact, while the results generated by our implementation are worse than those crafted by human experts, they are at least comparable in 50% of cases. We show that this ability for robust Generative UI is emergent, with substantial improvements from previous models. We also create and release PAGEN, a novel
dataset of expert-crafted results to aid in evaluating Generative UI implementations, as well as the results of our system for future comparisons. Interactive examples can be seen at https://generativeui.github.io
Read the full abstract ▾
All authors Yaniv Leviathan, Dani Valevski, Matan Kalman, Danny Lumen, Eyal Segalis, Eyal Molad, Shlomi Pasternak, Vishnu Natchu, Valerie Nygaard, Srinivasan Venkatachary, James Manyika, Yossi Matias
Computation and Language · Cross-listed from cs.LG
Frayed RoPE and Long Inputs: A Geometric Perspective
Davis Wertheimer, Aozhong Zhang, Derrick Liu et al. (+2 more)
Rotary Positional
Embedding (RoPE) is a widely adopted technique for encoding position in
language models, which, while effective, causes performance breakdown when input length exceeds training length. Prior analyses assert (rightly) that long inputs cause channels to rotate “out of distribution,” but it is not clear how extra rotation relates to or causes pathological behavior. Through empirical and theoretical analysis we advance a unified
geometric understanding of
attention behavior with RoPE. We find that
attention induces tight clustering of separated key and query latent point clouds, allowing for creation of sink
tokens: placeholders that allow
attention heads to avoid
token mixing when not required. RoPE applied to longer inputs damages this key/query cluster separation, producing pathological behavior by inhibiting sink
token functionality. From this
geometric perspective, we propose RoPE-ID (In Distribution), a straightforward modification that allows
attention layers to generalize to longer inputs out of the box: apply RoPE with high frequency to a subset of channels. We demonstrate the effectiveness of RoPE-ID for extended inputs using 1B and 3B parameter
Transformers on the LongBench and RULER information
retrieval benchmarks.
Read the full abstract ▾
All authors Davis Wertheimer, Aozhong Zhang, Derrick Liu, Penghang Yin, Naigang Wang
Computation and Language
MineDraft: A Framework for Batch Parallel Speculative Decoding
Zhenwei Tang, Arun Verma, Zijian Zhou et al. (+4 more)
Speculative decoding (SD) accelerates large
language model inference by using a smaller draft model to propose draft
tokens that are subsequently verified by a larger target model. However, the performance of standard SD is often limited by the strictly sequential execution of these drafting and verification stages. To address this, this paper proposes MineDraft, a batch parallel speculative decoding (PSD) framework designed to effectively hide drafting latency by overlapping it with verification. Our theoretical analysis shows that PSD is substantially more efficient than standard SD. MineDraft realizes the PSD through a novel batch-parallel design that maintains two batches of requests, overlapping drafting for one batch with verification for the other. Our experimental results show significant improvements of \alg{} in both throughput (up to 75%) and end-to-end latency (up to 39%) over standard SD. Furthermore, we have implemented MineDraft as a plugin for vLLM, demonstrating its practicality for production-ready inference systems.
Read the full abstract ▾
All authors Zhenwei Tang, Arun Verma, Zijian Zhou, Zhaoxuan Wu, Alok Prakash, Daniela Rus, Bryan Kian Hsiang Low
Computation and Language
Beyond Accuracy: An Explainability-Driven Analysis of Harmful Content Detection
Trishita Dhara and Siddhesh Sheth
Although automated harmful content detection systems are frequently used to monitor online platforms, moderators and end users frequently cannot understand the logic underlying their predictions. While recent studies have focused on increasing classification accuracy, little focus has been placed on comprehending why neural models identify content as harmful, especially when it comes to borderline, contextual, and politically sensitive situations. In this work, a neural harmful content detection model trained on the Civil Comments
dataset is analyzed explainability-drivenly. Two popular post-hoc explanation methods, Shapley Additive Explanations and Integrated Gradients, are used to analyze the behavior of a RoBERTa-based classifier in both correct predictions and systematic failure cases. Despite strong overall performance, with an area under the curve of 0.93 and an accuracy of 0.94, the analysis reveals limitations that are not observable from aggregate evaluation metrics alone. Integrated Gradients appear to extract more diffuse contextual attributions while Shapley Additive Explanations extract more focused attributions on explicit lexical cues. The consequent divergence in their outputs manifests in both false negatives and false positives. Qualitative case studies reveal recurring failure modes such as indirect toxicity, lexical over-attribution, or political discourse. The results suggest that explainable AI can foster human-in-the-loop moderation by exposing model uncertainty and increasing the interpretable rationale behind automated decisions. Most importantly, this work highlights the role of explainability as a transparency and diagnostic resource for online harmful content detection systems rather than as a performance-enhancing lever.
Read the full abstract ▾
Computation and Language
Real-Time Trustworthiness Scoring for LLM Structured Outputs and Data Extraction
Hui Wen Goh and Jonas Mueller
Structured Outputs from current LLMs exhibit sporadic errors, hindering enterprise AI deployment. We present CONSTRUCT, a real-time uncertainty
estimator that scores the trustworthiness of LLM Structured Outputs. Lower-scoring outputs are more likely to contain errors, enabling automatic prioritization of limited human review bandwidth. CONSTRUCT additionally scores the trustworthiness of each field within a Structured Output, helping reviewers quickly identify which parts of the output are incorrect. Our method is suitable for any LLM (including black-box LLM APIs without logprobs), does not require labeled training data or custom model deployment, and supports complex Structured Outputs with heterogeneous fields and nested JSON schemas. We also introduce one of the first public LLM Structured Output
benchmarks with reliable ground-truth values. Over this four-
dataset benchmark, CONSTRUCT detects errors in outputs from various LLMs (including Gemini 3 and GPT-5) with significantly higher precision/recall than existing techniques.
Read the full abstract ▾
Computation and Language
Learned but Not Expressed: Capability-Expression Dissociation in Large Language Models
Toshiyuki Shigemura
Large
language models (LLMs) demonstrate the capacity to reconstruct and trace learned content from their training data under specific elicitation conditions, yet this capability does not manifest in standard generation contexts. This empirical observational study examines the expression of non-causal, non-implementable solution types across 300
prompt-response generations spanning narrative and problem-solving task contexts. Drawing on recent findings regarding memorization contiguity and
alignment-induced discourse priors, we document a systematic dissociation between learned capability and expressed output. Across three distinct LLMs, ten task scenarios, and both creative narrative and practical advisory contexts, we documented zero instances of non-causal solution frames in generated outputs (0%, 95% CI: [0%, 1.2%]), despite verified reconstruction capability under conditional extraction. These findings challenge the prevailing assumption that training data presence directly predicts output probability, demonstrating instead that task-conditioned generation policies can comprehensively suppress learned content across diverse contexts. The results offer implications for understanding generation dynamics, output distribution control, and the behavioral boundaries of contemporary LLMs.
Read the full abstract ▾
Computation and Language
DynaRAG: Bridging Static and Dynamic Knowledge in Retrieval-Augmented Generation
Penghao Liang, Mengwei Yuan, Jianan Liu et al. (+4 more)
We present DynaRAG, a
retrieval-augmented generation (RAG) framework designed to handle both static and time-sensitive information needs through dynamic knowledge integration. Unlike traditional RAG pipelines that rely solely on static corpora, DynaRAG selectively invokes external APIs when retrieved documents are insufficient for answering a query. The system employs an LLM-based reranker to assess document relevance, a sufficiency classifier to determine when fallback is necessary, and Gorilla v2 – a state-of-the-art API calling model – for accurate tool invocation. We further enhance robustness by incorporating schema filtering via FAISS to guide API selection. Evaluations on the CRAG
benchmark demonstrate that DynaRAG significantly improves accuracy on dynamic questions, while also reducing
hallucinations. Our results highlight the importance of dynamic-aware routing and selective tool use in building reliable, real-world question-answering systems.
Read the full abstract ▾
All authors Penghao Liang, Mengwei Yuan, Jianan Liu, Jing Yang, Xianyou Li, Weiran Yan, Yichao Wu
Computation and Language · Cross-listed from eess.SP
NeuroNarrator: A Generalist EEG-to-Text Foundation Model for Clinical Interpretation via Spectro-Spatial Grounding and Temporal State-Space Reasoning
Guoan Wang, Shihao Yang, Jun-en Ding et al. (+2 more)
Electroencephalography (
EEG) provides a non-invasive window into neural dynamics at high temporal resolution and plays a pivotal role in clinical neuroscience research. Despite this potential, prevailing computational approaches to
EEG analysis remain largely confined to task-specific classification objectives or coarse-grained pattern recognition, offering limited support for clinically meaningful interpretation. To address these limitations, we introduce NeuroNarrator, the first generalist
EEG-to-text foundation model designed to translate electrophysiological segments into precise clinical narratives. A cornerstone of this framework is the curation of NeuroCorpus-160K, the first harmonized large-scale resource pairing over 160,000
EEG segments with structured, clinically grounded natural-
language descriptions. Our architecture first aligns temporal
EEG waveforms with spatial topographic maps via a rigorous contrastive objective, establishing spectro-spatially grounded representations. Building on this grounding, we condition a Large
Language Model through a state-space-inspired formulation that integrates historical temporal and spectral context to support coherent clinical narrative generation. This approach establishes a principled bridge between continuous signal dynamics and discrete clinical
language, enabling interpretable narrative generation that facilitates expert interpretation and supports clinical reporting workflows. Extensive evaluations across diverse
benchmarks and zero-shot transfer tasks highlight NeuroNarrator's capacity to integrate temporal, spectral, and spatial dynamics, positioning it as a foundational framework for time-frequency-aware, open-ended clinical interpretation of electrophysiological data.
Read the full abstract ▾
All authors Guoan Wang, Shihao Yang, Jun-en Ding, Hao Zhu, Feng Liu
Computation and Language
Training-Free Agentic AI: Probabilistic Control and Coordination in Multi-Agent LLM Systems
Mohammad Parsa Hosseini, Ankit Shah, Saiyra Qureshi et al. (+3 more)
Multi-
agent large
language model (LLM) systems enable complex, long-horizon
reasoning by composing specialized
agents, but practical deployment remains hindered by inefficient routing, noisy feedback, and high interaction cost. We introduce REDEREF, a lightweight and training-free controller for multi-
agent LLM collaboration that improves routing efficiency during recursive delegation. REDEREF integrates (i) belief-guided delegation via Thompson sampling to prioritize
agents with historically positive marginal contributions, (ii) reflection-driven re-routing using a calibrated LLM or programmatic judge, (iii) evidence-based selection rather than output averaging, and (iv)
memory-aware priors to reduce cold-start inefficiency. Across multi-
agent split-knowledge tasks, we show that while recursive retry alone saturates task success, belief-guided routing reduces
token usage by 28%,
agent calls by 17%, and time-to-success by 19% compared to random recursive delegation, and adapts gracefully under
agent or judge degradation. These results demonstrate that simple, interpretable probabilistic control can meaningfully improve the efficiency and robustness of multi-
agent LLM systems without training or fine-tuning.
Read the full abstract ▾
All authors Mohammad Parsa Hosseini, Ankit Shah, Saiyra Qureshi, Alex Huang, Connie Miao, Wei Wei
Computation and Language
Steering at the Source: Style Modulation Heads for Robust Persona Control
Yoshihiro Izawa, Gouki Minegishi, Koshi Eguchi et al. (+2 more)
Activation steering offers a computationally efficient mechanism for controlling Large
Language Models (LLMs) without fine-tuning. While effectively controlling target traits (e.g., persona), coherency degradation remains a major obstacle to safety and practical deployment. We hypothesize that this degradation stems from intervening on the residual stream, which indiscriminately affects aggregated features and inadvertently amplifies off-target noise. In this work, we identify a sparse subset of
attention heads (only three heads) that independently govern persona and style formation, which we term Style Modulation Heads. Specifically, these heads can be localized via
geometric analysis of internal representations, combining layer-wise cosine similarity and head-wise contribution scores. We demonstrate that intervention targeting only these specific heads achieves robust behavioral control while significantly mitigating the coherency degradation observed in residual stream steering. More broadly, our findings show that precise, component-level localization enables safer and more precise model control.
Read the full abstract ▾
All authors Yoshihiro Izawa, Gouki Minegishi, Koshi Eguchi, Sosuke Hosokawa, Kenjiro Taura
Computation and Language · Cross-listed from cs.IR
OpenSanctions Pairs: Large-Scale Entity Matching with LLMs
Chandler Smith, Magnus Sesodia, Friedrich Lindenberg et al. (+1 more)
We release OpenSanctions Pairs, the first large-scale public
benchmark for entity matching on sanctions and OSINT data. The
dataset includes 755,540 expert-labeled pairs over 1 million entities, aggregated from 293 source
datasets across 45 jurisdictions. It captures real-world diversity in compliance data, spanning multiple
languages and writing systems (e.g., Latin, Cyrillic, Arabic), inconsistent structure, and time-varying provenance, and is substantially more heterogeneous than prior entity matching
benchmarks. As baselines, we evaluate the production rule-based matcher (nomenklatura RegressionV1) alongside open- and closed-source LLMs in both zero- and few-shot settings, each tested with and without MIPROv2
prompt optimization to control for
prompt sensitivity. The rule-based baseline reaches 91.3% F1; GPT-4o achieves the best result at 99.0% F1, and a locally deployable open-source model (DeepSeek-R1-Distill-Qwen-14B) achieves 98.2% F1. The rule-based baseline and LLMs fail in complementary ways: rules over-match, while LLMs struggle with cross-script transliteration. These results suggest that pairwise matching performance is approaching a practical ceiling and shift
attention toward pipeline components such as blocking, clustering, and uncertainty-aware review.
Read the full abstract ▾
All authors Chandler Smith, Magnus Sesodia, Friedrich Lindenberg, Christian Schroeder de Witt
Computation and Language
MERRY: Semantically Decoupled Evaluation of Multimodal Emotional and Role Consistencies of Role-Playing Agents
Zhenyu Wang, Xiaofen Xing, Yirong Chen et al. (+1 more)
Multimodal Role-Playing
Agents (MRPAs) are attracting increasing
attention due to their ability to deliver more immersive
multimodal emotional interactions. However, existing studies still rely on pure textual
benchmarks to evaluate the text responses of MRPAs, while delegating the assessment of their
multimodal expressions solely to modality-synthesis metrics. This evaluation paradigm, on the one hand, entangles semantic assessment with modality generation, leading to ambiguous error attribution, and on the other hand remains constrained by the heavy reliance on human judgment. To this end, we propose MERRY, a semantically decoupled evaluation framework for assessing
Multimodal Emotional and Role consistencies of Role-playing
agents. This framework introduce five refined metrics for EC and three for RC. Notably, we transform the traditional subjective scoring approach into a novel bidirectional-evidence-finding task, significantly improving the human agreement of LLM-as-Judge evaluations. Based on MERRY, we conduct extensive evaluations. Our empirical results primarily reveal that: (1) Training on synthetic
datasets tends to reduce emotional consistency, whereas training on real-world
datasets improves it; (2) Existing models suffer from emotional templatization and simplification, exhibiting positive-bias and performance bottleneck in fine-grained negative
emotions; (3) Simple prompting method strengthens the weak models but constrains the strong ones, while simple fine-tuning method suffers from poor role generalization. Codes and
dataset are available.
Read the full abstract ▾
All authors Zhenyu Wang, Xiaofen Xing, Yirong Chen, Xiangmin Xu
Computation and Language · Cross-listed from cs.CR
Adversarial Intent is a Latent Variable: Stateful Trust Inference for Securing Multimodal Agentic RAG
Inderjeet Singh, Vikas Pahuja, Aishvariya Priya Rathina Sabapathy et al. (+8 more)
Current stateless defences for
multimodal agentic RAG fail to detect
adversarial strategies that distribute malicious semantics across
retrieval, planning, and generation components. We formulate this security challenge as a Partially Observable Markov Decision Process (POMDP), where
adversarial intent is a latent variable inferred from noisy multi-stage observations. We introduce MMA-RAG^T, an inference-time control framework governed by a Modular Trust
Agent (MTA) that maintains an approximate belief state via structured LLM
reasoning. Operating as a model-agnostic overlay, MMA-RAGT mediates a configurable set of internal checkpoints to enforce stateful defence-in-depth. Extensive evaluation on 43,774 instances demonstrates a 6.50x average reduction factor in Attack Success Rate relative to undefended baselines, with negligible utility cost. Crucially, a factorial ablation validates our theoretical bounds: while statefulness and spatial coverage are individually necessary (26.4 pp and 13.6 pp gains respectively), stateless multi-point intervention can yield zero marginal benefit under homogeneous stateless filtering when checkpoint detections are perfectly correlated.
Read the full abstract ▾
All authors Inderjeet Singh, Vikas Pahuja, Aishvariya Priya Rathina Sabapathy, Chiara Picardi, Amit Giloni, Roman Vainshtein, Andrés Murillo, Hisashi Kojima, Motoyoshi Sekiya, Yuki Unno, Junichi Suga
Computation and Language · Cross-listed from cs.AI
The Headless Firm: How AI Reshapes Enterprise Boundaries
Tassilo Klein and Sebastian Wieczorek
The boundary of the firm is determined by coordination cost. We argue that agentic AI induces a structural change in how coordination costs scale: in prior modular systems, integration cost grew with interaction
topology (O(n^2) in the number of components); in protocol-mediated agentic systems, integration cost collapses to O(n) while verification scales with task throughput rather than interaction count. This shift selects for a specific organizational
equilibrium – the Headless Firm – structured as an hourglass: a personalized generative interface at the top, a standardized protocol waist in the middle, and a competitive
market of micro-specialized execution
agents at the bottom. We formalize this claim as a coordination cost model with two falsifiable empirical predictions: (1) the marginal cost of adding an execution provider should be approximately constant in a mature hourglass ecosystem; (2) the ratio of total coordination cost to task throughput should remain stable as ecosystem size grows. We derive conditions for hourglass stability versus re-centralization and analyze implications for firm size distributions, labor
markets, and software economics. The analysis predicts a domain-conditional Great Unbundling: in high knowledge-velocity domains, firm size distributions shift mass from large integrated incumbents toward micro-specialized
agents and thin protocol orchestrators.
Read the full abstract ▾
Computation and Language
MrBERT: Modern Multilingual Encoders via Vocabulary, Domain, and Dimensional Adaptation
Daniel Tamayo, Iñaki Lacunza, Paula Rivera-Hidalgo et al. (+4 more)
We introduce MrBERT, a family of 150M-300M parameter encoders built on the ModernBERT architecture and pre-trained on 35
languages and code. Through targeted adaptation, this model family achieves state-of-the-art results on Catalan- and Spanish-specific tasks, while establishing robust performance across specialized biomedical and legal domains. To bridge the gap between research and production, we incorporate Matryoshka Representation Learning (MRL), enabling flexible vector sizing that significantly reduces inference and storage costs. Ultimately, the MrBERT family demonstrates that modern encoder architectures can be optimized for both localized linguistic excellence and efficient, high-stakes domain specialization. We open source the complete model family on Huggingface.
Read the full abstract ▾
All authors Daniel Tamayo, Iñaki Lacunza, Paula Rivera-Hidalgo, Severino Da Dalt, Javier Aula-Blasco, Aitor Gonzalez-Agirre, Marta Villegas
Computation and Language
Beyond Subtokens: A Rich Character Embedding for Low-resource and Morphologically Complex Languages
Felix Schneider, Maria Gogolev, Sven Sickert et al. (+1 more)
Tokenization and sub-tokenization based models like word2vec, BERT and the GPTs are the state-of-the-art in natural
language processing. Typically, these approaches have limitations with respect to their input representation. They fail to fully capture orthographic similarities and morphological variations, especially in highly inflected and under-resource
languages. To mitigate this problem, we propose to computes word vectors directly from character strings, integrating both semantic and syntactic information. We denote this
transformer-based approach Rich Character
Embeddings (RCE). Furthermore, we propose a hybrid model that combines
transformer and convolutional mechanisms. Both vector representations can be used as a drop-in replacement for dictionary- and subtoken-based word
embeddings in existing model architectures. It has the potential to improve performance for both large context-based
language models like BERT and small models like word2vec for under-resourced and morphologically rich
languages. We evaluate our approach on various tasks like the SWAG, declension prediction for inflected
languages, metaphor and chiasmus detection for various
languages. Our experiments show that it outperforms traditional
token-based approaches on limited data using OddOneOut and TopK metrics.
Read the full abstract ▾
All authors Felix Schneider, Maria Gogolev, Sven Sickert, Joachim Denzler
Computation and Language
Small Language Models for Privacy-Preserving Clinical Information Extraction in Low-Resource Languages
Mohammadreza Ghaffarzadeh-Esfahani, Nahid Yousefian, Ebrahim Heidari-Farsani et al. (+4 more)
Extracting clinical information from medical transcripts in low-resource
languages remains a significant challenge in healthcare natural
language processing (NLP). This study evaluates a two-step pipeline combining Aya-expanse-8B as a Persian-to-English translation model with five open-source small
language models (SLMs) – Qwen2.5-7B-Instruct, Llama-3.1-8B-Instruct, Llama-3.2-3B-Instruct, Qwen2.5-1.5B-Instruct, and Gemma-3-1B-it – for binary extraction of 13 clinical features from 1,221 anonymized Persian transcripts collected at a cancer palliative care call center. Using a few-shot prompting strategy without fine-tuning, models were assessed on macro-averaged F1-score, Matthews Correlation Coefficient (MCC), sensitivity, and specificity to account for class imbalance. Qwen2.5-7B-Instruct achieved the highest overall performance (median macro-F1: 0.899; MCC: 0.797), while Gemma-3-1B-it showed the weakest results. Larger models (7B–8B parameters) consistently outperformed smaller counterparts in sensitivity and MCC. A bilingual analysis of Aya-expanse-8B revealed that translating Persian transcripts to English improved sensitivity, reduced missing outputs, and boosted metrics robust to class imbalance, though at the cost of slightly lower specificity and precision. Feature-level results showed reliable extraction of physiological symptoms across most models, whereas psychological complaints, administrative requests, and complex somatic features remained challenging. These findings establish a practical,
privacy-preserving blueprint for deploying open-source SLMs in
multilingual clinical NLP settings with limited infrastructure and annotation resources, and highlight the importance of jointly optimizing model scale and input
language strategy for sensitive healthcare applications.
Read the full abstract ▾
All authors Mohammadreza Ghaffarzadeh-Esfahani, Nahid Yousefian, Ebrahim Heidari-Farsani, Ali Akbar Omidvarian, Sepehr Ghahraei, Atena Farangi, AmirBahador Boroumand
Computation and Language · Cross-listed from cs.LG
Black-Box Reliability Certification for AI Agents via Self-Consistency Sampling and Conformal Calibration
Charafeddine Mouzouni
Given a black-box AI system and a task, at what confidence level can a practitioner trust the system's output? We answer with a reliability level – a single number per system-task pair, derived from self-consistency sampling and conformal calibration, that serves as a black-box deployment gate with exact, finite-sample, distribution-free guarantees. Self-consistency sampling reduces uncertainty exponentially; conformal calibration guarantees correctness within 1/(n+1) of the target level, regardless of the system's errors – made transparently visible through larger answer sets for harder questions. Weaker models earn lower reliability levels (not accuracy – see Definition 2.4): GPT-4.1 earns 94.6% on GSM8K and 96.8% on TruthfulQA, while GPT-4.1-nano earns 89.8% on GSM8K and 66.5% on MMLU. We validate across five
benchmarks, five models from three families, and both synthetic and real data. Conditional coverage on solvable items exceeds 0.93 across all configurations; sequential stopping reduces API costs by around 50%.
Read the full abstract ▾
Computation and Language
Alignment-Weighted DPO: A principled reasoning approach to improve safety alignment
Mengxuan Hu, Vivek V. Datla, Anoop Kumar et al. (+4 more)
Recent advances in
alignment techniques such as Supervised Fine-Tuning (SFT), Reinforcement Learning from Human Feedback (RLHF), and Direct Preference Optimization (DPO) have improved the safety of large
language models (LLMs). However, these LLMs remain vulnerable to jailbreak attacks that disguise harmful intent through indirect or deceptive phrasing. Using causal intervention, we empirically demonstrate that this vulnerability stems from shallow
alignment mechanisms that lack deep
reasoning, often rejecting harmful
prompts without truly understanding why they are harmful. To mitigate this vulnerability, we propose enhancing
alignment through
reasoning-aware post-training. We construct and release a novel Chain-of-Thought (CoT) fine-tuning
dataset that includes both utility-oriented and safety-critical
prompts with step-by-step rationales. Fine-tuning on this
dataset encourages models to produce principled refusals grounded in
reasoning, outperforming standard SFT baselines. Furthermore, inspired by failure patterns in CoT fine-tuning, we introduce
Alignment-Weighted DPO, which targets the most problematic parts of an output by assigning different preference weights to the
reasoning and final-answer segments. This produces finer-grained, targeted updates than vanilla DPO and improves robustness to diverse jailbreak strategies. Extensive experiments across multiple safety and utility
benchmarks show that our method consistently improves
alignment robustness while maintaining overall model utility.
Read the full abstract ▾
All authors Mengxuan Hu, Vivek V. Datla, Anoop Kumar, Zihan Guan, Sheng Li, Alfy Samuel, Daben Liu
Computation and Language
ToolMATH: A Diagnostic Benchmark for Long-Horizon Tool Use under Systematic Tool-Catalog Constraints
Hyeonje Choi, Jeongsoo Lee, Hyojun Lee et al. (+1 more)
We introduce \ToolMATH, a math-grounded diagnostic
benchmark for evaluating long-horizon tool use under controllable tool-catalog conditions. \ToolMATH converts stepwise MATH solutions into reusable Python tools with natural-
language descriptions and typed schemas, and pairs each problem with a tool environment requiring sequential tool use, intermediate-output reuse, and logically connected tool-call chains. \ToolMATH controls tool availability and catalog difficulty by constructing gold tools and graded distractors with varying similarity to gold tools. \ToolMATH also incorporates behavior-conditioned metrics, enabling diagnostic evaluation beyond final accuracy. Building on these measurements, \ToolMATH emphasizes three evaluation axes: (1) Adaptability measures how much Gold-only success is retained when gold tools are replaced entirely by distractors; (2) Robustness measures stability under adding distractors as a noise; and (3) Tool Connectivity measures whether models preserve accuracy over long executed tool-call chains. Furthermore, trace-level failure analyses characterize how models fail under each tool-catalog condition. Together, these diagnostics reveal distinct model profiles: reliable tool use, tool avoidance, adaptive substitution, and impacts of unreliable tool catalogs. Overall, \ToolMATH provides a controlled testbed for evaluating how
language models adapt to changing tool availability, remain robust to distractors, and maintain correctness across long-horizon tool-use trajectories.
Read the full abstract ▾
All authors Hyeonje Choi, Jeongsoo Lee, Hyojun Lee, Jay-Yoon Lee
Computation and Language
Under the Influence: Evaluating and Improving LLM Vigilance in Agentic Tasks
Sasha Robinson, Katherine M. Collins, Ilia Sucholutsky et al. (+1 more)
Large
Language Models (LLMs) are becoming increasingly autonomous assistants, capable of accomplishing series of complex tasks by interacting with open world domains such as software databases and the internet. This presents great opportunity but also great risk: open domains routinely present opportunities for autonomous
agents to be influenced by others, e.g. deciding whether to accept pull requests or whether to use advice on a message board in order to accomplish a task. Critically,
agents must be capable of deciphering when to incorporate benevolent advice into their
reasoning, and when to reject malicious advice. This involves the social capacity of vigilance: the ability to determine which information to use, and which to discard, for effective communication. While well-studied in humans, the extent to which LLM
agents exhibit vigilance is relatively unknown. Here, we examine LLM
agent vigilance in three realistic tasks: navigating dangerous websites, avoiding e-commerce scams, and patching an unsafe code repository. We find that all frontier models are highly susceptible to malicious influence across all environments. We also find that model performance itself is surprisingly dissociable from vigilance. In light of these results, we propose a vigilance harness which uses
Bayesian reasoning over a model's estimates of the value of an advisor's advice to decide whether to act on that advice. The vigilance harness helps recover baseline performance while retaining the benefit of positive influence on
agents, paving the way for safer agentic AI systems.
Read the full abstract ▾
All authors Sasha Robinson, Katherine M. Collins, Ilia Sucholutsky, Kelsey R. Allen
Computation and Language · Cross-listed from cs.IR
Multi-Vector Index Compression in Any Modality
Hanxiang Qin, Alexander Martin, Rohan Jha et al. (+3 more)
We study efficient multi-vector
retrieval for late interaction in any modality. Late interaction has emerged as a dominant paradigm for information
retrieval in text, images, visual documents, and
videos, but its computation and storage costs grow linearly with document length, making it costly for image-,
video-, and audio-rich corpora. To address this limitation, we explore query-agnostic methods for compressing multi-vector document representations under a constant vector budget. We introduce four approaches for index compression: sequence resizing,
memory tokens, hierarchical pooling, and a novel
attention-guided clustering (AGC). AGC uses an
attention-guided mechanism to identify the most semantically salient regions of a document as cluster centroids and to weight
token aggregation. Evaluating these methods on
retrieval tasks spanning text (BEIR), visual-document (ViDoRe), and
video (MSR-VTT, MultiVENT 2.0), we show that
attention-guided clustering consistently outperforms other parameterized compression methods (sequence resizing and
memory tokens), provides greater flexibility in index size than non-parametric hierarchical clustering, and achieves competitive or improved performance compared to a full, uncompressed index. The source code is available at: github.com/hanxiangqin/omni-col-press.
Read the full abstract ▾
All authors Hanxiang Qin, Alexander Martin, Rohan Jha, Chunsheng Zuo, Reno Kriz, Benjamin Van Durme
Computation and Language · Cross-listed from cs.AI
Aletheia tackles FirstProof autonomously
Tony Feng, Junehyuk Jung, Sang-hyun Kim et al. (+14 more)
We report the performance of Aletheia (Feng et al., 2026b), a mathematics research
agent powered by Gemini 3 Deep Think, on the inaugural FirstProof challenge. Within the allowed timeframe of the challenge, Aletheia autonomously solved 6 problems (2, 5, 7, 8, 9, 10) out of 10 according to majority expert assessments; we note that experts were not unanimous on Problem 8 (only). For full transparency, we explain our interpretation of FirstProof and disclose details about our experiments as well as our evaluation. Raw
prompts and outputs are available at https://github.com/google-deepmind/superhuman/tree/main/aletheia.
Read the full abstract ▾
All authors Tony Feng, Junehyuk Jung, Sang-hyun Kim, Carlo Pagano, Sergei Gukov, Chiang-Chiang Tsai, David Woodruff, Adel Javanmard, Aryan Mokhtari, Dawsen Hwang, Yuri Chervonyi, Jonathan N. Lee, Garrett Bingham, Trieu H. Trinh, Vahab Mirrokni, Quoc V. Le, Thang Luong
Computation and Language · Cross-listed from cs.LG
Learning from Trials and Errors: Reflective Test-Time Planning for Embodied LLMs
Yining Hong, Huang Huang, Manling Li et al. (+4 more)
Embodied LLMs endow
robots with high-level task
reasoning, but they cannot reflect on what went wrong or why, turning deployment into a sequence of independent trials where mistakes repeat rather than accumulate into experience. Drawing upon human reflective practitioners, we introduce Reflective Test-Time Planning, which integrates two modes of reflection: reflection-in-action, where the
agent uses test-time scaling to generate and score multiple candidate actions using internal reflections before execution; and reflection-on-action, which uses test-time training to update both its internal reflection model and its action policy based on external reflections after execution. We also include retrospective reflection, allowing the
agent to re-evaluate earlier decisions and perform model updates with hindsight for proper long-horizon credit assignment. Experiments on our newly-designed Long-Horizon Household
benchmark and MuJoCo Cupboard Fitting
benchmark show significant gains over baseline models, with zero-shot generalization to photorealistic HM3D environments and real-
robot experiments on a Franka Panda arm. Ablations confirm that reflection-in-action and reflection-on-action are mutually dependent, and that retrospective reflection achieves better credit assignment than step-wise external feedback at lower computational overhead. Qualitative analyses further highlight behavioral correction through reflection.
Read the full abstract ▾
All authors Yining Hong, Huang Huang, Manling Li, Li Fei-Fei, Leonidas Guibas, Jiajun Wu, Yejin Choi
Computation and Language
On Data Engineering for Scaling LLM Terminal Capabilities
Renjie Pi, Grace Lam, Mohammad Shoeybi et al. (+3 more)
Despite rapid recent progress in the terminal capabilities of large
language models, the training data strategies behind state-of-the-art terminal
agents remain largely undisclosed. We address this gap through a systematic study of data engineering practices for terminal
agents, making two key contributions: (1) Terminal-Task-Gen, a lightweight synthetic task generation pipeline that supports seed-based and skill-based task construction, and (2) a comprehensive analysis of data and training strategies, including filtering, curriculum learning, long context training, and scaling behavior. Our pipeline yields Terminal-Corpus, a large-scale open-source
dataset for terminal tasks. Using this
dataset, we train Nemotron-Terminal, a family of models initialized from Qwen3(8B, 14B, 32B) that achieve substantial gains on Terminal-Bench 2.0: Nemotron-Terminal-8B improves from 2.5% to 13.0% Nemotron-Terminal-14B improves from 4.0% to 20.2%, and Nemotron-Terminal-32B improves from 3.4% to 27.4%, matching the performance of significantly larger models. To accelerate research in this domain, we open-source our model checkpoints and most of our synthetic
datasets at https://huggingface.co/collections/nvidia/nemotron-terminal.
Read the full abstract ▾
All authors Renjie Pi, Grace Lam, Mohammad Shoeybi, Pooya Jannaty, Bryan Catanzaro, Wei Ping