Computation and Language · Cross-listed from cs.CV
AVA-Encoder: Towards Agent-Native Video Representation Learning
Chuyue Li, Jinpeng Yu, Haozhe Wang et al. (+7 more)
Agentic
video creation systems remain far from cinematic-grade quality: their underlying models struggle to plan how story, characters, visuals, and sound fit together, and high-quality creation trajectories to learn such planning abilities are scarce. Finished human films embody this knowledge, but in a tightly coupled audiovisual form that
agents cannot directly read or manipulate. We propose the Agentic
Video Auto-Encoder (AVA-Encoder), which works backward from finished films, turning them into
agent-native representations with reconstruction itself as supervision. It encodes a film into a Film Knowledge Graph (Film KG) of structured-text nodes, typed dependency edges, and linked
multimodal assets, then regenerates the film from this graph with a fixed decoder. Reconstruction residuals are distilled into textual gradients that drive dual-loop self-
evolution: an outer loop learns a shared encoding policy across
videos, and an inner loop refines each
video's Film KG at test time. AVA-Encoder improves reconstruction fidelity by 20.7 percentage points over the strongest baseline; its learned policy outperforms a finely human-tuned policy with over 70 percent fewer system-
prompt tokens; and Film KGs support coordinated editing and improve all tested downstream creation systems. We release the framework, a reconstruction
benchmark, and a Film KG
dataset.
Read the full abstract ▾
All authors Chuyue Li, Jinpeng Yu, Haozhe Wang, Tian Xueyun, Zhijing Zhang, Bingnan Li, Shuqi Gu, Kan Ren, Jiaming Liu, Ruihua Huang
Computation and Language · Cross-listed from cs.LG
AI4AI at Test-Time: Strong-to-Weak Capability Transfer via Harnesses
Cheng Qian, Wenting Zhao, Liangwei Yang et al. (+6 more)
Recent work on
distillation transfers the capabilities of large models to smaller ones often by updating the latter's parameters, through teacher forcing, on-policy
distillation, and related training-time methods. In this paper, we ask whether such transfer can instead occur at test time. We study strong-to-weak scaffolding: whether a stronger builder model can construct inference-time harnesses that help a weaker target model solve tasks more reliably without any parameter updates. Using four representative Theory-of-Mind
benchmarks, each builder model uses 5% of the data as a validation set to iteratively refine its harness over multiple rounds, after which the finalized harness is evaluated on the full test set. Empirically, this form of test-time capability transfer is highly effective, nearly doubling average target-model performance from 0.49 to 0.91. Our analysis shows that the gains come primarily from offloading unstable model
reasoning into deterministic code,
benchmark-specific routing, and strict answer-format enforcement, rather than from encouraging the target model to reason more extensively or sample more broadly. We further find that builder-model
reasoning effort improves harness quality monotonically, platform effects are modest relative to the builder model's own capability, and weaker target models receive the largest gains. These results suggest that inference-time harness design is an important complement to conventional training-time
distillation, enabling strong models to transfer cognitive structure to weaker models without retraining.
Read the full abstract ▾
All authors Cheng Qian, Wenting Zhao, Liangwei Yang, Heng Wang, Jielin Qiu, Heng Ji, Silvio Savarese, Huan Wang, Shelby Heinecke
Computation and Language · Cross-listed from q-fin.PM
Large Language Model-Driven Small-Capitalization Trading: Integrating Financial News Sentiment, Macroeconomic Indicators, and Technical Signals
Alireza Kargarzadeh, Nariman Khaledian, Navid Parvini et al. (+1 more)
Large
language models can extract richer signals from financial news than fixed sentiment lexicons, and recent work has explored feeding such signals into portfolio construction. We study an uncertainty-aware construction that feeds model-predicted risk – decomposed into aleatoric and epistemic components – directly into the covariance
matrix of portfolio allocators, rather than treating portfolio risk as fixed or adjusting only expected returns. We evaluate the pipeline on Russell 2000 equities under three stock-selection regimes: a pure-alpha trigger that isolates abnormal stock moves not explained by macro indicators, a pure-beta trigger that captures macro-indicator moves before the stock itself fires, and a beta trigger in which both channels agree. Across the full holding-period grid, the separated pure-alpha and pure-beta legs usually dominate the beta intersection on Sharpe and return. Two horizons are especially informative. At one day, pure beta can work under low and moderate transaction costs because it captures immediate lead-lag spillovers from
liquid macro and sector indicators into exposed small-cap stocks, but this advantage disappears at 100 bps when turnover and microstructure noise dominate. At 40 days, pure beta works for a different reason: slower macro repricing overtakes the firm-specific pure-alpha channel. The strongest conservative row is pure beta with GPT-4o mini sentiment, a Student-t target, a 40-day holding period, and risk parity allocation, reaching Sharpe 2.33 at 100 bps. The results suggest that stock-selection regime and allocator choice matter at least as much as the sentiment model, and that separating firm-specific and macro-exposure triggers is more informative than requiring both to fire simultaneously.
Read the full abstract ▾
All authors Alireza Kargarzadeh, Nariman Khaledian, Navid Parvini, Arman Khaledian
Computation and Language
Structural Silence: When AI Infrastructure Fails Speakers of Underrepresented Languages
Avijit Roy and Proma Roy
Artificial intelligence tools for education and
language support are increasingly framed as scalable responses to access gaps in under-resourced communities. Yet the infrastructure underlying these tools, including training corpora, tokenization schemes, evaluation
benchmarks, and deployment architectures, can systematically disadvantage speakers of underrepresented
languages before a model is trained. This paper examines these structural barriers through Bengali, one of the world's most widely spoken
languages, focusing on AI-assisted education in low-connectivity environments. We identify four interlocking failures: a severe web presence gap, with Bengali accounting for less than 0.5% of global web content despite representing nearly 4% of the global population; a 67:1 training-
token deficit between English and Bengali in major
multilingual corpora; a tokenization penalty associated with Bengali's alphasyllabary script that compounds the data deficit through higher
token fertility; and connectivity exclusion, with individual internet penetration at 36.5% in rural areas compared with 71.4% in urban areas. These failures reflect longstanding resource-allocation decisions, institutional priorities, and design defaults that did not center underrepresented
languages in mainstream AI development. We argue that
dataset scarcity should be understood as a structural barrier rather than an isolated technical limitation, and that offline-first design should be treated as an equity-oriented infrastructure strategy. We conclude with directions for linguistics and AI research aimed at reducing these structural inequalities.
Read the full abstract ▾
Computation and Language
A Cascaded Unsupervised-Supervised NLP Pipeline for Detecting Accusatory Language in Public Procurement
Bryan Torres, Daniel Riofrío, José Vega-Sánchez et al. (+4 more)
Public procurement involves the allocation of substantial financial resources; therefore, continuous oversight through audits, controls, and monitoring mechanisms is essential. However, stakeholder comments and publicly available government data are often underutilized, despite their potential to reveal procedural irregularities. To address this gap, this paper analyzes metadata from Ecuador's Sistema Oficial de Contratación Pública (SOCE, Official Public Procurement System), with particular emphasis on participant comments generated during the pre-contractual phase. We propose a hybrid modeling framework that integrates unsupervised clustering and supervised classification within a natural
language processing (NLP) pipeline to uncover latent patterns and detect potentially irregular procurement processes. Semantic
embeddings are generated using Word2Vec, LLaMA, and RoBERTa, followed by Gaussian Mixture Models (GMMs) for unsupervised clustering. A supervised classification stage is then applied to identify accusatory or whistleblowing-style comments. Experimental results show that the combination of domain-trained Word2Vec
embeddings, GMM-based clustering, and a Random Forest classifier achieves high precision and recall, even under severe class imbalance. These findings demonstrate that lightweight, domain-adapted NLP architectures can effectively support risk identification and enhance transparency in public procurement systems without requiring large-scale computational infrastructure.
Read the full abstract ▾
All authors Bryan Torres, Daniel Riofrío, José Vega-Sánchez, Nathaly Orozco, Carla Parra, Karen Rosero, Felipe Grijalva
Computation and Language
One Frozen Simulator Is Not Enough: Simulator Collapse in Multi-Agent RL
Simon Yu, Nicholas Tomlin, Marwa Abdulhai et al. (+7 more)
Multi-
agent reinforcement learning for human-AI interaction typically relies on a single large
language model to simulate user behavior. We show that this approach systematically fails to generalize, and trace the failure to simulator collapse: because the simulator LLM is mode-collapsed, an LLM policy trained against it overfits to narrow strategies that exploit the simulator's dominant mode, and such a policy transfers poorly to unseen simulators and real users. We formalize this collapse theoretically and propose two complementary solutions, one at inference time and one at training time. The inference-time solution, Verbalized Sampling, broadens the simulator's behavior by sampling from a verbalized response distribution, reducing mode collapse. The training-time solution, Co-Training, jointly optimizes the policy against a population of trainable simulators, preventing it from overfitting to any single simulator's mode. We validate both solutions on three multi-turn
benchmarks: Persuasion for Good, $\tau^2$-bench, and CooperBench. Verbalized Sampling improves held-out success by up to 9% over single-simulator RL, and Co-Training pushes gains further to 14%; the human study shows similar gain on real users. Both solutions preserve the policy diversity that collapses under single-simulator RL. To support further work in this direction, we release SCOPE, an open-source framework for Population Co-Training multi-
agent RL. More broadly, our results suggest that the diversity of the training environment, not only the policy, is critical to the generalization of multi-turn RL to real-world deployment.
Read the full abstract ▾
All authors Simon Yu, Nicholas Tomlin, Marwa Abdulhai, Ximing Lu, Derek Chong, Abe Hou, Dilara Soylu, Sergey Levine, Christopher D. Manning, Weiyan Shi
Computation and Language · Cross-listed from cs.CR
VICBench: A Multi-Language Benchmark for Code Vulnerability Detection
Jin Lu, Xuening Han, Yang Zhong et al. (+4 more)
Evaluating security vulnerability detection tools requires
benchmark datasets with vulnerability-inducing commits (VICs) - the commits that first introduce vulnerabilities into codebases. VICs are essential for determining the full range of vulnerable software versions. Existing vulnerability
datasets suffer from limited programming
language coverage, restricted patch complexity, and narrow project scope. Through our dual annotation by human experts and an agentic workflow, we create a
benchmark - VICBench - of 100 verified VICs for 100 CVEs across 88 projects in Python, Java, and C++, covering 48 CWE types. VICBench features complex real-world vulnerability fixes averaging 38.6 lines and corresponding VICs of 252.5 lines - significantly larger than prior work. Our evaluation shows that state-of-the-art
algorithms V-SZZ and LLM4SZZ achieve only 33.3%-40.1% F1, confirming that using existing approaches still entails significant manual effort. VICBench enables robust evaluation of vulnerability detection approaches.
Read the full abstract ▾
All authors Jin Lu, Xuening Han, Yang Zhong, Lin Tan, Kevin Luo, Andrew Gacek, Neha Rungta
Computation and Language
Information Abundance Paradox: Long-Context Training Undermines Parametric Knowledge
Arda Uzunoglu, Benjamin Van Durme and Daniel Khashabi
Large
language models are increasingly trained and deployed with long contexts that span documents, code repositories, and interaction histories. This scaling reflects the implicit assumption that training on longer contexts will only help the model by exposing it to richer evidence. We challenge this view by studying how the context window shapes a model's mode of learning, shifting it between parametric internalization and contextualization. We propose the Information Abundance Paradox, which hypothesizes that abundant relevant information in the training context can reduce the incentive to encode that information parametrically, thereby increasing reliance on context. In pretraining with long documents, increasing the context window improves
language modeling, natural
language understanding, and closed-book MCQA only up to an intermediate optimum, after which performance consistently declines. In supervised fine-tuning, more task-relevant train-time context improves performance with supporting context, but reduces robustness when context is absent or misleading at test time. Our analysis suggests that this behavior arises when longer context provides a lower complexity solution. Mechanistically, training with informative context shifts gradient pressure from feed-forward networks, often linked to parametric knowledge, toward
attention modules, and causal interventions show that this shift increases reliance on context during inference. Overall, these findings support the Information Abundance Paradox and suggest that scaling toward near-infinite context is not simply a matter of supplying more data, even when high-quality long-context data is abundant.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.AI
Who Thinks Best Depends on How Long You Let Them: Budget-Dependent Rankings in LLM Evaluation
Rodrigo Guedes de Souza and Alison R. Panisson
Standard evaluation of large
language models assumes stable model rankings across inference conditions. We challenge this assumption by varying the
token generation budget, i.e., the maximum
tokens a model may produce, across seven levels (64–4,096), evaluating four models on three
reasoning benchmarks (56,476 inferences). We report four findings: (i) 3–19% of items exhibit non-monotone behavior (accuracy decreasing with more budget), even after controlling for truncation, and this phenomenon is model-specific (cross-model overlap: 6–14%). (ii) Model rankings reverse across budgets on all
benchmarks ($p {<} 0.01$, McNemar). (iii) Oracle analysis reveals model complementarity up to $+27.8$pp, most pronounced at constrained budgets. (iv) A budget-aware router captures 14.1% of the oracle gap cross-domain; budget features help within-domain ($+1.6$ to $+5.7$pp) but are domain-specific and hurt transfer ($-1.2$pp). These results argue for budget-conditioned evaluation protocols.
Read the full abstract ▾
Computation and Language
Massive Activations in Hybrid Linear Attention Large Language Models: Pre-Attention Spikes and Inter-Spike Plateaus
Zunhai Su, Bohan Sun, Xialie Zhuang et al. (+9 more)
We present the first systematic study of massive activations (MAs) in layer-interleaved Hybrid linear
attention large
language models (HLA LLMs), examining their architectural organization, training-time emergence, underlying mechanisms, and functional significance. Across five linear
attention architectures, six hybridization configurations, and five input domains, we identify two architecture-aligned morphologies: pre-
attention spikes (PAS) immediately before full
attention and inter-spike plateaus (ISP) persisting through intervening linear
attention layers. Denser full
attention increasingly connects PAS through ISP, approaching the persistent MAs of conventional
Transformers. This organization also recurs across 12 public checkpoints spanning 1.2B-397B parameters, covering linear
attention and state-space hybrids. Controlled pretraining of Gated DeltaNet (GDN) hybrids up to 1.3B reveals early emergence and consolidation of both morphologies, alongside asymmetric gating effects. Specifically, full
attention output gates strongly attenuate MA magnitudes without eliminating their organization, whereas removing GDN output gates yields modest amplification. Mechanistically, we develop a shared systematic-outlier account: PAS follows a localized write-sink-cancel process, while ISP is consistent with delayed cancellation. Functionally, our interventions show that deleting only the four largest-magnitude PAS coordinates at each full
attention input reduces mean downstream accuracy by 21.9%-63.6% relative to normal inference. Moreover, reference-conditioned spike-to-plateau connection consistently improves mean real-world
retrieval accuracy, yielding relative gains of 1.1%-12.6% without retraining. Our code is available at https://github.com/StartLuxLabs/Massive-Activations-HLA.
Read the full abstract ▾
All authors Zunhai Su, Bohan Sun, Xialie Zhuang, Shuibai Zhang, He Xiao, Hui Shen, Jing Xiong, Hengyuan Zhang, Zhongzhu Zhou, Tiantian Zhang, Ngai Wong, Chuan-Wei Kuo
Computation and Language
A corpus-specific clinical RAG system matches or outperforms newer frontier LLMs on HealthBench
Praveen Reddy, Charuta Mandke, Suvrankar Datta et al. (+4 more)
General-purpose large
language models (LLMs) have recently been reported to match or exceed specialized clinical AI tools on medical
benchmarks, but such comparisons draw on a narrow set of systems and on
benchmarks developed largely in high-income settings. We evaluate VITA, a
retrieval-augmented generation (RAG) system purpose-built for contextual knowledge
retrieval in India and other low- and middle-income (LMIC) settings. VITA retrieves from a curated corpus of disease-specific guidelines, India-specific antimicrobial resistance data, national formulary constraints, and resource-limited care protocols; its architecture and corpus are proprietary, but the
benchmark, the physician-written rubrics, and our full response and scoring outputs are public for independent verification. On 4,023 English-
language HealthBench questions (80.5% of the
benchmark), scored with a GPT-4.1 judge, VITA ranked first with 51.9% of possible rubric points, ahead of GPT-5.4 (46.1%), o4-mini (44.3%), Gemini 3.1 Pro (42.6%), and Claude Sonnet 4.6 (37.3%), and scored highest on 45.4% of questions. To test robustness to newer models and judge lineage, a 500-question subset was re-run against current-generation models (GPT-5.5, Claude Opus 4.8, Gemini 3.5 Pro, Grok 4.3) and graded by a neutral open-weight judge (DeepSeek-V4-Pro) sharing no lineage with any system tested. Here the gap narrowed to parity: VITA and GPT-5.5 were statistically indistinguishable on mean per-question score, while VITA led on points-weighted score and won the most questions. VITA's advantages in accuracy and completeness persisted under the neutral judge; its communication scores were lower. These results indicate that a purpose-built clinical RAG system remains competitive with frontier LLMs on an open
benchmark, consistent with corpus specificity as a design variable that improves grounding at some cost to communication polish.
Read the full abstract ▾
All authors Praveen Reddy, Charuta Mandke, Suvrankar Datta, Sarah Khan, Siddharth Reddy Anthireddy, Shitij Arora, Vishal Singh
Computation and Language
SAG: SQL-Retrieval Augmented Generation with Query-Time Dynamic Hyperedges
Yuchao Wu, Junqin Li, XingCheng Liang et al. (+4 more)
While
retrieval-augmented generation (RAG) has proven effective at giving LLMs access to external knowledge, mainstream dense-
retrieval implementations remain inherently limited in handling structured constraints and multi-hop
reasoning. Graph-based methods address this by constructing knowledge graphs offline, but they often fragment semantics, incur high maintenance, and complicate incremental updates. We propose SAG (SQL-
Retrieval Augmented Generation), a structured
retrieval architecture that organizes documents into an event-entity index without building a global knowledge graph. SAG represents each chunk as a semantically complete event paired with its entities, forming a latent hyperedge that preserves n-ary relations without decomposing them into triples. At query time, SAG treats shared entities as join keys to connect related chunks. This dynamically yields a query-scoped neighborhood of events, and yet every piece of evidence remains the original chunk throughout. Experiments on HotpotQA, 2WikiMultiHopQA, and MuSiQue show that SAG achieves the best
retrieval and end-to-end QA performance on every
benchmark, with gains that widen as
reasoning-chain complexity increases. On MuSiQue, where multi-hop evidence chaining is most demanding, SAG reaches 80.36% Recall@5, outperforming the strongest baseline by 11.52 points. This work paves the way for knowledge infrastructure that enables LLM
agents to retrieve and reason over continually growing organizational knowledge.
Read the full abstract ▾
All authors Yuchao Wu, Junqin Li, XingCheng Liang, Yongjie Chen, Yinghao Liang, Linyuan Mo, Guanxian Li
Computation and Language · Cross-listed from cs.GT
Do LLMs Take Care of Their Own? Similarity Signals Can Induce Cooperation
Akash Kundu, Emanuel Tewolde, Ratip Emin Berker et al. (+2 more)
As LLM-based
agents with user-instructed goals are becoming widely deployed, they increasingly encounter each other in strategic interactions, and face challenges of finding mutually beneficial outcomes. Prior literature has argued that cooperation problems such as the Prisoner's Dilemma are resolvable in settings where
agents know they follow very similar decision making patterns, as for example in monocultural AI ecosystems. Following that line of work, this paper introduces the first framework for evaluating LLM decision making when
agents are provided with graded similarity signals. Among our findings, we establish that different LLM models vary drastically in how they navigate similarity signals, with some modern models showing consistent behavior across cooperation problems, payoff structures, and
prompt framing. Perhaps surprisingly, our experiments also show that the
dataset based on which the similarity signal is computed has small to no impact on induced cooperation, and that LLM models systematically self-identify as highly similar when asked to evaluate another model's chain-of-thought
reasoning by themselves. Finally, we develop an LLM-behavioral-game-theoretic model that captures some of their
reasoning rationale, and show that it can support cooperative outcomes in
equilibrium under sufficiently high similarity scores.
Read the full abstract ▾
All authors Akash Kundu, Emanuel Tewolde, Ratip Emin Berker, Samuel F. Brown, Vincent Conitzer
Computation and Language
QV-PIC: Query-Aware Visual Position-Independent Caching for Efficient RAG Serving
Yilin Liu, Rui Meng, Wangze Ni et al. (+5 more)
Retrieval-Augmented Generation (RAG) repeatedly prefills identical text chunks across queries, incurring redundant computations. Position-Independent Caching (PIC) mitigates it by reusing precomputed Key-Value (KV) across positions, but its efficiency is constrained by the large volume of text
tokens. Rendering text chunks as images can compress the text into fewer visual
tokens, but the rendered-image PIC suffers more severe quality degradation than the text PIC. This representation-specific gap primarily arises from contextual mismatches across independently compiled caches and the loss of fine-grained textual evidence during visual compression. Existing PIC repair methods mainly address the former through selective recomputation, but they incur online computation and cannot recover lost textual details. We propose QV-PIC, a query-aware dual-resolution PIC reuse framework guided by model-native templates. Offline, QV-PIC compiles visual caches under the model's native chat-template prefix, improving PIC quality without online recomputation. Online, it preserves global context with low resolution and restores fine-grained textual evidence within a high-resolution budget by cumulative query relevance scores, retaining the efficiency benefit of visual compression. Across six tasks, QV-PIC improves average F1 by 21.6 points over vanilla rendered-image PIC, closes the gap to vanilla text PIC, and surpasses optimized text PIC by 2.58 F1 while reducing TTFT by 17.2%. Relative to full prefill, it cuts TTFT by 83.8%.
Read the full abstract ▾
All authors Yilin Liu, Rui Meng, Wangze Ni, Jianxin Yan, Heng Cao, Libin Zheng, Peng Cheng, Jinfei Liu
Computation and Language
Structuring the Space of Perspectives
Agnese Daffara, Sebastian Padó and Tanise Ceron
The same event can be reported from different perspectives depending on the experiences, background, and beliefs of the writer or speaker. A variety of NLP areas engage with perspectives, spanning from text analysis to
algorithm optimization. A wide range of operative concepts (such as stances, sentiment, frames, and arguments) has been used to capture perspectives in texts, however the precise relationships among those concepts remain unclear. Arguably, a deeper theoretical understanding of these concepts would empower more effective research on perspectives. In this paper, we address this gap by reviewing the space of perspectives in NLP and defining a set of properties that help distinguishing perspective-related concepts. Our analysis leads us to posit a hierarchy which organizes these concepts linearly along a single axis. Finally, we show how this principled conceptual hierarchy can help researchers navigate the field and select operationalizations of perspective that align with their specific research objectives.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.SD
RT-SEMamba: Real-Time Speech Enhancement Mamba via Progressive Knowledge Distillation
Rong Chao, Sung-Feng Huang, Moreno La Quatra et al. (+4 more)
We present RT-SEMamba, a fully causal
speech enhancement (SE) model built upon causal time-frequency Mamba blocks. Unlike
Transformer-based architectures that rely on a growing key-value cache, Mamba propagates a fixed-size recurrent state per layer, enabling
memory- and bandwidth-efficient long-form inference. We further introduce a progressive knowledge
distillation (KD) strategy that compresses an 8-layer teacher into a shallow 1-layer student by jointly distilling complex spectral outputs and intermediate representations. On Voicebank-DEMAND, the 8-layer RT-SEMamba achieves 3.32 PESQ with a 25 ms algorithmic latency constraint, and the distilled 1-layer student improves over a naive 1-layer baseline from 3.06 to 3.18 PESQ while preserving the same steady-state RTF, delivering a 2.64x speedup over the teacher. These results demonstrate that state-space models with progressive KD provide a competitive quality-latency trade-off for real-time SE.
Read the full abstract ▾
All authors Rong Chao, Sung-Feng Huang, Moreno La Quatra, Sabato Marco Siniscalchi, Wen-Huang Cheng, Szu-Wei Fu, Yu Tsao
Computation and Language
Preference Tree Optimization: Enhancing Goal-Oriented Dialogue with Look-Ahead Simulations
Lior Baruch, Moshe Butman, Kfir Bar et al. (+1 more)
Developing dialogue systems capable of engaging in multi-turn, goal-oriented conversations remains a significant challenge, especially in specialized domains with limited data. This research proposes a novel framework called Preference Tree Optimization (PTO), designed to iteratively improve
agent models in such dialogue systems, by generating preference data using a method called Preference Tree with Look-Ahead. Focusing on Motivational Interviewing (MI) – a counseling technique aimed at facilitating behavioral change – we leverage virtual patients and an oracle evaluator to simulate conversations and generate rich preference
datasets. By combining this method with Direct Preference Optimization (DPO), we aim to enhance the
agent's decision-making capabilities over iterative training cycles. The proposed framework addresses data scarcity and advances the development of more nuanced and effective dialogue systems in goal-oriented domains. Experimental evaluations demonstrate that the PTO framework enhances dialogue
agents' performance in goal-oriented conversations within the domain of Motivational Interviewing (MI). Models trained with PTO consistently outperformed the baseline in key metrics such as session satisfaction and working alliance. Additionally, incorporating look-ahead simulations led to improved long-term planning and more effective conversational strategies, with deeper look-ahead configurations yielding the most stable and high-scoring results.
Read the full abstract ▾
All authors Lior Baruch, Moshe Butman, Kfir Bar, Doron Friedman
Computation and Language · Cross-listed from cs.AI
Mechanist: AI as a Scientific Instrument for Discovering the Mechanisms of Intelligence
Mengru Wang, Junfeng Fang, Shuofei Qiao et al. (+17 more)
AI models are increasingly used in scientific discovery and human decision-making. Yet how AI models work and what risks they pose remain poorly understood. As AI development becomes faster and more automated, research on the mechanisms underlying AI remains largely manual. To bridge this gap, we introduce Mechanist, an agentic system that uses AI as a scientific instrument for the autonomous discovery of mechanisms underlying AI. To ground novel mechanism hypotheses, we construct a scientific knowledge graph of 13,000 studies on AI mechanisms, alongside a multidisciplinary database of 43 million papers spanning 26 fields. For reliable experiment execution, we curate a library of 32 foundational methods for mechanism analysis. Compared with Claude Code and existing AI-scientist systems, Mechanist generates higher-quality mechanism hypotheses and executes experiments more reliably. Across four case studies, Mechanist autonomously discovers new model behaviors and their underlying mechanisms, and translates these discoveries into mechanism-guided interventions and interdisciplinary design. Specifically, Mechanist first uncovers a counterintuitive safety risk in scientific laboratories, showing that unsafe traits can transfer to fine-tuned student models through apparently safe training data and emerge across modalities. Mechanist then develops a mechanism theory of belief, revealing how models represent world knowledge, form beliefs, infer the beliefs of others, and how these mechanisms emerge during pretraining. Building on this theory, Mechanist develops targeted interventions that improve model performance across diverse scenarios. Finally, Mechanist can also advance interdisciplinary discovery through mechanistic design, providing an alternative to the computationally intensive generate-and-rerank paradigm.
Read the full abstract ▾
All authors Mengru Wang, Junfeng Fang, Shuofei Qiao, Zhenqian Xu, Haoming Xu, Haoxiong Wang, Shumin Deng, Linyi Yang, Xin Xu, Yunzhi Yao, Dan Zhang, Fei Shen, Zhixiang Cui, Buqiang Xu, Haozhe Luo, Yunxiang Wei, Ningyu Zhang, Julian McAuley, Tat Seng Chua, Huajun Chen
Computation and Language
Unified Multi-Dialectal Neural Machine Translation for Bangla Using the Dwadash Benchmark Corpus
Rakib Ullah, Md. Ruhul Islam, Tanbir Ahmed et al. (+1 more)
Neural Machine Translation (NMT) and Large
Language Models (LLMs) excel at cross-lingual tasks but often fail to capture intra-lingual morphological variation, marginalizing dialectal speakers. In Bangla, existing translation frameworks commonly rely on Standard Colloquial Bangla (SCB) as an intermediate pivot, which can compound errors and reduce cross-dialectal nuance. To address this gap, we introduce a unified, multi-directional NMT system for direct translation between SCB and eleven regional variants. We first review prior dialectal NLP resources to identify existing technological gaps. As a foundational contribution, we construct and release a large multi-dialect parallel corpus for Bangla, comprising 14,562 aligned rows and 51,541 non-null sentence pairs through the integration of seven prior
datasets and native-speaker-verified manual augmentation. Using this corpus, we
benchmark state-of-the-art sequence-to-sequence architectures with parameter-efficient Weight-Decomposed Low-Rank Adaptation (DoRA). Results show that deep monolingual pre-training is more effective than large
multilingual capacity for this task. The compact BanglaT5 model outperforms NLLB-200 and mBART-50 by up to 13.96 BLEU, achieving 29.26 BLEU, 57.26 chrF++, and 49.68 METEOR. A
dataset scaling study shows diminishing returns beyond 3,000 parallel pairs and indicates that linguistic proximity to Standard Bangla is more important than raw data volume for translation quality. Finally, we deploy the optimized model as an INT8-quantized web application, providing a scalable, open-source framework for inclusive
language technology and equitable digital access.
Read the full abstract ▾
All authors Rakib Ullah, Md. Ruhul Islam, Tanbir Ahmed, Nayan Kumar Nath
Computation and Language
Asymptotic Risk Calibration for Selective Question Answering
Shufan Lin and Sijin Dong
Large
language models (LLMs) may generate fluent but incorrect answers, making uncertainty quantification important for reliable question answering. However, heuristic uncertainty scores cannot perfectly distinguish correct predictions from incorrect ones, and directly applying a fixed uncertainty threshold provides no statistical control over the error rate among accepted answers. To address this limitation, we propose A-CRC-QA, a post-hoc calibration framework for uncertainty-aware selective question answering. The proposed method reformulates selection-conditioned error control as a linear expectation constraint and applies a monotonized empirical-risk calibration procedure inspired by conformal risk control. Since the resulting instance-wise loss is generally non-monotone with respect to the acceptance threshold, our framework targets asymptotic rather than finite-sample risk control. A-CRC-QA is model-agnostic, requires no additional training, and can be combined with different uncertainty
estimators. Experiments on CoQA and MedMCQA demonstrate its applicability to both open-ended and closed-ended question answering, achieving a favorable trade-off between accepted-answer reliability and answer retention compared with uncalibrated and confidence-bound-based baselines.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.AI
Claim-Level Reliability Assessment for Efficient Test-Time Reasoning
Sen Xu, Wei Wang, Shixi Liu et al. (+5 more)
We propose claim-level falsification as a principle for test-time scaling and instantiate it through Claim-Level Reliability Assessment (CLR), a training-free framework that reallocates test-time compute from additional solution sampling to targeted verification. Since whole-trace evaluation often obscures decisive errors due to signal dilution from routine
tokens, CLR condenses each
reasoning trace into a compact set of decision-critical claims, thereby isolating its logical anchors. Furthermore, recognizing the inherent difficulty of generating entirely correct solutions under fixed model capabilities, CLR shifts the focus to semantic falsification. This approach exploits a fundamental asymmetry between solution construction and claim refutation. Constructing a valid solution requires a flawless
reasoning path, whereas refuting an incorrect claim requires identifying only a single decisive flaw. This targeted search for negative evidence systematically compresses the survival space of high-confidence incorrect traces, effectively suppressing erroneous consensus via nonlinear reliability scoring. Across four LLMs and four
reasoning benchmarks under matched budgets, CLR generally improves upon pass@1 and self-consistency. On GPT-OSS-20B/CMIMC25, for instance, CLR exceeds pass@1 by 27.15 percentage-points and raises self-consistency accuracy from 77.50% to 82.19% with 37.0% fewer
tokens.
Read the full abstract ▾
All authors Sen Xu, Wei Wang, Shixi Liu, Jixin Min, Yingwei Dai, Zhibin Yin, Yirong Chen, Junlin Zhang
Computation and Language
Benchmarking Trustworthiness of SLMs: Pre-trained vs. Compressed
Haokun Lin, Kaijie Zhu, Haobo Xu et al. (+4 more)
Small
Language Models (SLMs) have emerged as a more efficient alternative to traditional Large
Language Models (LLMs), offering promising potential in resource-constrained scenarios. Existing approaches to building SLMs typically follow two paths: training compact models from scratch, or compressing larger pre-trained models using methods such as
pruning, quantization, or
distillation. As
language models become increasingly integrated into real-world applications, ensuring their trustworthiness has become a critical concern. However, how to build trustworthy SLMs remains an underexplored question. In this work, we present a comprehensive evaluation of SLM trustworthiness across multiple dimensions, including
fairness, robustness,
privacy, and ethics. We first examine the effects of
pruning and quantization, and find that quantization is significantly more effective in preserving trustworthiness compared to
pruning. More importantly, we demonstrate that compressing a reliable large model via quantization can produce SLMs with superior trustworthiness and adaptability compared to using small models trained from scratch. Furthermore, knowledge
distillation from trustworthy teacher models can further enhance the reliability of SLMs. We hope our findings provide practical guidance and a foundation for future research into the development and deployment of trustworthy small
language models.
Read the full abstract ▾
All authors Haokun Lin, Kaijie Zhu, Haobo Xu, Yichen Wu, Zhichao Lu, Qingfu Zhang, Zhenan Sun
Computation and Language
A, B, C, or Stable? Accuracy and Order Sensitivity Diverge in LLM Multiple-Choice Evaluation
Karl Hanna and Chen Feng
Multiple-choice question (MCQ)
benchmarks are widely used to evaluate large
language models, yet a single reported accuracy number is not enough to distinguish a model's latent ability from order sensitivity. Debiasing methods tend to be judged on one axis, which implicitly assumes that fixing order sensitivity and improving accuracy go together. We test this assumption with two proprietary models and two open-weight models on MMLU and ARC-Challenge, across four families of strategies. We measure order sensitivity with flip rate, capturing per-question instability, and recall standard deviation (RStd), capturing aggregate positional imbalance. We find accuracy and order sensitivity do not move together consistently: two-stage prompting reduces both flip rate and RStd for GPT on MMLU without a significant change in accuracy. The two sensitivity metrics also diverge from each other. Adding
reasoning to direct answering leaves Llama's MMLU flip rate statistically indistinguishable while RStd falls sharply. A $2\times2$ ablation across generation-then-matching, crossing option visibility with matcher type, shows that withholding options mainly harms semantic matching, which achieves zero flip rate by construction, yet loses significantly in accuracy; the effect is smaller when using an LLM matcher. We recommend reporting flip rate and RStd along with accuracy when evaluating MCQ strategies.
Read the full abstract ▾
Computation and Language
Spark-to-Paper: End-to-End Research Paper Generation as a Composable Skill
Zhuoyang Qian, Biao Wu, Yiran Wang et al. (+6 more)
Turning a research idea into a complete paper requires more than text generation: the system must retrieve literature, design and execute experiments, revise claims according to evidence, produce publication-ready figures, and maintain consistency across a long generation process. We present Spark-to-Paper, an end-to-end research paper generation system implemented as thirteen composable skills inside an existing coding assistant, without requiring a separate
agent platform or orchestration service. Spark-to-Paper separates model-based judgment from deterministic operations that can be directly executed and checked. It further separates experiment planning from reporting, so that required evidence is specified before results are observed and manuscript claims are revised according to measured outcomes. To improve reliability over long research trajectories, the system combines deterministic integrity checks with self-critique and bounds a failure mode we call the Self-Refutation Loop, in which repeated experiments continue to reject the original research objective. Spark-to-Paper also produces editable vector figures through programmatic plotting for experimental results and code-based reconstruction for generated method diagrams. Across eight controlled research topics, Spark-to-Paper achieves 99.5% citation validity and 96.4% figure editability. A controlled ablation increases fabrication detection from 14% for a single-pass draft to 92% with the full integrity and review stack, while
adversarial review achieves 74% precision. The full system uses 11.9M
tokens, costs $8.1 per manuscript, and requires 3.2 hours on average. These results show that end-to-end research paper generation can be implemented as a lightweight, composable workflow inside existing coding assistants while keeping experimental evidence central to how claims are accepted, revised, or abandoned.
Read the full abstract ▾
All authors Zhuoyang Qian, Biao Wu, Yiran Wang, Chris D Yan, Desan Dai, Liangwei Zheng, Jin Jiang, Junsheng Zhang, Wenhao Wang
Computation and Language
LODESTAR: Robust Entropy-Based Answer Selection in Retrieval-Augmented Generation for Question Answering – Directing Frozen-LLM Entropy with a Reinforcement-Learned Prompt Polarizer under Misleading Passages
Hung-Chun Hsu, Po-Jen Ko, Che-Cheng Wu et al. (+2 more)
Predictive-distribution
entropy is a strong answer-selection rule in
retrieval-augmented generation (RAG) for question answering: across five QA
benchmarks, selecting the answer a frozen respondent LLM produces with the lowest answer-
token entropy lifts mean $F_1$ from 0.4769 to 0.5148 over the retriever's top-ranked passage, without gold answers. Yet this rule, which prior
entropy-based selectors adopt, fails: a misleading passage makes the respondent confidently wrong, driving
entropy down where the uncertainty signal looks most trustworthy. The failure comes from the passage the respondent reads, and the context it is read in is an input we can intervene on. We introduce LODESTAR: to our knowledge the first method to score a text intervention by the uncertainty it induces in a third-party frozen respondent, compared within one question. LODESTAR uses reinforcement learning (GRPO) to train, once and offline, a polarizer – a short fixed natural-
language string inserted into the respondent's
prompt and never into its weights, directing
entropy so that
entropy-based answer selection stays robust to misleading passages; training labels are built from gold answers and two LLM judges, and inference reads neither. With every competing selector under the same frozen respondent and candidate pools on 5,008 questions, LODESTAR attains the highest mean $F_1$ of any inference-ready selector (0.5339), the highest macro exact match (0.4136), and the highest GPT-4o judge score of the frozen-respondent configurations judged (0.6435); its three-seed mean wins all 70 $F_1$
cells against fourteen published configurations and is paired-significant on $F_1$ against every one. The gain holds in-domain on NQ-Open and out-of-domain over SQuAD, TriviaQA, EntityQuestions and WebQuestions. Ablating the polarizer shows it is what makes the respondent read a misleading passage less often (26.0% vs 30.3%).
Read the full abstract ▾
All authors Hung-Chun Hsu, Po-Jen Ko, Che-Cheng Wu, Li-Yang Chang, Chuan-Ju Wang
Computation and Language
LazyTrain: Limited-resource Allocation toward Zero-waste Yield Optimization in Large Language Model Training
Xiaojun Wu, Cehao Yang, Honghao Liu et al. (+5 more)
Training large
language models on limited hardware is increasingly a scheduling problem across GPU compute, host
memory, PCIe transfer, and storage bandwidth. Existing offloading systems reduce GPU residency, and MegaTrain shows that a CPU-master layer-streaming executor can train large models on a single GPU, but fixed checkpointing and placement heuristics still leave communication exposed on the critical path. We propose LazyTrain, an optimization layer over a layer-streaming executor. LazyTrain formulates checkpoint selection, activation placement, recomputation, and CPU-GPU-NVMe communication overlap as a mixed-integer scheduling problem, then executes the solved policy during training. It further couples 8-bit optimizer states with fast gradient clipping as a single Hybrid 8-bit operator: state compression reduces optimizer-state
memory, while fast clipping counteracts the additional CPU-side update overhead. Across H800 experiments from Qwen2.5-3B to Qwen3.6-27B, LazyTrain improves sustained TFLOPS over matched baselines runs by approximately 1.24$\times$; RTX 3090 experiments likewise increase the maximum feasible batch size by one at each model scale. In the primary Qwen3.6-27B H800 MetaMathQA run, LazyTrain reaches 219.95 TFLOPS and 1361
tokens/s at batch size 72, peaks at 68.84\,GB of GPU
memory, and obtains 95.42% exact-match accuracy on the full evaluation split. The source code is available at https://github.com/DataArcTech/LazyTrain.
Read the full abstract ▾
All authors Xiaojun Wu, Cehao Yang, Honghao Liu, Xueyuan Lin, Xuhui Jiang, Chengjin Xu, Jia Li, Jian Guo
Computation and Language · Cross-listed from cs.DB
DexterSQL: Deep Schema Exploration and Rule-based Correction for Text-to-SQL Generation
Anik Pramanik, Murat Kantarcioglu, Vincent Oria et al. (+1 more)
Prompting-based (i.e., non-fine-tuning) Text-to-SQL methods, where underlying large
language model parameters are not changed for the task, face three problems: (i) relying on coarse-grained schema information that may not reveal the fine-grained relationships needed to distinguish ambiguous columns, (ii) failing to capture recurring SQL-generation failures, and (iii) suffering from omission or
hallucination of components in complex questions. This paper develops DexterSQL, a prompting/non-fine-tuning-based Text-to-SQL system that improves SQL generation with three novel components: (i) deep schema explorator that identifies ambiguous columns, analyzes their individual and joint data distributions to uncover their relationships and the distinct role of each, (ii) database-agnostic rule creator that mines mismatches between generated and gold SQL only on the training database and converts them into database-agnostic corrective rules that capture recurring LLM failure patterns; and (iii) multi-path SQL generation that introduces a dependency-tree-based intermediate representation that uses the question's sentence structure to guide its decomposition into an SQL skeleton for final SQL generation. DexterSQL achieves a higher accuracy compared to the state-of-the-art using both open-source/weight and closed-source/weight models. Particularly, DexterSQL shows a high improvement of at least 5.5% using an open-weight model (GPT-OSS-120B) on BIRDDev, with total accuracy 70.4%. DexterSQL also shows better improvement of at least 1.4% using closed-weight models, with total accuracy 72.1% and 72.9% on BIRD-Dev with GPT-4o and GPT-5.2.
Read the full abstract ▾
All authors Anik Pramanik, Murat Kantarcioglu, Vincent Oria, Shantanu Sharma
Computation and Language
Total Recall at What Cost? Benchmarking the Serving Cost of Agentic Memory Systems
Natchanon Pollertlam and Witchayut Kornsuwannawit
Long-running conversational
agents increasingly rely on a
memory system to avoid resending the whole conversation each turn, yet how much that costs to serve has received little systematic benchmarking. We compare three
memory systems (Mem0, Hindsight, and Mastra Observational
Memory) against two reference strategies – a fixed-size rolling window and resubmitting the full transcript – across two backbones and conversations of up to 400 turns, pairing every cost measurement with answer accuracy on 665 LoCoMo questions. First, a
memory system's serving cost cannot be predicted from conversation length and message size alone: a regression that tracks the two reference strategies closely misses the
memory systems by 18-69%, their cost driven instead by internal
memory behavior. Second, a break-even analysis shows that whether – and when – a
memory system becomes cheaper to serve than the full transcript is highly sensitive to the system and the backbone, from the first tens of turns for the cheapest to never within 400 turns for the most expensive. Third, no system wins on both axes: accuracy spans 21-54%, and the backbone choice drives cost as much as the
memory system does.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.CR
ToolHazard: Scaling Adversarial Environments for Security Evaluation and Alignment of LLM-based Agents
Yutao Mou, Pengfei Yang, Zhe Yin et al. (+6 more)
Large
language model (LLM)
agents integrated with external tools are vulnerable to indirect
prompt injections embedded in environmental states. However, existing studies largely rely on manually implemented or reused environments,
stochastic LLM-based tool simulation, and predefined injection locations, limiting scalable security research across broader domains. To bridge this gap, we propose **ToolHazard**, a scalable
adversarial environment synthesis framework that reduces human engineering and supports expansion with additional seed domains and compute. Through an Environment Simulator, an Attacker
Agent, and a User Simulator, ToolHazard synthesizes executable stateful environments, discovers viable injection points and generates environment-specific payloads, and constructs state-grounded long-horizon tasks. Based on ToolHazard, we build **ToolHazard-Bench** for stress-testing
agents under complex workflows and diverse environmental attacks. Experiments reveal substantial
agent vulnerabilities and show that injection timing and placement affect attack effectiveness. Moreover, ToolHazard-generated
alignment data improves security on both ToolHazard-Bench and AgentDojo while preserving benign task utility.
Read the full abstract ▾
All authors Yutao Mou, Pengfei Yang, Zhe Yin, Zhangchi Xue, Xiaotian Luan, Dingyao Yu, Tong Zhang, Shikun Zhang, Wei Ye
Computation and Language · Cross-listed from cs.CV
LookBack: Where and How to Score LVLM Responses via Visual Reference Usage
Beomsik Cho, Jinhyeong Kim, Dongseok Lee et al. (+1 more)
Large
Vision-
Language Models (LVLMs) integrate visual perception with
language generation, enabling responses that span image understanding and complex
reasoning. However, LVLMs do not just inherit the text-level
hallucinations; they also hallucinate against the image, producing fluent responses ungrounded in what they see. This makes LVLM response scoring inherently harder, and our diagnostics show that existing confidence-based metrics adopted from LLMs are insufficient for LVLMs. Specifically, removing the input image barely changes confidence-based selection, suggesting that output-space confidence primarily captures textual plausibility rather than agreement with the image. To address this gap, we propose LookBack, a training-free LVLM response scoring method that augments
token likelihood with visual lookback score, a lightweight measure of how strongly each response
token refers to image
tokens. Across four
benchmarks and three models, LookBack consistently improves Best-of-$N$ selection over existing baselines with negligible additional overhead.
Read the full abstract ▾
All authors Beomsik Cho, Jinhyeong Kim, Dongseok Lee, Jaehyung Kim