Computation and Language
Do Agents Need to Plan Step-by-Step? Rethinking Planning Horizon in Data-Centric Tool Calling
Naoki Otani, Nikita Bhutani, Hannah Kim et al. (+2 more)
Explicit planning is a critical capability for LLM-based
agents solving complex data-centric tasks, which require precise tool calling over external data sources. Existing strategies fall into two paradigms based on planning horizon: (1) full-horizon (FH), which generates a complete plan before execution, and (2) single-step horizon (SH), which interleaves each action (tool call) with incremental
reasoning and observation. While step-by-step execution is a common default under the assumption that eager execution monitoring is necessary for adaptability, we revisit this assumption for well-defined data-centric tasks. Our controlled empirical study isolates planning horizon as the key architectural feature and systematically analyzes the effects of topological complexity and tool robustness on both paradigms. Our experiments across Knowledge Base Question Answering and Multi-hop QA show that FH planning with lazy replanning achieves accuracy parity with SH across varying depths, breadths, and robustness levels, while using 2-3x fewer
tokens. These findings suggest that for well-defined data-centric tasks, eager step-wise monitoring is often unnecessary, and full-horizon planning with on-demand replanning can offer a more efficient default.
Read the full abstract ▾
All authors Naoki Otani, Nikita Bhutani, Hannah Kim, Dan Zhang, Estevam Hruschka
Computation and Language
A Computational Operationalisation of Competing Maturational Theories of Syntactic Development via Statistical Grammar Induction
Mila Marcheva, Suchir Salhan and Weiwei Sun
This paper is concerned with what intermediate syntactic categories children acquire during first
language development, and in what order. Maturational theories make different predictions. Bottom-up accounts (GROWING) propose that lexical and inflectional structure emerges first, while inward accounts (INWARD) predict early access to discourse-related categories. We computationally operationalise these hypotheses of staged syntactic emergence using statistical grammar induction, asking what each proposed ordering makes learnable when input and learning
algorithm are held constant. Our framework makes category acquisition explicit and allows us to explore how different maturational orderings shape the structure that can be learned under identical conditions. Based on this operationalisation, the GROWING account significantly outperforms the INWARD account across three evaluation metrics.
Read the full abstract ▾
Computation and Language
PYTHALAB-MERA: Validation-Grounded Memory, Retrieval, and Acceptance Control for Frozen-LLM Coding Agents
Mehmet Iscan
Local LLM-based coding
agents increasingly work in settings where correctness is earned through execution feedback, persistent state, and bounded repair, not through a single fluent answer. Static
retrieval, long-context prompting, self-refinement, execution-feedback repair, and reinforcement learning over model weights each address part of this setting, but they do not jointly provide validation-grounded episodic
memory, adaptive
retrieval-action selection, delayed credit assignment, and structural skill reuse around a frozen local model. We introduce PYTHALAB-MERA, a lightweight external controller for local validation-conditioned code generation. The frozen
language model proposes complete source files; the controller decides which
memory records and AST-derived skills should enter the next
prompt, validates each candidate through a fail-fast pipeline, converts validation outcomes into bounded shaped rewards, and propagates delayed credit through TD(lambda)-style eligibility traces. We evaluate the implementation as a local CLI artifact on reinforcement-learning coding tasks with strict validation gates. In the measured hard RL setting with three tasks, three repetitions, and a three-attempt budget, PYTHALAB-MERA passed 8/9 strict validations; the self-refinement baseline and the investigated GRACE extension each passed 0/9. These results support a deliberately bounded claim: in this recorded setting, the external
memory-and-
retrieval controller improved validation success. They do not establish general-purpose code synthesis, state-of-the-art performance, formal program correctness, or formal safety.
Read the full abstract ▾
Computation and Language
Do Benchmarks Underestimate LLM Performance? Evaluating Hallucination Detection With LLM-First Human-Adjudicated Assessment
I. F. Atasoy, B. Mutlu, E. A. Sezer et al. (+1 more)
Hallucination remains a persistent challenge in Large
Language Models (LLMs), particularly in context-grounded settings such as RAG and agentic AI systems. This study focuses on contextual
hallucination detection in summarization tasks. We analyze the QAGS-C and SummEval
datasets by comparing original
benchmark annotations with reason and span-based predictions from Gemini 2.5 Flash and GPT-5 Mini. To address systematic divergences between human labels and LLM judgments, we re-evaluated all conflicted samples through a human adjudication process involving 2 cross-cultural adjudicators. Following this re-evaluation, triple agreement (between human, GPT, and Gemini) increased by 6.38% for QAGS-C and 7.62% for SummEval. Similarly, model accuracy improved, with GPT increasing by 4.25% on QAGS-C and 2.34% on SummEval, while Gemini showed gains of 8.51% and 3.80%, respectively. Notably, adjudicators frequently sided with the models' judgments over original human annotations when LLMs provided explicit
reasoning. Overall human adjudicator agreement ranged between 83% and 87%. These findings suggest that for ambiguity-prone tasks, single-pass annotations may be insufficient, and model-assisted re-evaluation yields more reliable
benchmarks.
Read the full abstract ▾
All authors I. F. Atasoy, B. Mutlu, E. A. Sezer, A. Wahdan
Computation and Language · Cross-listed from cs.AI
LLM-guided Semi-Supervised Approaches for Social Media Crisis Data Classification
Jacob Ativo, Bharaneeshwar Balasubramaniyam, Anh Tran et al. (+4 more)
Semi-supervised learning approaches have been investigated as a means to enhance the analysis of social media data in disaster management contexts. In this work, we present the first empirical evaluation of large
language model (LLM) guided semi-supervised learning for crisis related tweet classification. We compare two recent LLM assisted semi-supervised methods, VerifyMatch and LLM guided Co-Training ( LG-CoTrain), against established semi-supervised baselines. Our results show that LG-CoTrain significantly outperforms classical semi-supervised approaches in low resource settings with 5, 10 and 25 labeled examples per class, achieving the highest averaged Macro F1 across events. VerifyMatch achieves competitive performance while also demonstrating strong calibration properties. As the number of labeled examples increases, the performance gap narrows and Self Training emerges as a strong baseline. We further observe that compact semi-supervised models can, in some cases, outperform very large LLMs operating in zero-shot settings. This finding highlights the potential of transferring knowledge from LLMs into smaller and more deployable models through LLM guided semi-supervised learning, offering a practical pathway for real world disaster response applications. Our project repository on Github is here.
Read the full abstract ▾
All authors Jacob Ativo, Bharaneeshwar Balasubramaniyam, Anh Tran, Khushboo Gupta, Hongmin Li, Doina Caragea, Cornelia Caragea
Computation and Language
Revisiting the syntax of imperatives in Yemeni Arabic: An Agree across phases approach
Mohammed Q. Shormani
This article revisits the syntax of imperatives in Yemeni Arabic proposing an Agree acros phases (AAP) approach. I argue that the AAP approach successfully accounts for both simple and complex imperative constructions, including A'-chain structures, by establishing a close interactions between syntax and discourse. The study demonstrates that this interface is motivated by the interpretive and performative functions associated with imperatives, linking informational structure with propositional structure. It is also proposed that the thematic subject of imperatives is a 2-person pro, whereas any overt pronominal or nominal element occurring preverbally is not a subject, but rather a C-domain element, precisely aboutness topic. These topics serve as the logical subjects of imperatives and enter into a coreferentiality relationship with pro. This relation is analyzed as APP involving Match, yielding both local and non-local A'-chains. For core imperatives, viz., lacking an overt topic, I propose a null topic to (re)merge in Spec,TopP, whose interpretation depends on the discourse.
Read the full abstract ▾
Computation and Language
Can Language Models Identify Side Effects of Breast Cancer Radiation Treatments?
Natalie Seah, Danielle S. Bitterman, Daphna Spiegel et al. (+1 more)
Accurately communicating the side effects of cancer treatments to cancer survivors is critical, particularly in settings such as informed consent, where clinicians must clearly and comprehensively convey potential treatment toxicities. However, this task remains challenging due to clinical knowledge deficits about adverse treatment effects and fragmentation across electronic health record (EHR) systems. Large
language models (LLMs) have the potential to assist in this task, though their reliability in oncology survivorship contexts remains poorly understood. We present a deployment-oriented stress-testing framework for evaluating LLM-generated radiation side effect lists in breast cancer treatment and survivorship care. Using 21 breast cancer patient profiles, we construct paired patient clinical scenarios that differ only in radiotherapy regimens to evaluate seven instruction-tuned LLMs under multiple prompting regimes. We then compare LLM outputs to a clinician-curated reference derived from informed consent documents at two major academic medical centers and developed by a team including more than seven breast radiation oncologists. The reference maps radiation dose-fractionation, fields, and locations to associated toxicities, broken down by frequency and temporal onset. Across models, we reveal sensitivity to minor documentation changes, trade-offs between precision and recall, and systematic under-recall of rare and long-term side effects. When used alone, constraints on the number of side effects generated reduce precision, and grounding outputs in clinician-curated side effect lists substantially improves reliability and robustness. These findings highlight important limitations of LLM use in oncology and suggest practical design choices for safer and more informative survivorship-focused applications.
Read the full abstract ▾
All authors Natalie Seah, Danielle S. Bitterman, Daphna Spiegel, Thomas Hartvigsen
Computation and Language
Magis-Bench: Evaluating LLMs on Magistrate-Level Legal Tasks
Ramon Pires, Thales Sales Almeida, Celio Larcher Junior et al. (+6 more)
Existing
benchmarks for legal AI focus primarily on tasks where LLMs must produce legal arguments or documents, yet the capacity to judge such arguments – weighing competing claims, applying doctrine to facts, and rendering reasoned decisions – is arguably as fundamental to a well-functioning legal system as advocacy itself. We introduce Magis-Bench, a
benchmark for evaluating LLMs on magistrate-level writing tasks derived from recent Brazilian competitive examinations for judicial positions. Magis-Bench comprises 74 questions from eight examinations conducted between 2023 and 2025, including discursive legal analysis questions with multi-turn structure and practical exercises requiring the composition of complete civil and criminal judicial sentences. We evaluate 23 state-of-the-art LLMs using an LLM-as-a-judge methodology with four independent frontier models as evaluators. Our results show strong inter-judge agreement (Kendall's $W = 0.984$; pairwise Kendall's $\tau \ge 0.897$), with Google's Gemini-3-Pro-Preview achieving the highest average score (6.97/10), followed by Gemini-3-Flash-Preview (6.67) and Claude-4.5-Opus (6.46). Even the best-performing models score below 70% of the maximum, indicating that judicial-level legal
reasoning and writing remain challenging for current LLMs. We release the complete
benchmark, model outputs, and evaluation code to support further research on legal AI capabilities.
Read the full abstract ▾
All authors Ramon Pires, Thales Sales Almeida, Celio Larcher Junior, Giovana Bonás, Hugo Abonizio, Marcos Piau, Roseval Malaquias Junior, Thiago Laitz, Rodrigo Nogueira
Computation and Language
A Semantic-Sampling Framework for Evaluating Calibration in Open-Ended Question Answering
Zhanliang Wang, Jiancong Xiao, Ruochen Jin et al. (+3 more)
Calibration measures whether a model's predicted confidence aligns with its empirical accuracy, and is central to the reliable deployment of large
language models (LLMs) in high-stakes domains such as medicine and law. While much recent work focuses on improving LLM calibration, the equally important question of how to evaluate it in realistic settings remains underdeveloped. Open-ended question answering (QA), the most common deployment setting for modern LLMs, is where existing evaluation methods fall short: logit-based metrics need restricted output formats and internal probabilities; verbalized confidence is self-reported and often overconfident; and sampling-based methods rely on task-specific extraction rules without a clear finite-sample target. We introduce Sem-ECE (Semantic-Sampling Expected Calibration Error), a calibration evaluation framework for open-ended QA that samples answers from the model, groups them into semantic classes, and uses the resulting frequencies as confidence. We study two
estimators within this framework: Sem$_1$-ECE, the same-sample self-consistency score, and Sem$_2$-ECE, a held-out variant that separates answer selection from confidence evaluation. We prove both are asymptotically unbiased, and further show that they agree on easy questions but diverge on hard ones with Sem$_2$ achieving strictly smaller calibration error, so their gap also serves as a diagnostic for question difficulty. Experiments on three open-ended QA
benchmarks across five leading commercial LLMs match our theoretical predictions and show that Sem-ECE outperforms verbalized confidence and existing sampling-based methods, while complementing logit-based evaluation when internal probabilities are unavailable.
Read the full abstract ▾
All authors Zhanliang Wang, Jiancong Xiao, Ruochen Jin, Shu Yang, Bojian Hou, Li Shen
Computation and Language · Cross-listed from cs.LG
Queryable LoRA: Instruction-Regularized Routing Over Shared Low-Rank Update Atoms
Omatharv Bharat Vaidya, Connor T. Jerzak, Nhat Ho et al. (+1 more)
We present a data-adaptive method for parameter-efficient fine-tuning of large neural networks. Standard low-rank adaptation methods improve efficiency by restricting each layer update to a fixed low-rank form, but this static parameterization can be too rigid when the appropriate correction depends on the input and on the evolving depth-wise computation of the network. Our approach replaces a purely layer-local adapter with a shared queryable
memory of low-rank update atoms. For each block of layers, the model forms a query from the current low-rank state and a running summary of previous blocks, uses this query to retrieve a content-dependent combination of shared update components via
attention, and applies the resulting routed operator within the low-rank bottleneck. In this way, the method retains the efficiency and scalability of low-rank adaptation while allowing the effective update to vary across inputs and to share reusable structure across layers. The resulting architecture provides a principled middle ground between static LoRA-style updates and fully generated parameter updates: it remains compact and parameter-efficient while supporting dynamic, context-sensitive adaptation. Further, we incorporate instruction-regularization by augmenting routing logits with a
language-induced prior over update atoms, thereby biasing the selection of low-rank transformations toward semantically relevant directions without generating unconstrained parameter updates. Experiments on noisy non-linear regression tasks and LLM fine-tuning suggest that this queryable update-
memory formulation can improve final test performance and training stability compared to standard low-rank adaptation, while using a comparable number of trainable parameters.
Read the full abstract ▾
All authors Omatharv Bharat Vaidya, Connor T. Jerzak, Nhat Ho, Chandrajit Bajaj
Computation and Language
Effective Explanations Support Planning Under Uncertainty
Hanqi Zhou, Britt Besch, Charley M. Wu et al. (+1 more)
Explaining how to get from A to B can be challenging. It requires mentally simulating what the listener will do based on what they are told. To capture this process, we propose a computational model that converts utterances into action plans: a large
language model translates an explanation into program-like guidance (a policy prior and value map), and a planning
agent executes it under partial observability. We score explanations by the efficiency and reliability of the resulting paths, penalizing replanning. Across four preregistered experiments, we collect a corpus of 1,200 explanations over 24 maps, elicit helpfulness judgments, measure baseline
navigation, and test behavior with explanations of differing quality. Higher-scored explanations are judged more helpful and improve
navigation: participants with explanations outperform those without, and high-scoring explanations help more than low-scoring ones. Together, these results show procedural explanation as utility-guided communication shaped by how
language can be grounded into action under uncertainty.
Read the full abstract ▾
All authors Hanqi Zhou, Britt Besch, Charley M. Wu, Tobias Gerstenberg
Computation and Language
Built Environment Reasoning from Remote Sensing Imagery Using Large Vision–Language Models
Dongdong Wang, Deepak Balakrishnan, Ravi Srinivasan et al. (+1 more)
This work investigates the use of large
language models (LLMs) for tasks in smart cities. The core idea is to leverage remote sensing imagery to characterize the built environment, including design suggestions, constructability assessment, landuse patterns, and risk identification. We examine remote sensing imagery at multiple spatial scales as inputs for
multimodal language modeling and evaluate their effects on built-environment-related
reasoning. In addition, we compare state-of-the-art LLMs, including InternVL and Qwen, in terms of accuracy and reliability when generating built environment recommendations. The results demonstrate the potential of integrating remote sensing imagery with large
language models to assist smart cities and decision-making.
Read the full abstract ▾
All authors Dongdong Wang, Deepak Balakrishnan, Ravi Srinivasan, Shenhao Wang
Computation and Language
AIPO: Learning to Reason from Active Interaction
Junnan Liu, Linhao Luo, Thuy-Trang Vu et al. (+1 more)
Recent advances in LLMs have demonstrated strong
reasoning capabilities, largely stimulated by RLVR. However, the exploration of existing RLVR
algorithms remains largely constrained by the knowledge and
reasoning strategies already accessible to the policy model. Although recent methods introduce external expert demonstrations to broaden exploration, they typically rely on complete trajectory-level guidance, which can be sample-inefficient, information-sparse, and insufficiently adaptive to intermediate
reasoning bottlenecks. Inspired by collaborative multi-
agent systems, we propose AIPO, an enhanced reinforcement learning framework that improves LLM
reasoning through active multi-
agent interaction during exploration. Specifically, when encountering
reasoning bottlenecks, AIPO enables the policy model to proactively consult three functional collaborative
agents, namely the verify
agent, knowledge
agent, and
reasoning agent, thereby obtaining fine-grained and state-dependent guidance during rollout. The resulting mixed-policy trajectories expose the policy to
reasoning directions that may be difficult to discover through isolated on-policy exploration. To learn effectively from collaborator-provided
tokens, we further introduce a corrected importance sampling coefficient together with a lower-bound clipping strategy to mitigate off-policy discrepancy and vanishing gradients. After training, the policy model reasons independently without relying on collaborative
agents. Extensive experiments across mathematical, scientific, coding, and puzzle
reasoning benchmarks show that AIPO consistently improves
reasoning performance and generalizes across different policy models, collaborator backbones, and RLVR
algorithms.
Read the full abstract ▾
All authors Junnan Liu, Linhao Luo, Thuy-Trang Vu, Gholamreza Haffari
Computation and Language
jina-embeddings-v5-omni: Geometry-preserving Embeddings via Locked Aligned Towers
Florian Hönicke, Michael Günther, Andreas Koukounas et al. (+3 more)
In this work, we introduce GELATO (Geometry-preserving
Embeddings via Locked Aligned TOwers), a novel approach to
multimodal embedding models. We build on the VLM-style architecture, in which non-text encoders are adapted to produce input for a
language model, which in turn generates
embeddings for all varieties of input. We present the result: the jina-
embeddings-v5-omni suite, a pair of models that encode text, image, audio, and
video input into a single semantic
embedding space. GELATO extends the two Jina
Embeddings v5 Text models to support additional modality by adding encoders for images and audio. The backbone text
embedding models and the added non-text modality encoders remain frozen. We only trained the connecting components, representing 0.35% of the total weights of the joint model. Training is therefore much more efficient than full-parameter retraining. Additionally, the
language model remains effectively unaltered, producing exactly the same
embeddings for text inputs as the Jina
Embeddings v5 Text models. Our evaluations show that GELATO produces results that are competitive with the state-of-the-art, yielding nearly equal performance to larger
multimodal embedding models.
Read the full abstract ▾
All authors Florian Hönicke, Michael Günther, Andreas Koukounas, Mohammad Kalim Akram, Saba Sturua, Han Xiao
Computation and Language
Change My View? The Dynamics of Persuasion and Polarization in Online Discourse
David Freeborn, Malihe Alikani and Anthony Sicilia
Philosophical accounts of persuasion often assume that shared evidence and rational argumentation should lead to a convergence of views between peers, yet everyday discourse often suggests otherwise. In this study, we use large
language models to analyze a corpus of debates on Reddit's r/ChangeMyView, where belief revision is publicly signaled. Large
language models were asked, halfway through each discussion, to forecast whether such an acknowledgement would arise; their probabilistic estimates serve as a conversational baseline. Each reply was then coded, through a hybrid machine-assisted procedure, for ten familiar rhetorical strategies – concession, empathy, logical challenge, credibility appeals, and so forth. Adding these strategic features markedly improves predictive power and yields a consistent pattern: moves that express concession or empathetic
alignment substantially increase the prospect of belief change, whereas frontal refutation, credibility attacks, and topic deflection diminish it. The findings indicate that effective public
reasoning depends as much on relational framing as on evidential content, and they invite a refinement of normative accounts of rational dialogue.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.CR
SecureForge: Finding and Preventing Vulnerabilities in LLM-Generated Code via Prompt Optimization
Houjun Liu, Lisa Einstein, John Yang et al. (+5 more)
LLM coding
agents now generate code at an unprecedented scale, yet LLM-generated code introduces cybersecurity vulnerabilities into codebases without human involvement. Even when frontier models are explicitly asked to write secure production code with relevant weaknesses to avoid in context, we find that they still produce verifiable vulnerabilities on average 23% of the time across a corpus of 250 benign coding
prompts. We introduce SecureForge, an automated pipeline that both audits security risks of frontier models and produces
auditing-informed secure system
prompts that reduce output security vulnerabilities while maintaining unit test performance. SecureForge first identifies benign
prompts that produce statically detectable vulnerabilities, and then amplifies them into a large synthetic
prompt corpus of diverse scenarios using a Markovian sampling technique to jointly maintain error rates and
prompt diversity. This corpus is then used to iteratively optimize the system
prompts to reduce output security vulnerabilities. On frontier models, SecureForge yields a statistically significant Pareto improvement in both unit test success and output security, with output vulnerabilities reduced by up to 48%. The resulting system
prompts transfer zero-shot to in-the-wild coding
agent prompts, without any exposure to real user
prompt distributions during optimization.
Read the full abstract ▾
All authors Houjun Liu, Lisa Einstein, John Yang, Joachim Baumann, Duncan Eddy, Christopher D. Manning, Mykel Kochenderfer, Diyi Yang
Computation and Language · Cross-listed from cs.LG
Reinforcement Learning for Scalable and Trustworthy Intelligent Systems
Guangchen Lan
Reinforcement learning has become a powerful paradigm for improving the capability of intelligent systems, but its practical deployment faces two central challenges. First, reinforcement learning must scale efficiently in distributed environments where communication bandwidth is limited and computation is heterogeneous across
agents. Second, as reinforcement learning is increasingly used in post-training large
language models and autonomous
agents, the optimized policies must also be aligned with human preferences and satisfy safety requirements such as
privacy-aware information disclosure. This dissertation addresses both challenges through four complementary contributions spanning
federated optimization, preference
alignment, and contextual safety. The first part of the dissertation studies scalable reinforcement learning in
federated settings. The second part of the dissertation studies trustworthy reinforcement learning for large
language models. Together, these contributions advance reinforcement learning along two complementary dimensions. On the one hand, they make reinforcement learning more scalable through communication-efficient and asynchronous
federated optimization. On the other hand, they make reinforcement learning more trustworthy by improving
alignment with human preferences and by reducing contextually inappropriate information disclosure in
language-based intelligent systems. As a whole, this dissertation argues that the next generation of intelligent systems will require both efficient optimization and trustworthy behavior, and that reinforcement learning provides a unifying framework for addressing both goals.
Read the full abstract ▾
Computation and Language
How Much Do Circuits Tell Us? Measuring the Consistency and Specificity of Language Model Circuits
Michael Li and Nishant Subramani
The
circuits framework in mechanistic interpretability aims to identify sparse subgraphs of model components that are causally responsible for a behavior, typically evaluated by measuring necessity and sufficiency. But these criteria say little about whether a
circuit consistently captures how a model performs a task, or if it is specific to that task. We study these two properties, consistency and specificity, across six tasks and five models, extracting
circuits at the component level (
attention heads and MLP blocks) and at the level of individual MLP
neurons. We find that component-level
circuits are highly consistent and causally important for most tasks, but they are not specific: for a given task, ablating its own
circuit drops accuracy by about as much as ablating another task's
circuit.
Neuron-level
circuits, on the other hand, exhibit higher specificity across tasks, but far less consistency within tasks. This is explained by
circuit overlap: component-level
circuits share most of their components across tasks, related or not, while
neuron-level
circuits overlap only between closely related tasks. In a case study of the components shared by the task
circuits of Llama-3.2-3B, we show that they consist mostly of MLP blocks, while the few
attention heads within turn out to have general-purpose roles. Overall, our findings raise questions about the degree to which
circuits can support targeted understanding of, and intervention on, model behavior.
Read the full abstract ▾
Computation and Language
Sanity Checks for Long-Form Hallucination Detection
Geigh Zollicoffer, Minh Vu, Hongli Zhan et al. (+2 more)
Hallucination detection methods for large
language models increasingly operate on chain-of-thought
reasoning traces, yet it remains unclear whether they evaluate the
reasoning itself or merely exploit surface correlates of the final answer. We introduce a controlled-invariance methodology that exposes this distinction through two oracle tests: Force, which replaces each response's final answer with the ground truth while preserving the
reasoning trace, and Remove, which strips answer-announcement steps while leaving the trajectory intact. This reveals if their predictive power derives from answer-level artifacts rather than from the structure or validity of intermediate
reasoning. We further show that once these artifacts are controlled for, effective detection does not necessarily require complex learned representations: TRACT, a lightweight scorer built on lexical trajectory features (hedging trends, step-length dynamics, and cross-response vocabulary convergence), achieves strong robustness while remaining competitive with or outperforming existing baselines on unperturbed traces. These findings suggest that the current central challenge in
reasoning-aware
hallucination detection is not the absence of signal in the trace, but the failure to isolate it from endpoint cues.
Read the full abstract ▾
All authors Geigh Zollicoffer, Minh Vu, Hongli Zhan, Raymond Li, Manish Bhattarai
Computation and Language
CustomerSim: Benchmarking and Aligning Multimodal Language Models as Retail User Simulators
Yada Pruksachatkun, Yixin Wan, Xingrun Chen et al. (+2 more)
We present CustomerSim, an environment and
benchmark to evaluate the extent to which
Multimodal Large
Language Models (MLLMs) can simulate realistic, persona-driven customer behavior in chat-based retail environments. While prior work treats user simulation as surface-level dialog generation, we focus on a model's ability to seek information and make decisions that adhere to customer specifications in multiturn, agentic simulations. CustomerSim consists of a human-curated set of 360 personas over five product categories, alongside a suite of metrics measuring consistency between a customer simulator's actions and its specifications and conversational quality. We find several behavioral gaps across five open and closed-source state-of-the-art models. First, while models produce fluent conversations, they display significantly lower lexical diversity than human shoppers, and open-source models overdisclose their criteria in the opening turn. Second, models tend to be persuaded by sales
agent tone and drift from persona specifications. Even the strongest closed-source models, Claude Opus 4.8 and GPT-5.6 Sol, achieves <74%
alignment with its persona specifications. To address these limitations, we propose UserGRPO, a multi-turn, multi-objective reinforcement learning recipe optimizing both conversational fluency and decision
alignment under persona specifications. UserGRPO raises the decision
alignment of the baseline model from 0.417 to 0.652, a gain of 23.5 points, without meaningful cost to conversational quality, and these gains transfer to held-out product categories. We further find that stylistic prompting is the only intervention that makes surface form more human-like, yet it nearly halves persona adherence. Through CustomerSim, we provide a testbed for the community to investigate and improve the adherence of user simulators in goal-oriented settings.
Read the full abstract ▾
All authors Yada Pruksachatkun, Yixin Wan, Xingrun Chen, Kai-Wei Chang, Chien-Sheng Wu
Computation and Language · Cross-listed from cs.LG
CDS4RAG: Cyclic Dual-Sequential Hyperparameter Optimization for RAG
Pengzhou Chen and Tao Chen
Retrieval-Augmented Generation (RAG) is sensitive to the vast hyperparameters of the retriever and generator, yet optimizing them using given queries is a challenging task due to the complex interactions and expensive evaluation costs. Existing
algorithms are ineffective and slow in convergence, since they often treat RAG as a monolithic black box or only optimize partial hyperparameters. In this paper, we propose CDS4RAG, a framework that optimizes the full RAG hyperparameters using given queries via a new cyclic dual-sequential formulation. CDS4RAG is special in the sense that it distinguishes the hyperparameters of the retriever and generator, cyclically optimizing them in turn. Such a paradigm allows us to design fine-grained within-cycle budget provision and expedite the optimization via cross-cycle seeding when optimizing the generator. CDS4RAG is also an
algorithm-agnostic framework that can be paired with diverse general
algorithms. Through experiments on four common
benchmarks and two backbone LLMs, we reveal that CDS4RAG considerably boosts the vanilla
algorithms in 21/24 cases while significantly outperforming state-of-the-art
algorithms in all cases with up to 1.54x improvements of generation quality and better speedup.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.LG
LLMSYS-HPOBench: Hyperparameter Optimization Benchmark Suite for Real-World LLM Systems
Siyu Wu, Yulong Ye, Zezhen Xiang et al. (+3 more)
Large
Language Model (LLM) systems have been the frontier of AI in many application domains, leading to new challenges and opportunities for hyperparameter optimization (HPO) for the AutoML community. However, this type of system exhibits an unprecedented compound space of hyperparameter configuration from both the AI and non-AI components; rich and nonlinear implications from the fidelity factors; and diverse costs of measuring hyperparameter configurations, none of which have been fully captured in existing
benchmarks. This paper presents the first (live)
benchmark suite and
datasets for HPO of real-world LLM systems, dubbed LLMSYS-HPOBench, covering data related to the inference objective values of hyperparameter configurations profiled from running the LLM systems. Currently, LLMSYS-HPOBench contains 364,450 hyperparameter configurations with a dimensionality of 12-23, 3-5 dimensions of fidelity factor leading to 932 settings, 3-9 inference objective metrics, and 2-10 cost metrics, together with generated logs from measuring the LLM systems. What we seek to advocate is not only a revalidation of the existing HPO
algorithms over the frontier LLM systems, but also to provide an evolving platform for the AutoML community to explore new directions of research in this regard. The
benchmark suite has been made available at: https://github.com/ideas-labo/llmsys-hpobench
Read the full abstract ▾
All authors Siyu Wu, Yulong Ye, Zezhen Xiang, Pengzhou Chen, Gangda Xiong, Tao Chen
Computation and Language · Cross-listed from cs.LG
mHC-SSM: Manifold-Constrained Hyper-Connections for State Space Language Models with Stream-Specialized Adapters
Abdulvahap Mutlu, Şengül Doğan and Türker Tuncer
Manifold-Constrained Hyper-Connections (mHC) introduce a stability-motivated variant of multi stream residual mixing by constraining residual stream mixing
matrices to the
manifold of doubly
stochastic matrices via Sinkhorn-Knopp projection. In his work, we study whether mHC-style constrained multi-stream residual
topology transfers effectively to state space model (SSM)
language modeling. We implement a static mHC mechanism around an SSM block by expanding the residual stream into multiple parallel streams, aggregating streams into a single SSM input through simplex-constrained pre-mixing, scattering the SSM output back to streams through simplex-constrained post-mixing, and applying Sinkhorn-projected residual stream mixing at each layer. We further introduce stream-specialized adapters that add lightweight stream-specific capacity through a shared bottleneck with per-stream scaling, applied both before stream aggregation and after the SSM output prior to scattering. We evaluate baseline single-stream SSM, static mHC SSM, and mHC SSM with adapters on WikiText-2 using identical training settings and report checkpoint-based validation loss, perplexity, throughput, and peak GPU
memory. Under the reported fair checkpoint evaluation, static mHC improves validation loss from 6.3507 to 6.2448 and reduces perplexity from 572.91 to 515.35, while mHC with adapters further improves validation loss to 6.1353 and perplexity to 461.88. These gains are accompanied by modest throughput reductions from 1025.52 to 964.81 and 938.90
tokens per second, and increased peak
memory from 2365 MB to 2568 MB and 3092 MB. The results suggest that mHC-inspired constrained multi-stream residual mixing can yield measurable quality improvements in SSM
language models and that stream-specialized adapter capacity can further enhance performance with predictable efficiency tradeoffs.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.LG
In-Context Fixation: When Demonstrated Labels Override Semantics in Few-Shot Classification
Ming Liu
While random demonstration labels barely hurt in-context learning (Min et al., 2022), we show that homogeneous labels–even semantically valid ones–collapse accuracy to <=12% across six models (Pythia, Llama, Qwen; 0.8B–8B) and four tasks. The trigger is label-slot content: the model treats
tokens occupying the label position as an exhaustive answer vocabulary, with homogeneity as the maximally collapsed case. A novel set-level fixation finding confirms this: when demonstrations carry varied nonsense
tokens from {foo,bar,vex,nit,orb}, the model places 42–67% of probability on the demonstrated set while P(dog) remains below 0.2%. This is inconsistent with latent-concept
Bayesian accounts (Xie et al., 2022) and reveals that ICL output is constrained vocabulary
retrieval–the model binds its output to the demonstrated
token inventory regardless of semantic plausibility. The effect generalizes to 4-way classification (0% accuracy across three models, 1B–8B) and multi-
token verbalizers ("very positive"), where we decompose fixation into format-level (template adoption) and content-level (polarity override) components that are experimentally dissociable. Mechanistically, per-item paired activation patching on Pythia-1B recovers 98.4% of the gap (95% CI [84%, 112%]), localizing fixation to a layer-7-centered
circuit (rank 2/560, 99.8th percentile; 4-fold CV mean 103%). Cross-architecture logit lens on Llama-3.2-1B replicates the encode-then-override trajectory with causal confirmation (top-5 layers: 89% recovery).
Read the full abstract ▾
Computation and Language · Cross-listed from cs.LG
HTPO: Towards Exploration-Exploitation Balanced Policy Optimization via Hierarchical Token-level Objective Control
Xincheng Yao, Ruoqi Li, Cheng Chen et al. (+4 more)
Reinforcement Learning with Verifiable Rewards (RLVR) has emerged as a pivotal technique for enhancing the
reasoning capabilities of Large
Language Models (LLMs). However, the de facto practice of mainstream RL
algorithms is to treat all
tokens of one response equally and assign the same optimization objective to each
token, failing to provide granular guidance for the
reasoning process. While in Chain-of-Thought (CoT)
reasoning, different
tokens usually play distinct roles. Therefore, the current RL
algorithms lack an effective mechanism to dynamically balance the exploration-exploitation trade-off during learning. To this end, we propose Hierarchical
Token-level Objective Control Policy Optimization (HTPO), a novel RL
algorithm that takes the divide-and-conquer idea to hierarchically partition the response
tokens into specific functional groups from three aspects (i.e.,
prompt difficulty, answer correctness, and
token entropy). Within each group, according to the contributions to exploration or exploitation, we design specialized optimization objectives to facilitate the effective execution of each
token's expected functionality. In this way, HTPO can achieve a more balanced exploration-exploitation trade-off. Extensive experiments on challenging
reasoning benchmarks validate the superiority of our HTPO
algorithm, which significantly outperforms the strong DAPO baseline (e.g., +8.6% and +6.7% on AIME'24 and AIME'25, respectively). When scaling test-time compute, the HTPO-trained model maintains a consistent performance advantage over the DAPO baseline, and the gap widens as the sampling budget increases, validating that our adaptive
token-level control method fosters effective exploration without sacrificing exploitation performance. Code will be at https://github.com/xcyao00/HTPO.
Read the full abstract ▾
All authors Xincheng Yao, Ruoqi Li, Cheng Chen, Daoxin Zhang, Yi Wu, Yao Hu, Chongyang Zhang
Computation and Language
LLMs Improving LLMs: Agentic Discovery for Test-Time Scaling
Tong Zheng, Haolin Liu, Chengsong Huang et al. (+10 more)
Test-time scaling (TTS) has become an effective approach for improving large
language model performance by allocating additional computation during inference. However, existing TTS strategies are largely hand-crafted: researchers manually design
reasoning patterns and tune heuristics by intuition, leaving much of the computation-allocation space unexplored. We propose an environment-driven framework, AutoTTS, that changes what researchers design: from individual TTS heuristics to environments where TTS strategies can be discovered automatically. The key to AutoTTS lies in environment construction: the discovery environment must make the control space tractable and provide cheap, frequent feedback for TTS search. As a concrete instantiation, we formulate width–depth TTS as controller synthesis over pre-collected
reasoning trajectories and probe signals, where controllers decide when to branch, continue, probe, prune, or stop and can be evaluated cheaply without repeated LLM calls. We further introduce beta parameterization to make the search tractable and fine-grained execution trace feedback to improve discovery efficiency by helping the
agent diagnose why a TTS program fails. Experiments on mathematical
reasoning benchmarks show that the discovered strategies improve the overall accuracy–cost tradeoff over strong manually designed baselines. The discovered strategies generalize to held-out
benchmarks and model scales, while the entire discovery costs only $39.9 and 160 minutes. Our data, and code will be open-source at https://github.com/zhengkid/AutoTTS.
Read the full abstract ▾
All authors Tong Zheng, Haolin Liu, Chengsong Huang, Huiwen Bao, Sheng Zhang, Rui Liu, Runpeng Dai, Ruibo Chen, Chenxi Liu, Tianyi Xiong, Xidong Wu, Hongming Zhang, Heng Huang
Computation and Language
Conformal Path Reasoning: Trustworthy Knowledge Graph Question Answering via Path-Level Calibration
Shuhang Lin, Chuhao Zhou, Xiao Lin et al. (+5 more)
Knowledge Graph Question Answering (KGQA) offers grounded, interpretable
reasoning, but existing methods often fail to provide reliable coverage guarantees over retrieved answers. While Conformal Prediction (CP) offers a principled framework for producing prediction sets with statistical guarantees, prior conformal KGQA methods suffer from two critical pitfalls: violated coverage guarantees due to invalid calibration, and weak score discriminability that yields excessively large prediction sets. We propose Conformal Path
Reasoning (CPR), a novel trustworthy KGQA framework built on two key innovations. First, query-level conformal calibration over path-level scores preserves exchangeability to ensure valid coverage guarantees. Second, we introduce the Residual Conformal Value Network (RCVNet), a lightweight module trained via PUCT-guided exploration to learn discriminative path-level nonconformity scores. Extensive experiments show that CPR significantly improves the Empirical Coverage Rate by 45% while reducing prediction set size by 52% on average over conformal baselines across
benchmark datasets, highlighting its effectiveness for reliable conformal
reasoning over knowledge graphs.
Read the full abstract ▾
All authors Shuhang Lin, Chuhao Zhou, Xiao Lin, Zihan Dong, Kuan Lu, Zhencan Peng, Jie Yin, Dimitris N. Metaxas
Computation and Language
The Memory Curse: How Expanded Recall Erodes Cooperative Intent in LLM Agents
Jiayuan Liu, Tianqin Li, Shiyi Du et al. (+7 more)
Context window expansion is often treated as a straightforward capability upgrade for LLMs, but we find it systematically fails in multi-
agent social dilemmas. Across 7 LLMs and 4 games over 500 rounds, expanding accessible history degrades cooperation in 18 of 28 model–game settings, a pattern we term the
memory curse. We isolate the underlying mechanism through three analyses. First, lexical analysis of 378,000
reasoning traces associates this breakdown with eroding forward-looking intent rather than rising paranoia. We validate this using targeted fine-tuning as a cognitive probe: a LoRA adapter trained exclusively on forward-looking traces mitigates the decay and transfers zero-shot to distinct games. Second,
memory sanitization holds
prompt length fixed while replacing visible history with synthetic cooperative records, which restores cooperation substantially, proving the trigger is
memory content, not length alone. Finally, ablating explicit Chain-of-Thought
reasoning often reduces the collapse, showing that deliberation paradoxically amplifies the
memory curse. Together, these results recast
memory as an active determinant of multi-
agent behavior: longer recall can either destabilize or support cooperation depending on the
reasoning patterns it elicits.
Read the full abstract ▾
All authors Jiayuan Liu, Tianqin Li, Shiyi Du, Xin Luo, Haoxuan Zeng, Emanuel Tewolde, Tai Sing Lee, Tonghan Wang, Carl Kingsford, Vincent Conitzer
Computation and Language
CA-SQL: Complexity-Aware Inference Time Reasoning for Text-to-SQL via Exploration and Compute Budget Allocation
James Petullo and Nianwen Xue
While recent advancements in inference-time learning have improved LLM
reasoning on Text-to-SQL tasks, current solutions still struggle to perform well on the most challenging tasks in the Bird-Bench (BIRD)
benchmark. This is due to inadequate solution space exploration, which is necessary to uncover promising candidate queries that can be further refined to produce the correct output. To address this challenge, we introduce CA-SQL, a novel Text-to-SQL pipeline that utilizes the estimated difficulty of a task to dynamically scale the breadth of the exploration for generating solution candidates. In addition, we use a custom
prompt seeding method, based on principles of evolutionary search, to further elicit exploratory behavior from the base LLM and a novel voting method to select the best candidate solution at the end of the search. Experiments demonstrate that our solution achieves a state-of-the-art score of 51.72% on the "challenging" tier of BIRD development set problems, using only GPT-4o-mini, out-performing other in-context learning approaches, even those that leverage larger models. Overall, our method attains a competitive 61.06% execution accuracy and 68.77% Soft F1 score on the BIRD development
dataset.
Read the full abstract ▾
Computation and Language
Accurate and Efficient Statistical Testing for Word Semantic Breadth
Yo Ehara
Measuring the breadth of a word's meaning, or its spread across contexts, has become feasible with contextualized
token embeddings. A word type can be represented as a cloud of
token vectors, with dispersion-based statistics serving as proxies for contextual diversity (Nagata and Tanaka-Ishii, ACL2025). These measurements are useful for deciding appropriate sense distinctions when constructing thesauri and domain-specific dictionaries. However, when comparing the breadth of two word types, naive hypothesis testing on dispersion can be misleading: differences in semantic direction can masquerade as dispersion differences, inflating Type-I error and yielding "statistically significant" outcomes even when there is no true breadth difference. This is problematic because significance testing should distinguish genuine effects from incidental fluctuations in small-difference regimes. We propose a Householder-aligned permutation test to isolate dispersion differences from directional differences. Our method applies a single Householder reflection to align the mean directions of the two word types and then performs a permutation test on the aligned
token clouds, yielding calibrated, non-parametric p-values. For practicality, we introduce a GPU-oriented implementation that batches permutations and linear
algebra operations. Empirically, our
alignment reduced Type-I error by 32.5% while preserving sensitivity to genuine breadth differences, and achieved a 23x speedup over the CPU baseline.
Read the full abstract ▾