Artificial Intelligence · Cross-listed from cs.CL
AI-Enabled Quality Assurance for Multiple-Choice Assessment Items
Steven Moore and Nicholas Diana
Generating multiple-choice questions is increasingly scalable, but establishing their assessment quality remains difficult. We present a focused narrative review of automated item-writing flaw detection, revision, psychometric screening, and NLP
benchmark auditing. Database searches, citation
retrieval, and nominated sources yield fourteen research reports reviewed in full text. We distinguish surface checks from content-sensitive judgments and map a 19-criterion rubric to detection methods and reported evidence. High label-level accuracy often coexists with weak positive case detection, while rubric definitions and reference standards vary. Revision evidence is mixed, and the associations reported in prior work do not establish the effects of repair. We propose evaluating quality assurance as a sequence of independently validated decisions, with criterion-specific reporting, calibrated human review, and outcome-based assessment of revisions.
Read the full abstract ▾
Artificial Intelligence
CADForge: Agentic Single-View CAD Reconstruction with Explicit Geometry Reasoning
Keyang Lu, Zhifei Yang, Tianao Dong et al. (+3 more)
Reconstructing editable parametric CAD models from a single-view image is of great practical value for modern manufacturing, yet remains challenging due to incomplete
geometric observations and complex inter-part relationships. To address it, we propose CADForge, an agentic framework that progressively converts a single image into CadQuery programs. CADForge decomposes an object into CAD-meaningful components and performs explicit
geometric reasoning for each component, a process that first identifies CAD-relevant constraints and then translates them into precise modeling parameters through mathematical code. The inferred parameters then drive component-wise synthesis of executable CadQuery programs, with a review
agent evaluating the resulting geometry and providing targeted feedback for iterative refinement. To further improve robustness and efficiency, CADForge incorporates a failure-guided toolkit construction mechanism to distill accumulated experience into tools, and maintains a compact parametric CAD
memory for retrieving modeling context on demand. Experiments on diverse single- and multi-part objects show that CADForge consistently outperforms existing baselines in reconstruction fidelity and perceptual quality, demonstrating an effective approach to accurate single-view CAD reconstruction.
Read the full abstract ▾
All authors Keyang Lu, Zhifei Yang, Tianao Dong, Mingzhe Xing, Zhen Xiao, Yikai Wang
Artificial Intelligence · Cross-listed from cs.CL
Playing social deduction games with reinforcement fine-tuned large language models
Lingzhe Zhang, Yunpeng Zhai, Tong Jia et al. (+7 more)
Reinforcement fine-tuning (RFT) is increasingly used in applications where large
language models (LLMs) interact with humans and other
agents. Here we use social deduction games to study how RFT changes LLMs' social behaviour. We let fine-tuned and base LLM
agents play hidden-role games that require hidden-state inference, social reading and vote steering. Our results show that LLM
agents do not reliably acquire social-deduction ability by directly optimizing terminal win–loss outcomes, suggesting that final game results provide a sparse and noisy signal for socially interactive learning. However, RFT is particularly effective at improving social reading, including tasks that require
agents to infer hidden roles from public discussion, update beliefs over time and predict other
agents' future decisions. We further show that RFT can also improve social influence, including tasks that require
agents to steer votes, team approvals and collective decisions, although these gains depend more strongly on behaviourally specific rewards and structured interaction settings. Finally, we show that LLMs' ability to play social deduction games can be further improved through multi-
agent social-cognitive reinforcement fine-tuning, which combines social-reading and social-influence signals during same-side multi-
agent training. These learned behaviours also receive more favourable human evaluations of strategic competence, persuasiveness and social usefulness. Together, these results enrich our understanding of how RFT changes LLMs' social behaviour and provide a step toward a behavioural learning theory for machine social intelligence.
Read the full abstract ▾
All authors Lingzhe Zhang, Yunpeng Zhai, Tong Jia, Kening Zheng, Chiming Duan, Minghua He, Zhaoyang Liu, Bolin Ding, Philip S. Yu, Ying Li
Artificial Intelligence
Spec2Game: Can LLMs Generate Complete Playable Games from Detailed Specifications?
Yixue Cai, Yuzhe Zhao, Hanxiang Chao et al. (+4 more)
Generating an executable program does not necessarily mean that it correctly implements the behavioral requirements specified in natural
language. To evaluate large
language models' ability to realize detailed specifications as complete interactive programs, we introduce Spec2Game, a
benchmark that requires models to generate complete Pygame projects from detailed natural-
language game specifications. Spec2Game comprises 15 game families and 150 task instances, with one canonical task and nine controlled rule variants per family, spanning three levels of implementation complexity. Using source-code, runtime, and visual evidence, we evaluate generated projects along four dimensions—Executability, Specification Realization, Code Quality, and User-Facing Quality. Across 14 LLMs and 3,330 generated projects, we find that high executability does not imply faithful specification realization. Component-level analysis further shows that models perform substantially better on Game Element Modeling than on Rule and Mechanism Modeling or Goal and Termination Modeling, indicating that faithfully implementing game rules and termination logic remains a major challenge.
Read the full abstract ▾
All authors Yixue Cai, Yuzhe Zhao, Hanxiang Chao, Qingsen Ma, Ziheng Xiong, Jinhu Qi, Irwin King
Artificial Intelligence · Cross-listed from cs.HC
Benchmarking Psychological Dynamics in Generative Agents
Sumer S. Vaid and Ashley V. Whillans
Large
language models (LLMs) are increasingly deployed to simulate human behavior, acting as computational replicas of human subjects. Yet the lived psychological experience of humans is difficult to
benchmark, particularly as it unfolds over time. We introduce a psychometric
benchmark for computational replicas: personas that carry a fixed identity through an evolving sequence of events. Built entirely from published norms and meta-analytic effects, the
benchmark scores two dimensions of psychological realism. The first, internal validity, quantifies whether generated trajectories reproduce the internal structure of repeated human measurement: distributions, the between- versus within-person variance partition, temporal dependence, and range. The second, external validity, quantifies whether replicas recover established trait, state, and indicator relations. Across 36 open-weight and proprietary LLMs from nine developers (1B-671B parameters), most recover the direction of established relations (84.3% mean agreement) and the variance partition (26 of 34), yet the within-person correlation and distributional structure elude recovery. Internal validity is independent of scale and capability: a mid-size open LLM (Gemma-3-27B) strikes the best trade-off between the two dimensions. The
benchmark is a precondition for using computational replicas in causal inference across domains (e.g., marketing, healthcare), and identifies within-person grounding as the central challenge ahead.
Read the full abstract ▾
Artificial Intelligence
On the Steering Dimensionality of Refusal in Language Models
Han Wang, Erik Miehling, Dennis Wei et al. (+2 more)
Existing activation steering methods often assume that a high-level concept can be mediated by a single steering direction. To support this, two complementary interventions should be achieved: additive steering should induce the target behavior, while directional ablation should suppress it. Yet behaviors may occupy richer activation geometries beyond a single direction, and semantically similar behaviors may be represented by distinct directions. In this work, we study how many directions can reliably control two different types of refusal behaviors: refusal triggered by the safety
alignment and refusal in general contexts. Given the limited expressive capability of a single steering vector, we study the general setting of steering subspaces and introduce the notion of steering dimensionality as the minimum subspace dimensionality required to reliably control a behavior. We characterize sufficient steering subspaces that cover the full extent of the target behavior through both the (monotonic) improvement before the sufficient dimensionality, and the saturation beyond it. Empirically, we find that refusal triggered by safety
alignment is 1-dim steerable, while multiple distinct steering directions can achieve comparable control. In contrast, refusal in general contexts exhibits substantially richer activation geometry where even 5-dim steering subspaces fail to reliably capture its full steerable variation. Our results reveal that the activation geometry underlying refusal is highly context-dependent and can be substantially more complex than a single linear steering direction.
Read the full abstract ▾
All authors Han Wang, Erik Miehling, Dennis Wei, Karthikeyan Natesan Ramamurthy, Huan Zhang
Artificial Intelligence · Cross-listed from cs.RO
Attention-Based Surface Representation Learning for Robot State Prediction and Open-Ended Surface Classification
Oleg Kushnarev and Alexander Belyaev
For ground
robots operating in outdoor environments, understanding the properties of the underlying terrain is essential for ensuring reliable operation. In most perception-based studies, this problem is formulated as categorical classification with a fixed number of classes defined during training. We propose an approach that enables new surface classes to be added as trainable vectors, which can subsequently be used to address higher-level tasks. By employing a learning paradigm based on predicting the
robot's next state in time and using
attention blocks, we improved classification accuracy to 98.56% on the Belyaev-Kushnarev
dataset and 94.8% on BorealTC.
Read the full abstract ▾
Artificial Intelligence · Cross-listed from cs.NE
Adaptive Operator Selection in Bilevel Large Neighborhood Search for Electric Autonomous Dial-a-Ride Problem under Uncertainty
Ishara Hewa Pathiranange and Aneta Neumann
The electric autonomous dial-a-ride problem (EADARP) extends the classical dial-a-ride problem by incorporating battery and charging constraints for electric vehicles. In practice, travel-time uncertainty can cause violations of time-window constraints. Large neighborhood search is effective for solving the EADARP, but its performance can depend on the choice of insertion operator during the repair phase. This paper investigates insertion-operator selection within a bilevel large neighborhood search framework for deterministic and chance-constrained variants of the EADARP. In the chance-constrained variant, arc travel times are modeled as independent normally distributed random variables, and upper time-window constraints are enforced probabilistically. We consider six selection methods, namely fixed greedy insertion, fixed regret-based insertion, random selection, a deterministic state-based rule, performance-adaptive ALNS selection, and LLM-based state-aware selection. Experimental results show comparable performance on smaller instances, while differences become more evident on larger and more constrained instances. There is no single strategy that performs best across all instances, and the relative performance of the LLM-based, rule-based, and ALNS strategies varies with the problem instance and experimental setting.
Read the full abstract ▾
Artificial Intelligence
TCMClinicalReason-Bench: Can Language Models Reason from Pathogenesis to Prescription over Real-World Clinical Cases?
Jirui Dai, Chenkai Zhang, Yan Jia et al. (+4 more)
Large
language models (LLMs) can generate clinical narratives that are insufficiently grounded in patient-specific evidence. In traditional Chinese medicine (TCM), errors can propagate from etiology and pathogenesis through syndrome diagnosis and treatment principles to prescription generation. We developed TCMClinicalReason-Bench using 2,000 multicenter electronic health record cases to distinguish case-grounded responses from fluent but unsupported diagnostic and therapeutic conclusions. Five general-purpose and two TCM-specific LLMs were evaluated in zero-shot settings. An evidence-constrained rubric assessed seven diagnostic and therapeutic components and three cross-block relations, allowing case-supported alternatives. Qwen3.7-Plus with TCM
retrieval served as the automated judge, alongside parallel blinded ratings by five senior TCM clinicians on a 600-case subset. Structural completeness was nearly saturated (99.3-100.0%), but normalized content scores ranged from 40.7% to 54.1%. The five general-purpose models averaged 50.0%, versus 41.3% for the two smaller TCM-specific models. Cross-block logic consistency ranged from 60.8% to 66.8% and correlated moderately with content across cases (Pearson's r = 0.515-0.656). Deficits were greatest in prescription generation, prescription analysis, and symptom-guided modification. In judge stress testing on 100 independent cases, perturbation detection rates across the three relations were 57%, 56%, and 31%, with contradictions detected more reliably than omissions. Separating component quality from cross-block consistency localizes failures missed by endpoint and completeness metrics and identifies where clinician oversight remains necessary.
Read the full abstract ▾
All authors Jirui Dai, Chenkai Zhang, Yan Jia, Yukai Wang, Ruiyang He, Changyong Luo, Zhi Liu
Artificial Intelligence · Cross-listed from cs.LG
FTD-GNO: Memory-Efficient Graph Neural Operators through Functional Tensor Decomposition of the Kernel
Xiaomin Zhang, Boyue Wang, Junbin Gao et al. (+1 more)
Graph Neural Operators (GNOs) provide flexible surrogate models for learning solution operators of partial differential equations (PDEs). However, standard GNOs typically parameterize the integral kernel with a monolithic neural network and evaluate kernel interactions over graph edges, leading to substantial computational and
memory overhead at high resolutions or with large neighborhoods. To address these limitations, we propose Functional Tensor Decomposition Graph Neural Operator (FTD-GNO), a
memory-efficient GNO framework that decouples the high-dimensional continuous integral kernel into low-dimensional mode-wise functions. By instantiating the kernel with classical tensor decomposition formats, including CP, Tensor-Train, and Tucker decompositions, FTD-GNO enables algebraic reconstruction of the integral operator without explicitly materializing full edge-wise kernel tensors. This factorized formulation reduces the
memory footprint of kernel evaluation and aggregation while retaining the continuous operator-learning structure of GNOs. Theoretical complexity analysis shows that FTD-GNO substantially lowers parameter and activation-
memory costs associated with high-dimensional kernel construction. Experiments show lower peak
memory than the corresponding unfactorized graph-integral baselines, with shorter recorded training times.
Fourier-graph experiments further demonstrate that FTD can improve the efficiency of a graph-integral layer within a hybrid operator and has good scalability.
Read the full abstract ▾
All authors Xiaomin Zhang, Boyue Wang, Junbin Gao, Yongli Hu anbd Baocai Yin
Artificial Intelligence · Cross-listed from cs.GR
CurveCodec 2: Skeleton-agnostic animation compression with a learned entropy model
Mingyi Shi, Huancheng Lin, Xuelin Chen et al. (+1 more)
Skeletal motion is stored as every joint's transform at every frame, yet most of it is implied by the body rather than by what the motion is about. Compression is one way to ask what a motion must still say once the body is known, and a production codec must answer it for any skeleton with a stated error bound. Our earlier codec, CurveCodec, matched the mean error of ACL, the production library of modern game engines, with a learned prior over sparse anchors, but not ACL's worst case, and it counted its payload as floats rather than bits. Here we ask where the redundancy of skeletal motion lies and which part of a codec a learned model should take over. Measurements give three answers. At production precision the largest saving comes from predicting each quantized curve from its own past, the second from choosing per joint, in closed loop through the hierarchy, which samples not to code. On the gaps such an encoder leaves, a nearest-neighbour oracle over millions of training samples is no better than linear interpolation, and no learned in-betweener we tried paid for itself. What a network does learn is the distribution of the residuals the codec must send. CurveCodec 2 codes every sub-track as a curve in the log map, quantized in closed loop and thinned to rate-distortion-selected keys, with residuals
entropy-coded under a small learned model whose integer inference is bit-exact across platforms. Two contracts are verified on every decoded clip: ACL's own worst case per joint within a stated tolerance, or ACL's mean error per clip. On a held-out test side of 4,472 clips from 33
datasets, CurveCodec 2 needs 0.37x ACL's bytes at ACL's default precision of 0.01 cm under the worst-case contract and 0.22x at 0.1 cm under the mean contract, decodes on one CPU core, and transfers without retraining to a species absent from training. Project page: https://rubbly.cn/publications/curvecodec/
Read the full abstract ▾
All authors Mingyi Shi, Huancheng Lin, Xuelin Chen, Taku Komura
Artificial Intelligence
Fine-Tuning VLM for Enhancing AI's Spatial Intelligence: Understanding 3D and 2D Rotations
Uttamasha Monjoree and Wei Yan
Spatial intelligence is a fundamental skill in multiple domains, such as Science, Technology, Engineering, and Mathematics (STEM), Medicine, Architecture, and Construction. Recent studies indicate that
Vision-
Language Models (VLMs) still face limitations in spatial
reasoning, which inhibits artificial intelligence (AI) from performing practical spatial tasks. Using multiple object-rotation
datasets developed for training and evaluation, our experiments demonstrated promising improvements in both 2D and 3D rotation detection. Fine-tuned Google DeepMind-built Gemma-4 mixture-of-experts (MoE) models significantly outperformed fine-tuned Gemma-4 generalist models in predicting rotations defined by both their axes and angles. Fine-tuning also substantially improved angle estimation for 2D representation without requiring an explicit coordinate system. Furthermore, identifiable objects did not improve angle-detection accuracy; instead, objects with prominent linear features showed improved performance.
Read the full abstract ▾
Artificial Intelligence · Cross-listed from cs.LG
Clean: Second-order LLM Training at Linear Memory Cost via Nyström Sketching
Beheshteh T. Rakhshan, Sahar Rajabi, Maziar Sargordi et al. (+3 more)
Training large
language models (LLMs) entails a fundamental trade-off:
memory-efficient optimizers such as Adam discard cross-parameter
curvature, whereas full-
curvature methods such as SOAP can accelerate convergence at prohibitive
memory costs. We introduce Clean, a
memory-efficient and full-
curvature optimizer designed to resolve this bottleneck. Clean leverages the randomized Nystrom method to accurately approximate the left and right preconditioners in SOAP, and to reduce the optimizer's
memory complexity from quadratic to linear in terms of model dimensions. We subsequently reintegrate the off-subspace components to capture
curvature information beyond the low-rank approximation, preserving rich
curvature at minimal
memory cost. We further propose Q-Clean, a low-precision variant that aggressively compresses optimizer states. Q-Clean reduces optimizer
memory consumption by over 50% compared to
Muon when pre-training a LLaMA-1.3B architecture, all while maintaining strong and competitive predictive performance. Notably, Clean operates with a smaller optimizer-state footprint than standard AdamW while reaching AdamW's final performance 26% faster in wall-clock time. Furthermore, our methods uniquely enable the pre-training of a 13B-parameter model on a single 80GB GPU, providing a scalable, efficient, and accessible approach to large-scale model optimization.
Read the full abstract ▾
All authors Beheshteh T. Rakhshan, Sahar Rajabi, Maziar Sargordi, Shikai Fang, Guillaume Rabusseau, Sirisha Rambhatla
Artificial Intelligence
Liancheng Fang, Zhuowei Li, Youngeun Kim et al. (+10 more)
Diffusion language models (DLMs) enable fast generation by predicting multiple
tokens in parallel, but their practical adoption remains limited by a persistent quality gap relative to comparably sized autoregressive (AR) models. We attribute this gap to a computation-difficulty mismatch: within a partially observed sequence, some unknown
tokens are easy to predict, while others require substantially more computation. Existing DLMs nevertheless apply uniform computational depth to all unknown positions at each denoising step. We introduce ALoDLM, which replaces uniform computation with
token-adaptive latent recurrence. At each denoising step, ALoDLM iteratively refines latent representations and allocates computation according to
token difficulty.
Tokens ready to commit are fed back as discrete context, while unresolved
tokens retain and further refine their latent states through additional recurrent passes. To learn
token prediction and computation allocation jointly, we formulate
token-wise computation schedules as latent variables and derive a conditional negative evidence lower bound (NELBO). We train ALoDLM at 1.7B and 8B parameter scales. Across eleven
benchmarks, ALoDLM outperforms all evaluated DLMs and the corresponding AR baselines in average
benchmark score at both scales. ALoDLM also retains fast parallel decoding, yielding a strong quality-efficiency trade-off among evaluated autoregressive and
diffusion models under optimized inference engines.
Read the full abstract ▾
All authors Liancheng Fang, Zhuowei Li, Youngeun Kim, Tianchen Zhao, Rajat Koner, Jiaye Wu, Linghan Xu, Xuanbai Chen, Xiang Xu, Zheng Zhang, Jakub Zablocki, Nishant Sankaran, Yifan Xing
Artificial Intelligence
Asynchrony Is Nearly Free for Evolution Strategies on Long-Horizon LLM Agentic Tasks
William Hoy, Jingxuan Fan, Nurcin Celik et al. (+1 more)
Synchronous policy updates can limit the efficiency of post-training for long-horizon LLM
agents. When trajectories span multiple interaction turns, completion times can vary substantially, and synchronous update barriers leave faster workers idling, wasting compute resources. Asynchronous reinforcement learning addresses this inefficiency but introduces policy lag and off-policy optimization.
Evolution strategies (ES) are a backpropagation-free alternative to gradient-based training, yet require many perturbed-policy evaluations and have remained largely synchronous in current LLM post-training. We introduce bounded-staleness asynchronous ES and investigate its training behavior under stale fitness measurements. On WebShop, we systematically vary the lag and fraction of stale fitness measurements and additionally introduce two stress tests to probe extreme staleness cases. Natural asynchrony closely tracks synchronous ES, and degradation in evaluation accuracy remains small even under extreme staleness, including when all fitness measurements are stale. These findings suggest that ES can tolerate substantial policy staleness without explicit off-policy correction, supporting its use in asynchronous post-training for long-horizon LLM
agents.
Read the full abstract ▾
All authors William Hoy, Jingxuan Fan, Nurcin Celik, Xu Pan
Artificial Intelligence
MemLeak: Cross-User Semantic Leakage in Multi-Tenant AI Agent Memory
Priyanka Mudgal, Kai Zhao, Guilin Zhang et al. (+4 more)
Personal AI
agents in enterprise multi-tenant deployments share a common vector store for long-term
memory. Shared
embedding spaces create a surface for cross-user
memory leakage: a user's query can retrieve semantically adjacent
memories belonging to another user through ordinary cosine-similarity
retrieval, without any exploit. We formalize this as cross-user admissibility failure and evaluate it across six experiments, plus follow-up ablations, under both sparse (TF-IDF) and production-faithful (MiniLM-L6-v2)
retrieval. Non-
adversarial, incidental leakage reaches 70–100% under pooled {same-team}
retrieval; adversarially crafted
memories achieve 90–100% top-$k$ placement, exceeding weaker keyword-based attacker baselines, with score lifts of $+0.416$ to $+0.511$ under production-faithful dense
retrieval (Config B); and end-to-end response contamination reaches 5.00/5 under a production
retrieval path and 4.67/5 with Claude Sonnet~4.5, with contaminated responses often scoring as helpful or more helpful than clean ones, a gap validated against human judgment. Among three architectural mitigations, only hard post-
retrieval ownership gating consistently restores the clean baseline (1.00/5) across {two generation models, at a measured latency overhead of roughly 1.4~ms per query.
Read the full abstract ▾
All authors Priyanka Mudgal, Kai Zhao, Guilin Zhang, Andy Olsen, Ezekiel Miller, Xu Chu, Aletta Johanna Blanken
Artificial Intelligence
Agentic AI with Structured CoT for Enhancing AI's Spatial Intelligence: Visualization and Reasoning of Rotation
Uttamasha Monjoree and Wei Yan
Recent studies show that artificial intelligence (AI) with
language and
vision capabilities still experiences limitations in spatial
reasoning. In this paper, we have studied the spatial capabilities of advanced generative AI to understand the rotations of objects in 3D space, utilizing AI's image processing and
language processing features. We trained and examined the spatial intelligence of a generative Agentic AI model (GPT-5.6) to understand the spatial rotation process with rotation diagrams based on the revised Purdue Spatial Visualization Test: Visualization of Rotations (Revised PSVT:R). We improvised the Revised PSVT:R by superimposing additional graphical and contextual features to evaluate how different Chain-of-Thought (CoT)
reasoning strategies influence model performance. The results indicate that structured CoT
reasoning improves the spatial
reasoning performance of the base GPT-5.6 model in both
datasets (PSVT:R and PSVT:R with coordinate system). We used three CoT approaches - (1) Structured CoT, (2) few-shot Structured CoT, and Structured CoT with Self-optimized
Prompt. The three CoT approaches evaluated in this study showed no significant performance difference. Results showed that combining structured CoT
reasoning with relevant contextual information leads to considerable improvements in VLM performance on 3D rotation tasks, demonstrating the potential of agentic AI for more effective spatial
reasoning. However, when contextual information is removed, structured CoT
reasoning alone provides limited improvement, and the models continue to exhibit notable difficulties in understanding spatial transformations. These findings suggest that effective spatial
reasoning in VLMs relies on the integration of visual, textual, and
reasoning-based information in future agentic AI systems for spatial intelligence.
Read the full abstract ▾
Artificial Intelligence
EvalResearchBench: Can AI Agents Build Their Own Feedback for Recursive Self-Improvement?
Yaolun Zhang, Tianyi Xu, Yujie Zhao et al. (+3 more)
Recursive self-improvement (RSI) relies on evaluation feedback to assess progress and guide further research, yet repeatedly running complex
benchmarks is costly and slows iteration. Human experts reduce this cost by selecting
benchmark subsets or designing compact suites. We ask whether AI
agents can automate this design process and introduce EvalResearchBench (ERB), a
benchmark for autonomous evaluation research. Given target materials, development references, candidate APIs, and fixed time and API budgets, an
agent called the researcher selects or synthesizes tasks, implements graders, and revises them in pilot tests before freezing an executable evaluator for coding, co-work, and
reasoning. We study 9 researchers and 13 candidate models and compare each frozen evaluator with 14 target
benchmarks on score concordance and pairwise agreement. The best evaluators order about 75% of candidate pairs as the targets do, below the 91% ceiling set by disagreements among the targets. No researcher leads on every metric, and the best evaluator on development targets is not the best on sealed targets hidden from the researcher. A human-designed sample of public tasks remains a strong baseline, and the evaluator with the lowest recorded execution cost attains the highest pairwise agreement.
Agents repair tasks and graders through pilot feedback, yet their evaluators can still truncate answers, exhaust the evaluation budget, or let a few questions dominate a domain score.
Read the full abstract ▾
All authors Yaolun Zhang, Tianyi Xu, Yujie Zhao, Jishen Zhao, Qingyun Wu, Huazheng Wang
Artificial Intelligence
Language Model Activations Inhabit Privileged Error-Correcting Basins
Matthew Finlayson, Francisco Pernice, Eric Todd et al. (+16 more)
Language models exhibit remarkable robustness, continuing to produce coherent text even when their activations are perturbed by interventions like linear steering. We hypothesize that this robustness is a result of passive dynamics, i.e., constraining mechanisms in the forward pass that funnel activations toward "good" regions that produce coherent outputs. To investigate these hypothesized error-correcting mechanisms, we probe the geometry of
language model activation space by observing the action of model layers on low-dimensional curves. In doing so, we discover that model activations occur within a cluster of distinct attracting basins, which differentiate natural activations geometrically from distributionally similar synthetic activations. Applying this lens to
language model steering, we observe feature-specific basins along semantic steering directions, and find that steering moves activations between these basins. To demonstrate the active role of this geometry in neural computation, we show that adaptively modulating steering strength to transport activations across basins improves inter-
language steering, significantly increasing the probability of sampling
tokens from the target
language compared to fixed-strength steering. Our findings establish analysis of activation space geometry as a promising approach to interpreting and controlling
language models.
Read the full abstract ▾
All authors Matthew Finlayson, Francisco Pernice, Eric Todd, Amir Zur, Daniel Wurgaft, Fenil R. Doshi, Vasudev Shyam, Matt Feiszli, Satchel Grant, Lucius Bushnaq, Tal Haklay, Usha Bhalla, Matthew Kowal, Thomas Fel, Jack Merullo, Atticus Geiger, Xiang Ren, Owen Lewis, Ekdeep Singh Lubana
Artificial Intelligence
SHarP: Saliency-based Pruning of Agent Harnesses
Xinyi Gao, Qiucheng Wu, Kaizhi Qian et al. (+3 more)
Agent harnesses are systems that coordinate model calls, tool use, and task execution to help large
language models complete complex tasks. To meet task requirements and address failures, these systems are often iteratively refined by amending and patching their instructions, tools, and workflows, continuously increasing harness complexity. It is therefore unclear whether some resulting harness modules are redundant, introducing substantial
token overhead with little, if any, performance gain. Inspired by neural network
pruning, in this paper, we study harness
pruning as a means of striking a better balance between task performance and
token cost. We propose SHarP (Saliency-based Harness
Pruning), a simple yet effective
pruning strategy based on the saliency of each harness module with respect to performance and efficiency. Specifically, we first identify tools, instructions, and supporting mechanisms as components that can be individually disabled. We then estimate the saliency of each module by ablating it and assessing its task performance and
token cost relative to the full set of single-module ablations. Modules with the smallest contribution to performance or largest computational overhead are subsequently pruned. Our evaluation across various harnesses on held-out validation sets reveals a surprising finding: most harnesses that we studied are highly redundant and can maintain comparable performance and efficiency even after a substantial portion of their modules are pruned. Our
pruning approach and empirical findings provide new perspectives on
agent harness design and optimization.
Read the full abstract ▾
All authors Xinyi Gao, Qiucheng Wu, Kaizhi Qian, Handong Zhao, Shiyu Chang, Yang Zhang
Artificial Intelligence · Cross-listed from stat.ML
Risk-Calibrated Proposal Transport for Finite-Particle Diffusion Steering
Ziseok Lee, Jaehyeon Kim, Seungwon Kim et al. (+6 more)
Inference-time steering combines pretrained
diffusion experts or rewards without retraining by changing the dynamics that transport noise to data. Feynman-Kac correction compensates for proposal mismatch through importance-weighted sequential Monte Carlo (SMC), whose finite-particle behavior depends on the proposal. Variance-controlling guidance (VCG) improves that proposal by fitting a linear drift correction to minimize empirical log-weight-rate variance. Although its population optimum cannot worsen residual variance, finite-particle VCG can nearly eliminate its fitting residual while increasing residual risk on new states by orders of magnitude. The resulting update can degrade unweighted generation or accelerate particle collapse. We show that the centered Feynman-Kac rate is the normalized transport residual and that expected out-of-fit benefit is exactly population headroom minus coefficient-estimation penalty. Under regularity assumptions, a Wasserstein analysis bounds the unweighted proposal's terminal error using this residual. These results motivate Risk-Calibrated Proposal Transport (RCPT), which uses deletion leave-one-out residuals to calibrate the retained fraction of the VCG update, adding no model calls and only small linear-
algebra overhead. Experiments on 2D checker distributions, scaffold decoration, molecular property optimization, and class-conditional CIFAR-10 generation demonstrate recovery from harmful fitted updates. Across molecular and image domains, RCPT mitigates harmful fitted updates and improves a broad range of terminal metrics relative to uncalibrated VCG.
Read the full abstract ▾
All authors Ziseok Lee, Jaehyeon Kim, Seungwon Kim, Seunghyun Moon, Haneul Choi, Wooyeol Lee, Donghyun Koh, Minhyeong Lee, Kyungsu Kim
Artificial Intelligence · Cross-listed from cs.CR
Language Model Fingerprinting Requires Rethinking Watermark Teachers
Jeongyeon Hwang, Anshul Nasery, Sewoong Oh et al. (+1 more)
LLM fingerprinting via watermark
distillation embeds a statistical watermark signal into model weights, enabling model owners to identify their models behind black-box APIs. Revisiting a recent protocol, we find that its utility evaluation understates text quality degradation in open-ended generation, favoring overly strong watermark teachers. Weakening the watermark improves text quality but sacrifices detectability. To move beyond this trade-off, we rethink whether text watermarks designed for verifying generated text are suitable
distillation teachers for model fingerprinting. Such watermarks are typically designed to remain detectable from an individual output, limiting how sparse the watermark signal can be. In contrast, fingerprint verification can aggregate signal across queries, making sparser watermark signals viable. This raises a key question: where should the sparse signal be placed? We analyze signal placement through
token surprisal and show that, even at comparable watermark strength, different placements can target
tokens with different plausibility under the base model. This motivates near-tie restriction, which uses top-1-relative logit gaps to restrict the watermark bias to
tokens close to the base model's top prediction. Across multiple models, near-tie improves detection–quality frontiers under deployment changes, preserves higher text quality across query budgets, and further improves existing watermarking schemes when combined with them.
Read the full abstract ▾
All authors Jeongyeon Hwang, Anshul Nasery, Sewoong Oh, Jungseul Ok
Artificial Intelligence
Agentic Cognitive Depth: Operational Criteria for Evaluating LLM Agents
Nijesh Upreti, Chris Sypherd and Vaishak Belle
Agentic large
language model (LLM) systems are commonly implemented as an LLM in a loop with Planning,
Memory, Tools, and Control Flow. This application-focused view connects agentic LLM research with deployable systems and leaves open how such systems should be evaluated beyond end-to-end task success. Building on this view, we define agentic cognitive depth as a trajectory-level profile across five operational criteria. The profile contains context sensitivity ($C$), temporal continuity ($T$),
multimodal coordination ($M$), adaptive interaction ($A$), and metacognitive monitoring ($Mc$). The first four criteria measure how well Control Flow,
Memory, Tools, and Planning are used across a trajectory. The fifth measures whether the system monitors and regulates the full run. For each criterion, we give operational proxies and a perturbation procedure, then connect the profile to the
agent's world model. We provide the structure needed to extend
benchmarks such as GAIA, SWE-bench, WebArena, and TRIP-Bench with per-criterion diagnostics. Symbolic verifiers, structured
memory, planner coupling, and tool constraints provide practical ways to build and test these capacities.
Read the full abstract ▾
Artificial Intelligence · Cross-listed from cs.LG
How RL Reshapes LLM Reasoning: Transferability, Coverage, and Scaling Laws
Ziheng Cheng, Yixiao Huang, Hanlin Zhu et al. (+1 more)
Recent studies on reinforcement learning (RL) report seemingly conflicting evidence about large
language model (LLM)
reasoning. Training on mathematics can improve performance in other domains, yet gains in Pass@1 can coincide with lower Pass@$N$ than the base model. This raises a fundamental question: does RL expand an LLM's
reasoning boundary, or merely reweight its existing
reasoning space? We revisit these phenomena across Qwen and Gemma model families, showing both cross-domain gains and forgetting, while coverage at large sampling budgets increases on some tasks and decreases on others. Detailed analysis of solution traces before and after RL indicates a shift in the
reasoning strategies the model employs, motivating a two-stage autoregressive policy model that separates strategy selection from problem-specific execution. Within this framework, we prove how RL's implicit bias reshapes strategy preferences, allowing gains on some tasks while suppressing strategies required by others. This mechanism can also broaden or narrow coverage at a given sampling budget even without expanding strategy support. We further provide theoretical justifications for log-sigmoid and log-linear scaling laws in RL compute, and evaluate their predictive power. Together, these results connect changes in strategy selection to cross-domain transfer,
reasoning coverage, and compute scaling.
Read the full abstract ▾
All authors Ziheng Cheng, Yixiao Huang, Hanlin Zhu, Somayeh Sojoudi