Artificial Intelligence · Cross-listed from cs.CL
Two-Stage Prompt Optimization for Few-Shot Relation Extraction: From Reasoning-Guided Search to Gradient-Guided Refinement
Aunabil Chakma, Mihai Surdeanu and Eduardo Blanco
Automatic
prompt optimization is still underexplored for episodic few-shot relation extraction with smaller
language models. We propose a two-stage framework that combines
reasoning-based
prompt optimization with gradient-based
prompt optimization. The first stage can use any
reasoning-based optimizer to make broadprompt improvements in natural
language. The second stage applies our GradPO, which uses loss and gradient signals to identify high-impact
prompt spans and refine them with local edits. Experiments on FS-TACRED and FS-FewRel show that local refinement usually improves
prompts found by the first stage, and GradPO is the most consistent refiner. Our framework achieves state-of-the-art performance on FS-TACRED with Qwen3-4B and remains competitive on FS-FewRel.
Read the full abstract ▾
Artificial Intelligence
SFBench: The SciFy Scientific Feasibility Benchmark
Cash Costello, James Mayfield, Elsbeth Turcan et al. (+7 more)
We present SFBench, a
benchmark dataset for evaluating systems that assess the feasibility of scientific claims. SFBench includes 197 claims in materials science, each annotated with a ground-truth feasibility score on a five-point scale along with an explanation of that assessment. The collection differs from previous collections in several important ways: 1) it defines a complex task that requires
reasoning over claims of varying scientific feasibility; 2) its claims are not extracted from existing scientific publications but are created de novo, greatly reducing the chances that LLMs have trained on them; 3) claims and ground truth are established by subject matter experts, not by artificial intelligence; and 4) unlike many
benchmarks that ask about question/answer pairs, provide multiple choice answers, or ask questions requiring short, fixed answers, SFBench explanations are completely open-ended. We describe the
benchmark design, data creation process, and evaluation metrics, and we report baseline results using recent GPT models.
Read the full abstract ▾
All authors Cash Costello, James Mayfield, Elsbeth Turcan, Christine Piatko, Christina K. Pikas, Justin Rokisky, Sam Scheck, Chris Ribaudo, Ritwik Bose, Alex Memory
Artificial Intelligence
SCARCE: Scalable Cascade Analysis for Rare-event Characterisation via Embeddings
Yingjie Wang, Yi Dong, Edmund Lau et al. (+3 more)
Rare events govern the safety profile of modern AI systems, yet their probabilities are extremely difficult to estimate: direct Monte Carlo requires prohibitive sample budgets. Subset Simulation (SS) addresses this by decomposing a rare-event probability into moderate conditional probabilities over nested intermediate events. However, classical SS requires a handcrafted scalar performance function whose sublevel sets define those events, demanding detailed knowledge of the failure geometry and limiting transfer to new domains. We propose SCARCE (Scalable Cascade Analysis for Rare-event Characterisation via
Embeddings), which replaces the performance function with learned latent representations and
geometric rulers that score proximity to failure regions. Adaptive thresholding constructs nested intermediate events directly from data. We formalise SCARCE through a non-negative supermartingale, yielding a high-probability upper envelope that remains valid under early stopping. On MNIST misclassification, where dense Monte Carlo provides ground truth, SCARCE achieves approximately 400–500 times lower mean absolute error than grid-searched traditional SS while eliminating systematic over-counting. We then study PAIR-style LLM jailbreaks under a fleet-level threat model with
adversarial fraction $\eta$. On Llama-Guard-3-8B hidden states, a PCA-based ruler attains 2.6% mean relative error for $\eta \geq 10^{-3}$ against finite-sample references whose average bootstrap relative half-width is 27.9%, and transfers to a GCG-style corpus with 2.93% relative error after recalibration. A directional criterion $\mathrm{KL}(p_{\mathrm{good}}\,\|\,p_{\mathrm{bad}})$ ranks rulers consistently with estimation error (Spearman $\rho=0.83$).
Read the full abstract ▾
All authors Yingjie Wang, Yi Dong, Edmund Lau, Jie Meng, Taylor T Johnson, Xiaowei Huang
Artificial Intelligence · Cross-listed from cs.CL
Do We Still Need Fine Tuning? Turkish Sentiment Analysis in the Era of Large Language Model
Sercan Karakaş and Yusuf Şimşek
This study examines whether supervised fine-tuning remains necessary for Turkish sentiment analysis in the era of large
language models. We compare classical machine learning methods, fine-tuned pretrained
language models, and prompted large
language models on a Turkish e-commerce review
dataset with negative, neutral, and positive labels. Fine-tuned BERTurk models perform best overall and outperform all prompted large
language models in the full three-class task. The neutral class emerges as the main difficulty: while several large
language models are much more competitive in binary positive–negative classification, they degrade substantially in the three-class setting by collapsing neutral reviews into polarized categories. The findings suggest that, in realistic Turkish sentiment classification, prompted large
language models do not yet match supervised fine-tuning in the zero-shot setting, and that including the neutral class is crucial for robust evaluation.
Read the full abstract ▾
Artificial Intelligence · Cross-listed from cs.LG
Does Role Specialization Matter for Explanation Faithfulness in Mixture-of-Experts?
Yeji Kim, Housam Babiker, Mi-Young Kim et al. (+1 more)
Mixture-of-Experts (MoE) architectures have recently been extended with role-based mechanisms for interpretability. This is typically done by assigning semantic roles to individual expert components, for example roles like synergy, redundancy, and uniqueness in
multimodal settings. However, whether such structural role decomposition preserves explanation faithfulness of the overall architecture remains largely underexplored. We hypothesize that inter-expert representation overlap weakens effective role separation and degrades attribution-based faithfulness, even when semantic roles are explicitly defined. To address this limitation, we introduce representation-level decorrelation regularization to explicitly reduce inter-expert similarity in latent space. Using representation decorrelation objectives, we encourage clearer specialization among experts by minimizing representation overlap. Our experiments show that across multiple
multimodal benchmarks, this separation consistently improves explanation faithfulness, as measured by comprehensiveness, sufficiency, and their Area Over the Perturbation Curve (AOPC) summaries, while preserving task performance. We further show that these improvements are not limited to role-based architectures such as Interpretable
Multimodal Interaction-aware MoE (I2MoE). Similar trends are observed in a standard sparse MoE baseline, suggesting that representation-level separation may provide a more general mechanism for enhancing explanation faithfulness in MoE systems. Overall, our findings suggest that structural role decomposition alone may be insufficient to guarantee faithful explanations and that representation-level separation helps improve explanation faithfulness. To support reproducibility, the source code and supplementary material are publicly available at https://github.com/dut0817/FL-I2MoE_Decor.
Read the full abstract ▾
All authors Yeji Kim, Housam Babiker, Mi-Young Kim, Randy Goebel
Artificial Intelligence · Cross-listed from cs.LG
Mechanistically Eliciting Latent Behaviors in Language Models
Andrew Mack, Nina Panickssery and Alexander Matt Turner
We aim to discover diverse, generalizable perturbations of LLM internals that can surface hidden behavioral modes. Such perturbations could help reshape model behavior and systematically evaluate potential risks. We introduce Causal Perturbative Elicitation (CPE), an unsupervised method for discovering interpretable low-rank adapters (LoRAs) that can elicit these latent behaviors. CPE decomposes the computations of a deep
transformer slice using a heuristic tensor-decomposition-based
algorithm. CPE exhibits remarkable data efficiency, learning a large number of interpretable LoRAs from a single example. Even though CPE is unsupervised, we find that in some cases it can be competitive with supervised elicitation methods via brute-force enumerative search over weight space. For instance, CPE performs similarly to matched-wall-clock-time GRPO on the Countdown task for Qwen3-8B (85% vs 87%), demonstrating that CPE can efficiently elicit complex multi-
token behaviors. Since CPE is unsupervised, it can also surface hidden failure modes, such as sandbagging, restoring 85% of locked BigCodeBench performance on a password-locked version of Llama3-70B introduced by Taylor et al. (2025). Additionally, since CPE explores behaviors in weight-space rather than
token-space it can potentially ameliorate exploration hacking, a misalignment failure which may arise in sufficiently self-aware AI models (Ngo, 2022). In fact, we find that CPE virtually eliminates
alignment-faking (Greenblatt et al., 2024) behavior in a Llama3-70B-based model organism developed by Hughes et al. (2025). Finally, we find that CPE can be used to initialize GPT-OSS-20B in an aligned basin when running GRPO on an environment prone to reward-hacking. By providing a data-efficient method to systematically explore the space of latent model behaviors, CPE yields a powerful tool for aligning AI systems and evaluating their safety.
Read the full abstract ▾
Artificial Intelligence · Cross-listed from cs.PL
Langshaw: Declarative Interaction Protocols Based on Sayso and Conflict
Munindar P. Singh, Samuel H. Christie V and Amit K. Chopra
Current
languages for specifying multiagent protocols either over-constrain protocol enactments or complicate capturing their meanings. We propose Langshaw, a declarative protocol
language based on (1) sayso, a new construct that captures who has priority over setting each attribute, and (2) nono and nogo, two constructs to capture conflicts between actions. Langshaw combines flexibility with an information model to express meaning. We give a formal semantics for Langshaw, procedures for determining the safety and liveness of a protocol, and a method to generate a message-oriented protocol (
embedding needed coordination) suitable for flexible asynchronous enactment.
Read the full abstract ▾
Artificial Intelligence · Cross-listed from cs.CV
One Scene, Two Depths: Probing Geometric Ambiguity in Monocular Foundation Models
Xiaohao Xu, Feng Xue, Xiang Li et al. (+5 more)
A faithful 3D world representation should account for layered geometry, where a single camera ray may contain multiple visible and geometrically valid surfaces.
Monocular depth estimation, however, reduces this structure to one scalar depth per pixel. Transparent scenes make this ambiguity measurable: the same ray can pass through foreground glass and observe the background, turning the supervised target into a convention of annotation, data, and training rather than a scene-intrinsic truth. A learned predictor exposes this convention as its depth-layer preference. We introduce MultiDepth-3k (MD-3k), a sparse two-layer ordinal
benchmark for measuring depth-layer preference and multi-layer spatial relationship accuracy (ML-SRA). On MD-3k, leading depth foundation models exhibit diverse layer preferences under standard RGB input, showing that the same layered geometry can be resolved differently across models. We further find that
Laplacian Visual Prompting (LVP), a training-free spectral input transformation, can substantially change the reported layer for certain frozen models. The strongest RGB/LVP pair, DAv2-L, reaches 75.5% ML-SRA. These results suggest that depth foundation models may express complementary
geometric hypotheses that standard RGB inference leaves unexpressed. We invite the community to rethink depth supervision and evaluation through an ambiguity-aware lens, where multiple valid 3D interpretations are treated as
geometric structure to be measured, preserved, and expressed.
Read the full abstract ▾
All authors Xiaohao Xu, Feng Xue, Xiang Li, Haowei Li, Shusheng Yang, Tianyi Zhang, Matthew Johnson-Roberson, Xiaonan Huang
Artificial Intelligence · Cross-listed from cs.LG
How AI settled the complexity of the oldest SGD algorithm
Michał Dereziński and Xiaoyu Dong
In 1937, Stefan Kaczmarz proposed a simple
algorithm for solving systems of linear equations. This
algorithm turned out to be the earliest known example of
stochastic gradient descent, a ubiquitous computing paradigm that drives the training of modern AI models such as ChatGPT and Gemini. Now, those AI models have joined forces to discover the worst-case complexity of the Kaczmarz
algorithm. This paper tells the story of how it happened.
Read the full abstract ▾
Artificial Intelligence · Cross-listed from cs.CV
SonoCLIP: Mask-Guided Region-Aware Vision-Language Pretraining for Fetal Ultrasound Analysis
Hang Su, Chao Sun, Zhaofan Li et al. (+3 more)
Vision-
language foundation models have shown strong potential in medical image analysis. Although foundation models for ultrasound imaging have recently emerged, the domain remains particularly challenging due to severe speckle noise, acquisition variability, and subtle anatomical boundaries, leading to high inter-observer variability. Existing CLIP-based models rely primarily on global image-text
alignment, limiting their sensitivity to clinically decisive local structures. We propose SonoCLIP, the first million-scale region-controllable fetal ultrasound
vision-
language foundation model that integrates
segmentation masks as mask-channel visual
prompts within the
vision encoder, enabling joint global-local contrastive representation learning. To support scalable region-text
alignment, we introduce a sigmoid-based pairwise contrastive loss that improves stability under large-scale supervision. We further curate a 1.44M-image
multimodal fetal ultrasound
dataset spanning 24 standard planes for large-scale pretraining. Extensive cross-center evaluations demonstrate that SonoCLIP achieves superior zero-shot transfer performance under both global and mask-guided inference, establishing a controllable and clinically oriented foundation model for fetal ultrasound analysis. Our code and data are available at https://github.com/Harrison-one/SonoCLIP.
Read the full abstract ▾
All authors Hang Su, Chao Sun, Zhaofan Li, Wei Hu, Juhua Liu, Bo Du
Artificial Intelligence · Cross-listed from cs.LG
Bilevel Optimization for Neural Architecture Search
Abhishek Shukla, Ankur Sinha and Faiz Hamid
Bilevel optimization has become an influential and widely adopted framework for addressing hierarchical optimization problems in machine learning, providing an effective approach to modeling the interaction between two levels of optimization, with applications such as hyperparameter tuning, meta-learning,
adversarial training, and data poisoning. Neural Architecture Search (NAS), a subfield of hyperparameter optimization, is a
prime example of a bilevel optimization problem, with architecture parameters optimized at the outer-level and network weights optimized at the inner level. This paper presents a structured overview of NAS through the lens of bilevel optimization. We categorize existing NAS approaches into two main classes: sampling-based methods, which search optimal architectures using different architecture samplers, and bilevel theory-based methods, which solve the architecture search problem using bilevel optimization principles. We further highlight our current research direction, wherein the bilevel NAS formulation is addressed through an auxiliary mathematical programming framework. This framework enables the systematic integration of second-order information from the model's training loss function and ensures the optimality of the model parameters while modifying architecture parameters. By simultaneously updating the architecture and model parameters along their respective optimal descent directions derived from the auxiliary mathematical program, these methods achieve more principled and theoretically consistent results. The same auxiliary program can also be used for simultaneous hyperparameter and model fine-tuning. A comparative analysis shows that bilevel theory-based approaches generally outperform sampling-based methods, both in accuracy and efficiency.
Read the full abstract ▾
Artificial Intelligence · Cross-listed from cs.LG
The Joint Effect of Quantization and Sampling Temperature on LLM Safety Alignment: A Factorial Analysis
Hari Prasad and Ritam Pal
Modern LLM deployments often combine quantization with higher sampling temperatures to reduce cost, latency, or repetition, yet safety evaluations usually treat these as fixed implementation details. We test whether models that are safe at FP16 with greedy decoding remain safe after quantization and
stochastic sampling, or whether the two factors amplify each other. We evaluate 8 instruction-tuned models from five families across 3 precisions and 6 temperatures, covering 144 configurations on 7 harmfulness
benchmarks and generating about 2.0 million responses, which are scored by a six-judge safety ensemble. Contrary to concerns that low-bit deployment erodes
alignment, we find that standard quantization is approximately safety-neutral: for 7 of 8 models, AWQ INT4 keeps attack success within about 1.6 percentage points of FP16 or lowers it, with clear degradation only for SmolLM3-3B (34.5% to 44.1%). However, the larger risk comes from sampling: higher temperatures sharply increase decision instability, with DFR reaching 41.9% at T = 1.0, even when average ASR changes only modestly. The two factors do not compound: our Compound Degradation Index remains sub-additive (-0.071 to +0.018), indicating that quantization partially offsets rather than amplifies temperature-induced degradation. Finally, a per-
benchmark breakdown shows that single-
benchmark evaluation badly understates risk: several models scoring 0% on AdvBench exceed 80% on ManyHarm. Standard INT4/INT8 quantization can therefore be reasonable for well-aligned models, but safety claims should report multi-sample stability across multiple
benchmarks rather than rely on a single
benchmark at greedy decoding.
Read the full abstract ▾
Artificial Intelligence · Cross-listed from cs.CV
ScAle: Attention Head Scaling as a Minimal Adapter for Spatial Reasoning in Vision Language Models
Rahul Chowdhury, Timothy A Rupprecht, Xuan Shen et al. (+2 more)
Spatial
reasoning remains a persistent challenge for many
vision language models (VLMs), and improving it typically requires fine-tuning with substantial additional parameters. Our preliminary analysis reveals that rescaling activations in selected
transformer layers-without modifying pretrained weights-can significantly influence downstream performance. Motivated by this observation, we propose ScAle, an ultra-lightweight adaptation method that learns a small set of scalar coefficients to modulate last-
token attention and MLP activations in a fully frozen backbone. We evaluate our method on the synthetic spatial
reasoning benchmark SpatialEval and on real-world VQA
datasets (COCOQA and VGQA) across multiple model families. Our method, ScAle, achieves up to 134.1% relative accuracy gains using only 1K trainable parameters without requiring millions of trainable parameters as in standard PEFT methods such as LoRA. Despite its extreme compactness, our approach recovers a substantial fraction of standard PEFT performance while preserving strong non-spatial VQA accuracy. These results demonstrate that bounded activation reweighting provides a simple, architecture-agnostic, and highly parameter-efficient alternative for adapting pretrained VLMs.
Read the full abstract ▾
All authors Rahul Chowdhury, Timothy A Rupprecht, Xuan Shen, Pu Zhao, Yanzhi Wang
Artificial Intelligence · Cross-listed from cs.CV
ReMAP-PET: Beyond Visual Understanding – Learning Region-Guided Metabolic Alignment Semantics from Brain PET
Dasen Dai, Yanteng Zhang, Shuoqi Li et al. (+6 more)
Positron Emission
Tomography (PET) reveals
brain metabolism and is clinically central to neurodegenerative disease assessment, yet existing 3D
brain foundation models treat PET as generic volumetric data, missing the structured regional metabolic information that distinguishes it from structural neuroimaging. To address these limitations, we propose ReMAP-PET, a framework that moves beyond visual encoding by supervising a partially-tuned MedicalNet 3D ResNet-50 with
brain regional standardized uptake value ratio (SUVR) profiles through joint regression and contrastive objectives, enabling the encoder to learn the metabolic semantics underlying PET modality. On 1015 paired PET–SUVR samples, ReMAP-PET achieves 0.070 SUVR MAE and 77.8% PET SUVR Recall@1, substantially outperforming five frozen pretrained baselines. We further connect the metabolic
embedding to clinical
language via contrastive
alignment with frozen BioClinicalBERT and demonstrate end-to-end PET-to-report generation through SUVR-constrained verbalization. Linear probing on diagnostic classification and cognitive regression tasks confirms that the
embeddings retain clinically relevant information without task-specific fine-tuning. Our results show that grounding PET encoders in regional metabolic semantics – rather than treating PET as generic volumetric data – yields representations that are structured, interpretable, and
language-compatible, pointing to a new direction for metabolic-aware PET understanding.
Read the full abstract ▾
All authors Dasen Dai, Yanteng Zhang, Shuoqi Li, Yuxiang Wei, Hongjie Yu, Qingxin Zhang, Qizhen Lan, Jagath C. Rajapakse, Vince D. Calhoun
Artificial Intelligence · Cross-listed from cs.SD
TF-MoE: Time-Frequency Mixture-of-Experts for Efficient Speech Separation
Qinzhe Hu, Chenda Li, Wangyou Zhang et al. (+3 more)
Recent advances in
speech separation (SS) have led to compact front-end models with small parameter sizes, yet their high computational cost remains a major barrier for deployment on edge devices. To address this, we propose TF-MoE, a sparse Mixture-of-Experts (MoE) framework that enhances model capacity with almost no increase in inference cost. Our method introduces dynamic expert specialization in time and frequency dimensions through alternating time-wise and frequency-wise MoE modules, each dynamically selecting experts per frame or mel band. Built upon a mel-band-splitting Conformer backbone, TF-MoE achieves strong performance on SS tasks under low-compute settings. Experimental results demonstrate that TF-MoE consistently improves separation performance under computation cost constraints, outperforming BSRNN by +3.8 dB SDR on Libri2Mix with comparable 4.1 GMACs/s inference cost. This positions TF-MoE as a promising candidate for edge-device deployment.
Read the full abstract ▾
All authors Qinzhe Hu, Chenda Li, Wangyou Zhang, Shujie Liu, Yan Lu, Yanmin Qian
Artificial Intelligence · Cross-listed from cs.CL
Coverage-Driven KV Cache Eviction for Efficient and Improved Inference of LLM
Shuvendu Roy, Mengyao Zhai, Hossein Hajimirsadeghi et al. (+1 more)
Large
language models (LLMs) excel at complex tasks like question answering and summarization, thanks to their ability to handle long-context inputs. However, deploying LLMs is costly, not only due to the high computational demands of quadratic complexity of self-
attention and auto-regressive generation, but also because of the significant
memory overhead required for storing the key-value (KV) cache during inference. To reduce the
memory cost, existing KV-cache eviction strategies leverage the sparsity in
attention to selectively store a subset of
tokens. While reducing the
memory footprint, such approaches show a considerable drop in performance, especially in tasks that require long-context
reasoning. We identify that the drop in performance is linked to a reduction in the coverage of unique
tokens. Additionally, we theoretically show that reduced coverage limits the mutual information between inputs and outputs, thereby impairing predictive accuracy. To this end, we introduce K-VEC, a novel coverage-aware KV-cache eviction strategy that prioritizes
token coverage while evicting
tokens in the cache. K-VEC introduces a cross-head and a cross-layer coverage module to enhance
token retention across
attention heads and model layers, mitigating performance degradation caused by low coverage. Evaluated on 16 LongBench subsets, K-VEC exhibit up to 10.35 points improvement over the existing methods under the same eviction rate and
memory constraint. Comprehensive evaluations validate the effectiveness of our approach and demonstrate its potential for efficient LLM deployment in resource-constrained settings.
Read the full abstract ▾
All authors Shuvendu Roy, Mengyao Zhai, Hossein Hajimirsadeghi, Golnoosh Samei
Artificial Intelligence · Cross-listed from cs.LG
VISTA-DZ: Visual Semantic Trajectory Adaptation for Personalized Dilemma Zone Prediction
Chuheng Wei, Ziye Qin, Ziran Wang et al. (+1 more)
Driver decision making in the dilemma zone at signalized intersections is safety critical, as vehicles approaching a yellow signal must decide whether to stop or proceed within limited time and distance margins. Accurate prediction of both stop-go decisions and decision timing is important for adaptive signal control, advanced driver assistance systems, and human-centered intelligent transportation applications. However, dilemma zone behavior is strongly driver dependent. Similar approach trajectories may lead to different decisions across drivers because of differences in risk preference, braking habit, and decision threshold. Existing personalized models often rely on handcrafted scalar descriptors, which provide useful but limited summaries of individual behavior. This paper proposes VISTA-DZ, a semantic-profile-conditioned framework for personalized stop-go and decision-time prediction. Historical trajectories are converted into visual representations, interpreted by a
vision-
language model to generate behavioral profiles, and encoded as semantic
embeddings to condition a dual-output prediction network. The final model combines a bidirectional GRU encoder, driver-conditioned multi-head cross-
attention, and Feature-wise Linear Modulation for temporal evidence selection and feature adaptation. Experiments on the SDZ
dataset and a newly collected FDZ
dataset show that VISTA-DZ outperforms trajectory-only and handcrafted personalization baselines, achieving 93.26% in-domain simulation accuracy and 90.22% mean accuracy across 20 held-out simulation drivers. Cross-domain results further show feasible zero-shot simulation-to-real transfer and better real-world generalization when simulation data are combined with limited field data.
Read the full abstract ▾
All authors Chuheng Wei, Ziye Qin, Ziran Wang, Guoyuan Wu
Artificial Intelligence · Cross-listed from cs.SD
Proteus: Automated Adversarial Robustness Testing for Audio Deepfake Detectors
Nicolas M. Müller, Aditya Tirumala Bukkapatnam and Zohaib Ahmed
We present Proteus, a framework developed at Resemble AI for automated robustness testing of our audio deepfake detection system. Given a detector, Proteus systematically searches over sequences of everyday audio transformations (codec transcoding, additive noise, reverberation, dynamic-range compression, and VoIP simulation) to find combinations that fool the detector while preserving
speech quality. We propose two complementary search strategies: (1) a breadth-first search that exhaustively maps augmentation effectiveness across the parameter space, and (2) a Q-learning
agent designed to efficiently discover deeper attack chains by exploiting structural patterns in the BFS data. We report findings from continuous deployment of Proteus against our production detector, showing that specific augmentation chains can reliably flip detection verdicts while preserving
speech intelligibility and speaker identity. We discuss how these findings are used to harden the detector through targeted retraining.
Read the full abstract ▾
Artificial Intelligence
Learned Coordination Conventions in Cooperative MARL: Measuring the Translation Gap Between Theory-Informed Roles and Learned Routing
Yoosung Hong
Role-semantic assignments provide priors over how heterogeneous
agents may coordinate, but cooperative MARL systems instead settle on conventions through decentralized, non-stationary learning, with no guarantee that the resulting structure matches those priors. We study this translation gap between theory-informed role expectations and learned coordination structure through a diagnostic combining a role-routing
matrix, formation sensitivity ($\Delta_{\max}$), and gradient/occlusion attribution across three-role MiniGrid and SMACv2 (Terran) environments. We show that label-conditioned
attention produces substantially more concentrated and role-specific routing than flat MLP baselines, remains stable under 3v3–9v9 scaling, transfers zero-shot across team sizes, and is invariant to ally-slot padding. A 5-seed re-evaluation shows partial
alignment between learned conventions and designer-specified priors while revealing where small-n noise can manufacture apparent strategic divergence. We present these results as an empirical framework for measuring coordination structure in cooperative MARL rather than as a new
equilibrium concept or causal explanation.
Read the full abstract ▾
Artificial Intelligence · Cross-listed from cs.DL
Em-ergence of the em-dash: a population-level rise in em-dash frequency in medRxiv preprints at the dawn of the large-language-model era
Przemysław Czuma
Large
language models (LLMs) can leave subtle stylistic traces in assisted text; one of the most cited is the em-dash (Unicode U+2014). Yet no one has measured whether em-dash use has changed in the scientific literature. This study, pre-registered on the Open Science Framework (HFT8C), used the full set of medRxiv full-text XML preprints from the official Text-and-Data-Mining resource. The primary cohort was first, original versions deposited 2020-2025 with an extractable Discussion section of at least 500 characters (N = 69,632). The primary endpoint was the presence of at least one em-dash in the Discussion; the principal measure was the absolute change in its prevalence between the pre-ChatGPT era (before 30 November 2022) and the post-ChatGPT era, estimated with a logistic model with standard errors clustered by first author. The analysis plan (six supporting analyses, six sensitivity analyses, two falsification tests) was frozen before any confirmatory result was computed. Em-dash prevalence in Discussion sections rose from 4.23% before ChatGPT to 11.58% afterward, an absolute increase of 7.35 percentage points (95% CI 6.94-7.77; odds ratio 2.96, 95% CI 2.77-3.17). The rise was not a sharp jump but a gradual, delayed acceleration: near 4% through 2023, 8.0% in 2024, and 20.3% in 2025. The effect survived every feasible sensitivity analysis (7.35-7.60 pp) and both falsification tests; a placebo split within the pre-LLM era showed no meaningful change (+0.13 pp, 95% CI -0.33 to +0.58), and was essentially absent in boilerplate sections. Independent LLM-associated lexical markers and within-paper section comparisons pointed the same way. The em-dash is a population-level indicator, not a per-paper detector of LLM use, and the design cannot establish causality; it shows that something in how scientific literature is written changed markedly in the early 2020s, and roughly when.
Read the full abstract ▾
Artificial Intelligence · Cross-listed from cs.SE
RESOURCE2SKILL: Distilling Executable Agent Skills from Human-Created Multimodal Resources
Yijia Fan, Zonglin Di, Zimo Wen et al. (+8 more)
Skills are a useful abstraction for software
agents, turning human and
agent experience into reusable procedural knowledge. Yet existing skill libraries are mostly hand-written, text-centric, or derived from
agent traces, leaving tutorial
videos and other
multimodal human resources largely underused. We present RESOURCE2SKILL, a framework that distills
multimodal resources, including tutorial
videos, repositories, articles, and reference artifacts, into executable skills for software
agents. RESOURCE2SKILL organizes these skills as a hierarchical
multimodal Skill Wiki, where each entry combines structured text, code, visual examples, metadata, and provenance. This design preserves complementary signals from different resources:
videos capture temporal operations and visual effects, code captures executable tool patterns, and articles or artifacts provide conceptual and stylistic grounding. At inference time,
agents retrieve and compose relevant skills from the wiki; when coverage is insufficient, the same construction operator can acquire new skills online. Across seven practical authoring domains, RESOURCE2SKILL improves average overall score by +11.9 percentage points over no-skill
agents and outperforms strong harness baselines in 26 of 28 main-aggregate model-domain
cells. Ablations confirm the value of
multimodal skill format, hierarchical organization, source diversity, selection strategy, and online acquisition.
Read the full abstract ▾
All authors Yijia Fan, Zonglin Di, Zimo Wen, Yifan Yang, Mingxi Cheng, Qi Dai, Bei Liu, Kai Qiu, Yue Dong, Ji Li, Chong Luo
Artificial Intelligence
OSWorld 2.0: Benchmarking Computer Use Agents on Long-Horizon Real-World Tasks
Mengqi Yuan, Zilong Zhou, Xinzhuang Xiong et al. (+33 more)
Existing computer-use
benchmarks fail to capture the realism, complexity, and long-horizon demands of real-world computer use, limiting their ability to reveal the limitations of frontier
agents. We introduce OSWorld 2.0, a
benchmark of 108 long-horizon computer-use workflows across everyday and professional tasks, designed to capture complex and challenging real-world phenomena. Each task represents a realistic end-to-end workflow that takes human users a median of about 1.6 hours to complete and requires an average of 318 tool calls with Claude Opus 4.7 using maximum thinking, compared with about 30 in OSWorld 1.0. OSWorld 2.0 targets challenge phenomena that are common in real workflows yet underrepresented in prior
benchmarks, spanning interaction-design challenges such as streaming interaction and dynamic environments, as well as
agent-pattern challenges such as cross-source
reasoning, implicit-state inference, and visual-spatial precision. Tasks are grounded in authentic input artifacts and cross-referenced against realistic stateful user profile data, and include separate safety reports
auditing safety-sensitive execution. Under our primary binary-completion metric at 500 steps, Claude Opus 4.8 with maximum thinking and batched tool calls scores best but still completes only 20.6% of tasks at a 54.8% partial score; GPT-5.5 is far more
token-efficient yet plateaus near 13%. These results show that current
agents are still far from professional-level computer use: rather than stumbling on basic GUI control or coding, they lose track of constraints, miss information that arrives mid-task, guess rather than ask the user, and skip verification, struggling most when a task hinges on hidden state they must recover.
Read the full abstract ▾
All authors Mengqi Yuan, Zilong Zhou, Xinzhuang Xiong, Weiming Wu, Jiayang Sun, Jiamin Song, Kaiqian Cui, Bowen Wang, Haoyuan Wu, Yitong Li, Dunjie Lu, Haikong Lu, Qi Zhen, Xinyuan Wang, Jiaqi Deng, Yuhao Yang, Cheng Chen, Boyuan Zheng, Alex Su, Xiao Yu, Hao Zou, Saaket Agashe, Xing Han Lu, Manpreet Kaur, Zhengyang Qi, Vincent Sunn Chen, Frederic Sala, Dayiheng Liu, Junyang Lin, Zhou Yu, Yu Su, Siva Reddy, Xin Eric Wang, Peng Qi, Tianbao Xie, Tao Yu
Artificial Intelligence · Cross-listed from cs.DB
SemJoin: Semantic Join Optimization
Christopher Gou, Aditya Banerjee, Jiaxuan Wang et al. (+1 more)
Integrating unstructured data into relational database systems is increasingly important as demand grows for natural
language querying and analysis. A semantic join, joining two tables under a natural-
language predicate, can be evaluated with a large
language model (LLM), but comparing every pair of tuples requires O(M x N) LLM invocations and is cost-prohibitive at scale. Existing systems reduce this cost but typically commit to a single fixed strategy (e.g.,
embedding similarity or one batched scheme) regardless of the data or the join predicate. We propose an LLM-
agent-based decision pipeline that optimizes semantic joins by matching the execution strategy to the characteristics of the underlying tables. An LLM advisor routes each join to one of two strategies: a Cluster Join, which prunes candidates via unsupervised
embedding clustering and sample-based filtering, or a Classifier strategy for predicates that reduce to a shared discrete label set. Across three diverse
datasets (IMDb reviews, email contradictions, and Stack Overflow tags), the advisor consistently identifies the optimal execution strategy for each workload. This dynamic routing proves decisive: it outperforms adaptive block join (ABJ) by 20-33 F1 points across all
datasets while consuming fewer
tokens on two of the three, and achieves higher F1 scores than featurized-decomposition join (FDJ) at one to two orders of magnitude lower
token cost.
Read the full abstract ▾
All authors Christopher Gou, Aditya Banerjee, Jiaxuan Wang, Chunwei Liu
Artificial Intelligence · Cross-listed from cs.CV
MotionAtlas: Detailed Region Captioning for Motion-Centric Videos
Weisong Liu, Haochen Wang, Kuan Gao et al. (+8 more)
We propose MotionAtlas, a system for detailed captioning of motion-centric
videos, comprising (1) a dedicated human-annotated
benchmark, (2) a scalable, high-quality pipeline to construct training samples, and (3) a family of powerful
Video-MLLMs. Unlike conventional global motion captioning
datasets, we focus on region-aware motion captioning: given a
video and a spatiotemporal mask, the model generates precise descriptions of motion within the target region, thereby alleviating visual clutter and motion
entanglement and enabling reliable, quantifiable evaluation. Concretely, we first build MotionAtlas-Bench, a comprehensive
benchmark comprising 2,073 multiple-choice questions, meticulously annotated for a curated set of high-quality, motion-centric
videos, to evaluate fine-grained motion understanding of the objects in question. Second, we design a rigorous and scalable data pipeline that leverages self-bootstrap refinement to suppress fine-grained
hallucinations, yielding 159k high-quality motion captioning data. Third, we design a tailored training data composition strategy, which achieves consistent and substantial performance gains across diverse baseline
Video-MLLMs, including Molmo2 and Qwen3-VL. For instance, MotionAtlas-4B surpasses Qwen3-VL-4B by an average of 5.2 percentage points across general motion
benchmarks. The
benchmark,
dataset, and code have been released.
Read the full abstract ▾
All authors Weisong Liu, Haochen Wang, Kuan Gao, Yuhao Wang, Yikang Zhou, Zhongwei Ren, Jacky Mai, Anna Wang, Yanwei Li, Jason Li, Zhaoxiang Zhang
Artificial Intelligence · Cross-listed from cs.SE
SAKE: Software Architectural Knowledge Evaluation Benchmark for Large Language Models
Tiziano Santilli, Francesco Daghero and Mayhar Tourchi Moghaddam
Large
Language Models (LLMs) are increasingly used as assistants across the software development lifecycle, yet their ability to reason about software architecture remains largely unmeasured. Architectural decision-making depends on quality attribute trade-offs, design patterns, and system-level constraints, none of which are exercised by
benchmarks that target syntactic or algorithmic tasks. We introduce SAKE (Software Architectural Knowledge Evaluation), a standardized and reproducible
benchmark for assessing software architectural knowledge in LLMs. SAKE comprises 2154 expert-curated multiple-choice questions, each with four options, stratified across eight architectural categories and four context-length levels. We evaluate 11 proprietary and open-weight models in zero-shot and five-shot settings. Overall accuracy is high, but performance varies markedly across categories, revealing competency gaps in areas central to professional practice. SAKE, its evaluation scripts, and all results are released as open source to give the community a baseline for tracking architectural
reasoning in LLMs.
Read the full abstract ▾
Artificial Intelligence · Cross-listed from cs.CL
The Verbose Context Problem in Medical Records
Shiva Kaul, Min-Gyu Kim, Anjum Khurshid et al. (+1 more)
The verbose context problem occurs when structured concepts have
token-inefficient textual representations. This bottleneck is acute in population health: cohort-level analysis of longitudinal patient records requires
reasoning over thousands of medically-coded events, often exceeding 400K
tokens in total. We present PopMedQA, a
benchmark isolating this problem through computational tasks on groups of longitudinal patient records. We construct the
benchmark using neopatient, a new library for
language-controlled generation of artificial patient records. Through extensive ablations – including prompting strategies,
prompt compression, and agentic decomposition – we find that domain-independent methods fail to alleviate the verbose context problem. There remains significant opportunity to exploit domain-specific structure in
language model inputs for population-scale
reasoning.
Read the full abstract ▾
All authors Shiva Kaul, Min-Gyu Kim, Anjum Khurshid, Sriram Vishwanath
Artificial Intelligence
UCOB: Learning to Utilize and Evolve Agentic Skills via Credit-Aware On-Policy Bidirectional Self-Distillation
Songjun Tu, Chengdong Xu, Qichao Zhang et al. (+6 more)
Skill
memories can improve agentic reinforcement learning by reusing past experience as textual guidance, but retrieved skills are not oracular: they may help in one state while misleading the same policy in another. This makes the common privileged-teacher assumption fragile, namely that a skill-conditioned
prompt can be treated as a fixed teacher for the no-skill
prompt. We introduce UCOB, a framework for learning to utilize and evolve agentic skills via credit-aware on-policy bidirectional self-
distillation. UCOB treats skill-conditioned and no-skill
prompts as two on-policy context views of the same model, compares their return-to-go within the same task and anchor state, and uses the higher-return view as the local teacher. This local credit signal internalizes useful skill-conditioned behavior, corrects misleading skill usage, and guides task/state skill
memory updates, utility-aware
retrieval, and reflection self-training. Experiments on agentic tasks, including ALFWorld, WebShop, and Search-QA, show that UCOB outperforms skill-free RL, skill-
memory baselines, and self-
distillation methods across model scales, with up to 23.5 and 18.0 point gains over SOTA baselines on ALFWorld and WebShop. Ablations and analyses further validate its core mechanisms, continual adaptation across environments, and modest training overhead. Code is available at https://github.com/TU2021/UCOB.
Read the full abstract ▾
All authors Songjun Tu, Chengdong Xu, Qichao Zhang, Yiwen Ma, Yaocheng Zhang, Linjing Li, Dong Li, Xiangyuan Lan, Dongbin Zhao
Artificial Intelligence
Cognitive World Model for Progressive BDI/E Trajectory Evaluation of Conversational Agents
Minghui Ma, Bin Guo, Hao Wang et al. (+4 more)
As LLM-based conversational
agents advance toward increasingly open-ended and interaction-intensive scenarios, task completion alone provides an incomplete assessment of their effectiveness. The
evolution of users' internal states, including beliefs, desires, intentions, and
emotions (BDI/E), serves as an intermediate signal connecting
agent behaviors with interaction outcomes and reflects how conversational strategies shape users during multi-turn interactions. However, existing evaluation paradigms primarily focus on surface-level responses or final outcomes, providing limited insight into the underlying cognitive processes. This limitation makes it difficult to diagnose why
agents succeed or fail and to optimize their interaction strategies. To address this challenge, we propose Cognitive World Model (CogWM), an LLM-based cognitive user model that jointly models users' BDI/E states and corresponding responses, enabling explicit cognitive trajectory tracking. Trained on 150K user-turn samples with Qwen3-14B, CogWM achieves superior performance over existing user simulation baselines in both response fidelity and cognitive state understanding. Interactions with six state-of-the-art LLMs demonstrate that CogWM enables progressive comparison of
agents through cognitive trajectories, revealing distinct
agent patterns and complementary relationships between cognitive
evolution and behavioral outcomes.
Read the full abstract ▾
All authors Minghui Ma, Bin Guo, Hao Wang, Han Wang, Mengqi Chen, Jingqi Liu, Yan Liu
Artificial Intelligence
Faults in Our Formal Benchmarking: Dataset Defects and Evaluation Failures in Lean Theorem Proving
Pawan Sasanka Ammanamanchi, Siddharth Bhat and Stella Biderman
Benchmarks for LLM-assisted
theorem proving in Lean are often treated as intrinsically reliable because every solved instance comes with a machine-checked proof. However, the kernel only checks that a proof establishes a formal statement; it does not verify that the statement faithfully encodes the intended informal problem, nor that evaluation harnesses are robust to trivial or
adversarial solutions. We audit five widely used Lean
theorem-proving
benchmarks and their forks, using corpus-scale static checkers to surface 4,833 findings, including 398 mechanically certified issues such as counterexamples, vacuous
theorems, and unsound axioms. We also document semantic defects such as missing hypotheses, problem simplification, incomplete or incorrect translations, and Lean-specific specification hazards. Beyond
dataset construction, we survey evaluation-time failure modes and show, on corrected subsets, that defects can both inflate and deflate reported prover scores. We propose a fault taxonomy, a suite of automated checkers and recall-oriented semantic audit
prompts, and release standards to guide the creation of formal math
datasets and to make evaluation more reproducible and trustworthy. Our checkers, audit
prompts, and corrected
dataset snapshots are available at https://github.com/Shashi456/atp-checkers.
Read the full abstract ▾
Artificial Intelligence · Cross-listed from cs.LG
Reported Confidence in LLMs Tracks Commitment More Than Correctness
Dharshan Kumaran
Confidence is an estimate of the probability that a chosen answer is correct. Verbal confidence reports are widely used as uncertainty measures in large
language models, but whether they are best understood as estimates of correctness is unclear. We test this with a two-stage abstention paradigm from the neuroscience of perceptual decision making: a model first answers and reports its confidence, then decides whether to commit it to a user or abstain. Across four non-
reasoning models,
prompt framings, and confidence formats, verbal confidence predicted the commit/abstain decision substantially better than whether the answer was correct. Calibrated
token log-probabilities showed the opposite profile, with abstention-prediction coupled to correctness discrimination, the signature of an answer-evidence signal. After removing the variance verbal confidence shared with log-probabilities, the residual stayed aligned with commitment while its link to correctness fell to near chance. The dissociation generalised to four
reasoning models across four
benchmarks of varying difficulty, from hard multiple-choice to frontier-level freeform questions. Mechanistic analyses in Gemma 3 and 4 were convergent: a post-answer state known to causally support verbal-confidence generation already encoded the future abstention decision before the abstention
prompt, organised mainly by that decision rather than by correctness, the two lying in approximately orthogonal directions in activation space. Steering along a verbal-confidence-specific direction causally shifted abstention. Verbal and log-probability confidence are thus not interchangeable: log-probabilities track answer evidence and correctness, whereas verbal confidence is better understood as a behaviour-facing readout of an internal commit-readiness state, challenging the practice of treating verbal reports as proxies for reliability.
Read the full abstract ▾