Artificial Intelligence · Cross-listed from cs.CV
Which Modality Decides? Counterfactual Modality Attribution for Multimodal LLMs
Vahidin Hasic, Chao Wang, Luis C. Garcia-Peraza-Herrera et al. (+2 more)
Multimodal large
language models (MLLMs) increasingly support high-stakes decision making by combining complementary information from images and text. While existing explainability methods identify influential image regions or text
tokens, they cannot answer a fundamental question: which modality drives a prediction? Consequently, a model may produce the correct output while relying on the wrong source of evidence, masking shortcut learning and unsafe
reasoning. We formulate modality attribution as a complementary explainability objective for
multimodal foundation models and propose Counterfactual Modality Attribution (CMA), the first framework for quantifying modality-level contributions in MLLMs. CMA generates image-only, text-only, and joint
multimodal counterfactuals using coupled
diffusion priors and converts them into principled modality attribution scores through a cooperative game-theoretic formulation based on Shapley values. We evaluate CMA on controlled synthetic
benchmarks with known ground-truth modality reliance and on a real-world
multimodal clinical
dataset. CMA correctly identifies the decision-driving modality in 98% of controlled cases and consistently outperforms baselines, revealing failures of cross-modal
reasoning that remain invisible to predictive accuracy alone. Our results establish modality attribution as a complementary dimension of explainability beyond feature attribution, providing a principled framework for
auditing multimodal foundation models in safety-critical applications.
Read the full abstract ▾
All authors Vahidin Hasic, Chao Wang, Luis C. Garcia-Peraza-Herrera, David Watson, Senka Krivic
Artificial Intelligence · Cross-listed from cs.CV
Empirical investigation of 3D CT Foundation Models and Unsupervised Adaptation for Head and Neck Cancer Recurrence Prediction
Bilel Guetarni, Feryal Windal, David Pasquier et al. (+1 more)
The rapid emergence of 3D CT foundation models has opened new avenues for predictive modeling from CT imaging, offering a compelling alternative to traditional radiomics which is known to suffer from reproducibility issues and sensitivity to acquisition protocol variations. Yet, as these models grow in availability, a critical need arises to evaluate how well their learned representations generalize across diverse clinical settings and whether adaptation to specific downstream tasks is necessary to unlock their full potential. To address these questions, we benchmarked several 3D CT foundation models for predicting recurrence-free survival in head and neck cancer across two public
datasets totaling 3,644 patients, evaluating various adaptation strategies and modality fusion mechanisms. Our findings reveal persistent difficulty in identifying features that generalize consistently across different imaging distributions, as evidenced by significant performance drops on external validation cohorts. Ultimately, the integration of imaging features with clinical data remains the most accurate approach for prognostic prediction, though achieving universal generalization across varied clinical contexts continues to represent a substantial challenge for the current generation of models.
Read the full abstract ▾
All authors Bilel Guetarni, Feryal Windal, David Pasquier, Halim Benhabiles
Artificial Intelligence
H+ Embedding: Harmonizing Global and Token-Level Retrieval with Context-Dependent Phrases
Shusen Zhang, Junyi Hu, Ye Feng et al. (+6 more)
Terminology-intensive
retrieval, especially in medical settings, depends on preserving multi-word entities, abbreviations, numerical constraints, and compositional concepts. However, existing representations lie at two extremes: single-vector retrievers often over-compress local relevance signals, while
token-level late interaction retains every tokenizer subword at substantial indexing, storage, and scoring cost. This mismatch raises a natural question: can context-dependent phrases provide a useful
retrieval unit between global vectors and
tokens? We introduce H+
Embedding, a unified multi-granularity retriever that predicts variable-length phrase partitions, preserves uncovered
tokens as singletons, and applies importance-guided unit selection with weighted MaxSim interaction. Across 16 scientific, medical, and bilingual tasks, its phrase
retrieval branch exceeds the global
retrieval branch by 6.91 macro nDCG@10. It also nearly matches
Token while using 13.7% fewer document vectors and outperforms content-independent grouping rules under moderate vector budgets. Context-dependent phrase interaction therefore provides an intermediate quality-cost point between global compression and
token-level interaction for practical
retrieval systems.
Read the full abstract ▾
All authors Shusen Zhang, Junyi Hu, Ye Feng, Ziteng Wang, Zhaoyuan Pan, Xiaojun Yuan, Jiangshou Hong, Guosheng Dong, Xiangzhi Wang
Artificial Intelligence · Cross-listed from cs.LG
MPP-GNN: Subject-Adaptive Community Detection for fMRI-Based Alzheimer's Disease Classification
Yang Zhang, Xiao Zhou, Jonathan Warrell et al. (+3 more)
Functional
magnetic resonance imaging (fMRI) is a widely used technique for studying the
brain. Recent methods that utilize graph neural networks (GNNs) for analysis of
brain functional connectivity have shown great potential for the classification of
brain disorders, such as Alzheimer's disease (AD). However, these methods often assume a preset number of functional modules across all subjects, which overlooks inter-subject variability. In addition, the discovered modules are rarely used to directly guide the learned connectivity patterns. Here, to address these issues, we propose a Meta Probabilistic Pooling GNN (MPP-GNN). We frame the model's task as a coupled, bilevel optimization that performs adaptive graph partitioning hierarchically to discover subject-specific modules and then uses the discovered
brain modules as an explicit prior to guide edge refinement and representation learning. We validate MPP-GNN on two public
datasets for AD classification, achieving the highest AUC in comparison to established baselines for both
datasets. Furthermore, our analysis demonstrates that MPP-GNN shows significant
alignment with the canonical functional-network organization defined by the Yeo
brain atlas and reveals a network-level dedifferentiation pattern for AD.
Read the full abstract ▾
All authors Yang Zhang, Xiao Zhou, Jonathan Warrell, Avram Holmes, Xuan Zhang, Mark Gerstein
Artificial Intelligence
Multi-Agent Planning with Spatio-Temporal and Topological Constraints using STL-GO
Sheryl Paul, Vidisha Kudalkar, Anand Balakrishnan et al. (+3 more)
Multi-
agent planning problems arise in a variety of engineering applications, such as multi-
robot wildfire fighting and unmanned aerial inspection in factories. A particular challenge is the existence of spatio-temporal (i.e., when and/or where an
agent should do what) and topological constraints (i.e., how
agents should interact), as typically formalized via the notion of graphs. Over the last years, various frameworks have been proposed that can capture such constraints via spatio-temporal logics. We focus here on spatio-temporal logic with graph operators (STL-GO), a recent formalism that supports
reasoning about multiple
agents and their
topologies, such as sensing, communication, and task
topologies. In this paper, we consider the problem of planning multi-
agent paths that satisfy constraints written in STL-GO. This problem is particularly challenging due to the need of encoding multiple, potentially time-varying graphs via the graph operators inherent to STL-GO. We present two encodings of this problem, one based on mixed-integer programming (MIP) and another based on satisfiability modulo theory (SMT), with soundness guarantees. We provide a unified interface for specifying
agent constraints, their graph
topologies, and the STL-GO specification, enabling seamless use of both methods and facilitating direct comparison between them. We evaluate both encodings on a multi-
UAV search-and-rescue
benchmark, ablating over team size and graph complexity, highlighting the expressiveness of the proposed encodings under dynamic multi- graph interactions.
Read the full abstract ▾
All authors Sheryl Paul, Vidisha Kudalkar, Anand Balakrishnan, Lars Lindemann, Alberto Speranzon, Jyotirmoy V. Deshmukh
Artificial Intelligence
ViSAGE: Constructing Self-Correcting Memories for Long-Form Video Understanding
Xinkui Zhao, Enbo Chen, Yifan Zhang et al. (+4 more)
Multimodal agents operating in long-horizon environments must build and continually update multimedia
memories to support entity-consistent, temporally grounded
reasoning. However, existing agentic
memory approaches often discard fine-grained dentity cues under aggressive compression and segment-wise processing. They also rely heavily on vector similarity
retrieval, which can surface semantically related yet identity-mismatched evidence, leading to entity confusion, error propagation, and hallucinated answers. We propose ViSAGE, a
multimodal agentic
memory framework that constructs self-correcting, entity-centric
memories. Specifically, ViSAGE anchors entity identity via cross-modal binding over long temporal ranges. It then applies bidirectional
memory refinement to propagate delayed identity evidence, retroactively unifying historical records and improving future
reasoning. We also introduce multi-
agent cross-verification to assess retrieved evidence under an identity-evidence
alignment onstraint, enabling abstention instead of unsupported answers when evidence is missing. Extensive results demonstrate that ViSAGE consistently outperforms the strongest baseline, achieving 5.9% higher accuracy.
Read the full abstract ▾
All authors Xinkui Zhao, Enbo Chen, Yifan Zhang, Chang Liu, Guanjie Cheng, Naibo Wang, Yueshen Xu
Artificial Intelligence
Reasoning in Real World Clinical Care: Why Large Language Models Are Not Yet Safe for Autonomous Clinical Decision Support
Shayndhan Sivanathan, Shravan Nageswaran, Mehdi Zadem et al. (+10 more)
LLM now pass medical licensing examinations and, in curated cases, can rival physicians at diagnostic
reasoning. These developments have accelerated the use of LLMs for symptom assessment and clinical decision support in diagnostic and treatment guidance, administrative documentation, and rules-based alert enhancement. This Perspective concerns the most consequential of these applications: the autonomous triage of self-presenting, undifferentiated patients, with little or no clinician in the loop. For that task, the evidence of safety does not yet exist. The gap is not in medical knowledge but in the fidelity of clinical evaluation: a model optimized to continue the most probable text is not optimized to act safely when the safe answer is the improbable must-not-miss diagnosis. Safe triage is not the selection of the most likely diagnosis; it is a sequential decision under asymmetric cost, in which the single catastrophic miss outweighs many false alarms, and the decisive signal may be one the patient has not volunteered - and that the model has not been trained to seek. The core deficit is therefore one of information gathering under uncertainty. Under incomplete histories, LLM systems may fail to show the behaviors safe triage requires: broadening the differential; seeking the missing red flag; lowering the threshold for escalation; deferring judgement until sufficient information is obtained; and escalating concern where high-harm diagnoses remain unexcluded. These modes of failure for LLMs can be difficult to detect considering that evaluations to date often use complete, well-curated, confidence-gated simulations. The application of LLMs under these conditions may be amplified by assistant-like behaviors and positive bias, including credulity, agreeableness, and miscalibration - when these are not constrained by clinical triage logic.
Read the full abstract ▾
All authors Shayndhan Sivanathan, Shravan Nageswaran, Mehdi Zadem, Ryaan Sultan, Nicolas von Mallinckrodt, Max Solovyev, Alexey Matyushkin, Sumon Sadhu, Gabriele C DeLuca, Sanjeeva Jeyaretna, James Hillis, Manoj Ramachandran, Prakash Jayakumar
Artificial Intelligence
SciDataSailor: Deep Scientific Data Exploring
Jiyong Rao, Yicheng Qiu, Chi Zhang et al. (+2 more)
Scientific
datasets are commonly organized as hierarchical repositories containing heterogeneous and interdependent files, making their inspection, integration, and analysis labor-intensive and reliant on domain expertise. Although large
language model (LLM)
agents have advanced substantially in planning,
reasoning, and tool use, existing research has largely overlooked their ability to interact with real scientific data assets through executable environments. We introduce Deep Scientific Data Exploration, an agentic task paradigm in which
agents navigate repositories, interpret heterogeneous files and schemas, execute analyses, integrate cross-file evidence, and produce conclusions grounded in executed observations. To operationalize this paradigm, we present SciDataSailor, a framework for synthesizing tool-interactive trajectories by balancing broad exploration with targeted exploitation. SciDataSailor instantiates trajectory synthesis as Monte Carlo Tree Search (MCTS) with four task-specific mechanisms: difficulty-stratified exploration seeds, dual-feedback first-play urgency, hierarchical strategy-to-tool action generation, and
entropy-guided branching. Using this framework, we construct SciDataSailor-SFT-2K for supervised fine-tuning and SciDataSailor-Bench for evaluation, with the latter comprising 627 meta-information summarization tasks and 586 scientific question-answering tasks across 27
datasets spanning the life, earth, and physical sciences.
Read the full abstract ▾
All authors Jiyong Rao, Yicheng Qiu, Chi Zhang, Chunfeng Song, Runkai Zhao
Artificial Intelligence · Cross-listed from cs.CR
ThreatForest: Multi-Agent Attack Tree Generation with Pluggable TTP Framework Mapping
Cristian Leo, Anton Dykyi, Danny Cortegaca et al. (+2 more)
Threat modeling is essential for secure software development, yet manual analysis of cloud-native architectures is slow and demands scarce security expertise. We present ThreatForest, a multi-
agent system that generates structured attack trees from source code repositories, maps attack steps to adversary tactics, techniques, and procedures (TTPs) from a pluggable set of frameworks (MITRE ATT&CK, CAPEC, and cloud-specific threat
matrices), and synthesizes actionable mitigations. ThreatForest decomposes threat modeling into a multi-stage
agent pipeline – repository analysis, context refinement, threat generation, parallel attack-tree construction with TTP mapping and mitigation synthesis, and report generation – orchestrated as a directed graph with deterministic verification gates, bounded retries, and three human-in-the-loop validation points. A domain-specific sentence-
transformer maps each attack step to candidate techniques by cosine similarity; we show empirically that this
embedding stage, not the surrounding pipeline, is the dominant accuracy bottleneck. We evaluate ThreatForest across seven application domains on a sixteen-dimension rubric, scored by a panel of independent LLM raters with an
adversarial verification pass and expert review. Panel-measured quality reaches 0.63-0.68 (on a 0-1 scale) for threat statements, attack trees, and mitigations, but only 0.29 for
embedding-only TTP mapping – a gap stable across all seven domains that isolates the binding constraint. A controlled single-call baseline on the same model more than doubles mapping defensibility, pinning the limitation on the
embedding encoder rather than the multi-
agent design. To our knowledge, ThreatForest is the first end-to-end system that turns a code repository into TTP-mapped attack trees with evidence-based mitigations across adversary frameworks, with a reusable framework for benchmarking such systems.
Read the full abstract ▾
All authors Cristian Leo, Anton Dykyi, Danny Cortegaca, Daniel Begimher, Prakash Jha
Artificial Intelligence · Cross-listed from cs.IR
Hierarchical Reranking for Scalable Financial RAG System
Joohyun Lee and Sungwoo Hong
Analyzing financial documents such as 10-K filings, tabular disclosures, and macroeconomic reports demands expert
reasoning and extensive time. However, existing
Retrieval-Augmented Generation systems often struggle to process hybrid text-table structures or the massive scale of financial documents. To address these challenges, we propose Hierarchical Reranker, a RAG framework designed to improve
retrieval performance and generative reliability across large-scale financial
datasets. The system integrates three key innovations: Pre-
Retrieval Optimization, enhancing query clarity and search efficiency through normalization, keyword expansion, and table transformation; Hierarchical Reranker Architecture, improving
retrieval precision through a two-stage ranking mechanism; and Long-Context Management, preserving
reasoning accuracy through adaptive input partitioning and fusion under extensive contexts. Across multiple
benchmarks, including FinQA, FinanceBench, and ConvFinQA, the proposed system achieved an NDCG@20 score of 0.7918 and demonstrated superior factual consistency. Its robustness was further validated by achieving second place in the ACM-ICAIF '24 FinanceRAG Challenge. This work presents a deployable, domain-optimized RAG pipeline that enhances both the accuracy and scalability of financial
reasoning, paving the way for automated audit reporting and quantitative investment analysis. The source code will be made publicly available on GitHub upon acceptance.
Read the full abstract ▾
Artificial Intelligence · Cross-listed from cs.LG
Expanding Data-Agnostic Pivotal Instances Selection Models with Proximity Trees and Ensemble Learning
Alessio Cascione, Mattia Setzu, Cristiano Landi et al. (+2 more)
As decision-making processes grow more complex, machine learning tools have become essential for tackling business and societal challenges. However, many existing methods rely on decision-making procedures that are difficult to interpret. Since humans naturally make decisions by comparing new cases with a few representative examples, we aim to design an approach that selects such pivots to construct an interpretable predictive model. Inspired by decision trees, we propose a hierarchical, interpretable-by-design pivot selection model based on the similarity between pivots and input instances. Our method functions both as a pivot selection technique and a standalone predictive model. Extending beyond single pivots, we incorporate pairs of pivots that are used by proximity and oblique trees, as well as ensembles, which enhance the versatility and effectiveness of our proposal. Additionally, our approach is data modality-agnostic, leveraging pre-trained networks for data transformation. Experiments across diverse
datasets, including tabular data, text, images, and time series, demonstrate the effectiveness of our approach, outperforming alternative instance selection strategies and achieving competitive results against state-of-the-art interpretable models while maintaining a minimal number of pivots.
Read the full abstract ▾
All authors Alessio Cascione, Mattia Setzu, Cristiano Landi, Paolo Maria Mancarella, Riccardo Guidotti
Artificial Intelligence
Automated Transcript Analysis for Detecting Flaws in Agentic Benchmarks
Jeff Mohl, Nelson Gardner-Challis, Magda Dubois et al. (+6 more)
Capabilities of frontier models are often assessed using agentic
benchmarks. To trust these results,
benchmarks must accurately measure what they claim to and be free from invalidating flaws. Previous manual audits of
benchmarks such as SWE-Bench-Verified have uncovered several validity issues in transcripts. However, manual review is difficult to scale, and it is unclear whether automated methods can reliably surface flaws that compromise
benchmark validity. In this paper, we developed AI scanners to detect four types of validity issues: ground truth access, tool failure, guessing vulnerability, and answer format ambiguity. We produced grading rubrics for each to instruct human labeling, and evaluated the scanners against human labels on a held-out test set of Inspect Evals
benchmarks. Our scanners identified several verified quality issues in five widely used
benchmarks, including cases unlikely to be caught by random manual inspection. Not all cases were identified, and scanner performance varied substantially across
benchmarks, criteria and models. We highlight several open challenges to be addressed to improve scanners for stronger quality assurance claims, including broader standardization gaps in the evaluation field that degrade scanner performance. Together, these results serve as a proof of concept for using automated transcript analysis to audit
benchmark quality more broadly.
Read the full abstract ▾
All authors Jeff Mohl, Nelson Gardner-Challis, Magda Dubois, Harry Coppock, Benjamin Allan-Rahill, Kaelan Yim, Damian Sójka, James Mann, Justin Olive
Artificial Intelligence · Cross-listed from cs.RO
Corrigible Assistance in One Round: Pragmatic-Pedagogic Best Response
Elle Lazarski and Jaime Fernández Fisac
Assistance games formalize human-
robot collaboration under asymmetric information: the human knows the goal, while the
robot must infer it from observation and interaction in order to assist effectively. In general, computing optimal assistance game strategies online is intractable, since exact solutions require planning in a POMDP. We identify a class of assistance games in which pragmatic-pedagogic
reasoning resolves goal uncertainty in a single time step, rendering the full-horizon game exactly solvable by a tractable best-response procedure. Within this class, we show that mainstream inverse optimal control exhibits an inference ceiling that hinders
alignment, while pragmatic-pedagogic
reasoning overcomes this barrier by immediately disambiguating goals through actions that look equivalent under task execution alone. Finally, we validate our theoretical results and proposed method on a simple collaborative block-building example.
Read the full abstract ▾
Artificial Intelligence · Cross-listed from cs.CL
Models for minimalist RAG: B1ade 335M Embedding and 1B Parameter Small Language Models
Shreyas Subramanian, Mecit Gungor and Vikram Elango
Language and
embedding models used in RAG systems are conventionally assumed to require large-scale pretraining and explicit grounding supervision. We present B1ade, an efficient RAG architecture comprising two purpose-built components: a compact
embedding model and a purpose-built SLM. B1ade-embed, a 335M parameter
retrieval model constructed via parameter-free fusion of five pretrained encoders achieves top MTEB scores among sub-500M models with zero additional training, and B1ade-1B, an SLM trained on low-cost GPUs using Group Relative Policy Optimization (GRPO) on 723M
tokens (2.2M examples) of curated context-question pairs with rewards that optimize only answer similarity. Our central finding is emergent attribution: despite receiving no explicit supervision for source citation, B1ade-1B cites retrieved passages in 42.4% of responses, exceeding the attribution rate of its training distribution by 5.5 percentage points. This demonstrates that grounding behavior can emerge as an accuracy-maximizing strategy under RL training, without explicit reward engineering. On standard QA
benchmarks, B1ade-1B achieves 81.82% on PopQA, 65.8% on PubMedQA, and 51.09% on FEVER. In end-to-end RAG evaluation, B1ade-1B achieves an average score of 0.654 across correctness, completeness, coherence, and faithfulness, a 10.8% improvement over the SFT, while closing the gap with models 1.5x its size. These results show that strategic model composition and reward design suffice for resource-efficient RAG, without large-scale pretraining.
Read the full abstract ▾
Artificial Intelligence
A dataset of rated conceptual arguments
Emery Cooper, Caspar Oesterheld, Linh Chi Nguyen et al. (+2 more)
Large
language models have improved rapidly on tasks with verifiable answers, such as mathematics and programming. Much less is known about their ability to reason about what we call conceptual questions: questions for which no ground truth is realistically accessible and no widely accepted resolution methodology exists, but on which progress can still be made by debating arguments. Most philosophical questions are of this kind, as are central components of questions in AI safety, decision theory, and social choice. Our approach is based on the view that while bottom-line conclusions on such questions are hard to evaluate, individual contextualized arguments can be evaluated far more reliably. We therefore introduce a
dataset of 951 argumentative critiques of 442 position texts, spanning topics from AI safety and decision theory to ethics and politics, with 1,458 ratings by six expert raters along dimensions including centrality, strength, correctness, and clarity. We propose two scoring functions and
benchmark a range of models. Performance tracks general capability rankings.
Read the full abstract ▾
All authors Emery Cooper, Caspar Oesterheld, Linh Chi Nguyen, Alexander Kastner, Ethan Perez
Artificial Intelligence
MedLLM: An Open Medical Language Model at the Sub-Billion Scale
Maxx Richard Rahman, Asim Ahmed, Mihan Mohagheghzadeh et al. (+1 more)
Open medical
language models have converged on a single scale: every widely used system runs at 7B parameters or more, leaving the sub-billion regime uncharacterized. We present MedLLM, an open 0.1B-parameter medical
language model trained through a fully open three-phase pipeline: general pretraining with curriculum sequence-length scheduling, domain fine-tuning on MedFineWeb, a reference-guided medical corpus we release that is selected from general web data by
embedding similarity to medical question-answering (QA) data, and preference-aligned fine-tuning combining SFT with direct preference optimization (DPO). Across medical
benchmarks, MedLLM shows a pattern visible only at sub-billion scale: medical competence does not degrade uniformly under compression but splits by task type. On context-grounded QA it comes within $2.9$pp of a medically adapted 7B model and surpasses the instruction-tuned and general-purpose 7B baselines; on knowledge-recall QA it stays near the task floor on clinical-vignette MedQA yet significantly exceeds every 7B and sub-7B baseline on MedMCQA, indicating that where recall fails the constraint is model capacity rather than adaptation. This dissociation is masked at 7B, where both capabilities are present, and surfaces only when capacity is scarce.
Read the full abstract ▾
All authors Maxx Richard Rahman, Asim Ahmed, Mihan Mohagheghzadeh, Wolfgang Maass
Artificial Intelligence
INCLAIR: Inception-Based Longitudinal Clinical Anomaly Detection with Informed Reasoning
Maxx Richard Rahman and Wolfgang Maass
Detecting anomalies in longitudinal clinical profiles is clinically important but difficult: abnormal evidence is often sparse, patient histories have unequal length, and expert explanations are costly. We propose INCLAIR, a framework that scores each observation against multiple historical contexts, aggregates evidence at the profile level, and generates grounded natural-
language explanations under limited expert supervision. Under stated within-profile exchangeability assumptions, the complete mean subsequence score takes an order-$l$ U-statistic form, yielding a variance decomposition and an incomplete-subset approximation that controls combinatorial inference cost independently of profile length. The same analysis shows that mean aggregation attenuates localized anomalies by a factor set by the anomaly support and profile length, motivating validation-selected top-$k$ pooling. Across three clinical
datasets, INCLAIR consistently outperforms state-of-the-art baselines. We further validate practical relevance through a case study on longitudinal steroid profiles, comparing INCLAIR's predictions and explanations against domain-expert assessments supported by DNA analysis. The results show that INCLAIR enables clinically actionable anomaly detection under limited expert supervision.
Read the full abstract ▾
Artificial Intelligence
Skill Use or Skill Theater? Evaluating the Reasoning Backroom in Skill-Augmented Language Agents
Jinwei Hu, Yi Qi, Xinmiao Huang et al. (+3 more)
Reusable skills are becoming a standard interface for extending
language agents with task procedures. Yet evaluators usually infer skill use from visible
reasoning or the
agent's own attribution. These signals show what the
agent appears to use, not whether the skill changed its decision. We ask whether skill-augmented
agents exhibit a
Reasoning Backroom, a systematic gap between stated skill use and intervention-measured influence. We introduce BACKTRACE, an evaluation framework that pairs each skill-conditioned answer with a matched no-skill counterfactual, intervenes on skill meaning, wording, identity, content, and assignment, and elicits attribution only after the answer is committed. We instantiate the framework as BACKROOMBench, a verified testbed spanning controlled logic and competition mathematics, multiple skill conditions, single-
agent and multi-
agent settings, and diverse model families. Our evaluation reveals a pervasive provenance failure. Across models and domains, stated skill use often remains stable while causal reliance and signed utility vary, producing both silent uptake and performative use. Behavioral effects follow procedural content more reliably than displayed skill identity, whereas stated attributions respond strongly to artifact availability. Observational detectors based on direct skill-use claims, text mentions, trace similarity, and an LLM judge do not identify which decisions actually depend on the skill. In multi-
agent systems, skill influence can survive communication even after its source is lost, while no-skill teams still name skills and sources that were never supplied. These findings establish the
Reasoning Backroom as a general AI provenance problem whose audit requires intervention.
Read the full abstract ▾
All authors Jinwei Hu, Yi Qi, Xinmiao Huang, Youcheng Sun, Yi Dong, Xiaowei Huang
Artificial Intelligence · Cross-listed from cs.LG
Latent States in Neural Networks: Recovering the Temporal Structure of Drifting Data from Model Weights
Kevin Guan
A temporally drifting data stream may pass through discrete regimes rather than changing continuously. We ask whether such regimes are recoverable from the weights of models trained on the stream, using a hidden Markov model (HMM) fit to the chronologically ordered trajectory of those weights. We study this question in two domains known to drift over time:
multimodal misinformation detection, using the Fakeddit
dataset; and sentiment analysis, using the Yelp
dataset. We train classifiers on consecutive temporal windows and fit an HMM to the trajectory of their aligned weights, recovering latent states that partition each timeline into coherent phases. On both
datasets, classifiers generalize better to data from windows sharing the state of their training window than to windows across state boundaries. This within-state transfer advantage survives a control for temporal proximity and modestly exceeds the advantage recovered by a naive partition into contiguous states of equal size. Although the states are estimated solely from model weights, they correlate more strongly with shifts in the data's class distribution than with the weight-space geometry used to estimate them. After class divergence and lag are residualized out, the within-state advantage exceeds its permutation null on both tasks, indicating that the states recover structure relevant to transfer beyond the data distribution. Every effect replicates on both tasks but is attenuated on Yelp, whose label distribution is more temporally stable.
Read the full abstract ▾
Artificial Intelligence
VAmoS Bench: Voice Agent Simulation Bench
Joshua Meyer, Sahar Shayegan, Ritiz Tambi et al. (+5 more)
Production voice
agents span cascaded,
speech-to-
speech, and hybrid architectures. Voice-
agent benchmarks typically measure component quality and conversational properties such as word error rate, latency, naturalness, and turn-taking. Fewer measure whether the
agent handled a phone call correctly on its own. Contact centers refer to this as “containment”: the share of phone calls the automated system resolves without handing off to a human. On some phone calls the right outcome is refusal or a redirect. To address this gap, we introduce VAmoS Bench, the Voice
Agent Simulation Bench. It measures complete voice-
agent systems end to end in a stateful customer-support task. The
agent is Riley, a credit-card support representative for a fictional bank who can freeze, cancel, replace, or activate a card. Each of 100 scenarios supplies a simulated caller with a private goal and a seeded PostgreSQL backend. The platform uses each scenario to populate and activate an isolated simulation in which the caller reaches Riley over audio; roughly one-third apply
adversarial pressure. The
agent can use five tools that execute real SQL against the backend. Each scenario also defines binary assertions. A grader evaluates them against the complete trace of what the caller and
agent said and what the
agent did, including tool invocations, arguments, and returned rows. This catches an
agent that claims to have changed a card without updating the database, as well as one that makes the right database change while disclosing protected information. This first
benchmark version focuses on financial services. Its evaluation protocol supports an evolving leaderboard: additional voice
agents can be evaluated on the same version, while later versions can expand the tasks and scenarios.
Read the full abstract ▾
All authors Joshua Meyer, Sahar Shayegan, Ritiz Tambi, Ali Khan, Sun Kim, Victor Shih, Mehdi Jamei, Andi Partovi
Artificial Intelligence
Leveraging Trajectory Graphs for Pre-Execution Error Diagnosis in Agentic LLM Systems
Xu Zheng, Zhuomin Chen, Chaohao Lin et al. (+4 more)
Large
Language Model~(LLM)-based
agents have demonstrated exceptional performance across a wide range of complex interactive tasks. However, they often struggle with long-horizon interactive tasks common in domains, such as
embodied AI. The complexity and vast action spaces in these settings lead to compounding errors, where a single suboptimal action can derail an entire trajectory, causing the
agent to exhaust its limited step budget on inefficient or unrecoverable paths. To overcome this without costly fine-tuning, we draw inspiration from software debugging, where execution logs are analyzed to preemptively catch errors. We propose Trajectory Graph Copilot, a novel framework that acts as a “copilot” for LLM
agents by diagnosing potential action errors before they are executed. At its core,Graph Debugger models historical trajectories as a probabilistic graph and uses a Graph Neural Network to identify sequential action patterns that frequently lead to failure. Functioning as a proactive diagnostic sandbox, our method provides early warnings on potentially flawed actions, prompting the
agent to self-correct. This pre-action error diagnosis prevents costly mistakes, significantly enhancing the
agent's ability to complete long-horizon tasks successfully. The extensive experiments on four
benchmarks with three LLM
agents demonstrate a $14.69\%$ pass ratio improvement on average.
Read the full abstract ▾
All authors Xu Zheng, Zhuomin Chen, Chaohao Lin, Hua Wei, Haifeng Chen, Wei Cheng, Dongsheng Luo
Artificial Intelligence · Cross-listed from cs.LG
SE(3)-MeanFlow: Few-Step Protein Backbone Generation on Lie Groups
Yikun Bai, Binghang Lu, Yikai Liu et al. (+7 more)
Generative modeling of
protein backbones promises the de novo design of
proteins with prescribed structural and functional properties. Existing
diffusion and flow-matching models produce high-quality backbones on SE(3)^N, but inference requires numerically integrating an ODE over hundreds of network evaluations, each involving a Lie group exponential map - a bottleneck for high-throughput design campaigns. We introduce SE(3)-MeanFlow, a few-step generative framework that extends MeanFlow from Euclidean space to the Lie group geometry of
protein frames. Working natively in the Lie
algebra so(3) and in R^3, we derive closed-form average-velocity identities for rotations and translations, giving simulation-free training targets. We further introduce an SE(3) alpha-Flow objective that removes the Jacobian-vector product from the rotation branch and serves as a warm-up stage, after which training switches to a small-t stabilized MeanFlow loss that is used for the remainder of pretraining and for rectification-based post-training. In
protein backbone generation, SE(3)-MeanFlow matches or exceeds flow-matching baselines that use several times more sampling steps, and its advantage widens in the few-step regime, where rectification lets it lead at every matched budget - at a modest cost in diversity.
Read the full abstract ▾
All authors Yikun Bai, Binghang Lu, Yikai Liu, Elaheh Akbari, Soheil Kolouri, Linxuan Wang, Ping He, Shuchan Wang, Ruqi Zhang, Guang Lin
Artificial Intelligence · Cross-listed from physics.med-ph
Rethinking AI in Medical Imaging: Assumptions, Reality, and Reframing; Towards Physician-Centered Agentic Reasoning
Arman Rahmim, Nourhan Bayasi, Xiaoxiao Li et al. (+2 more)
Medical imaging has served as primary proving ground for clinical AI, yet a decade of intense research has not translated into proportionate bedside impact. We argue that this gap is not primarily a product of insufficient algorithmic performance, inadequate regulation, or limited explainability. Rather, it reflects a structural misalignment, between how AI systems are designed and evaluated, and how clinical decisions are made. This work identifies 8 interconnected dimensions of this misalignment: 1) the dominance of pixel-only models in a
multimodal clinical world; 2) the erosion of physician trust through opaque and inflexible systems; 3) the unfulfilled promise of foundation models in data-sparse medical domains; 4) the persistent bottleneck of non-shareable under-curated
datasets; 5) the gap between validated
algorithms and deployable clinical platforms; 6) the slow, unmonitored translation of evidence into routine practice; 7) the risk of AI displacing rather than extending physician judgment; and 8) the failure of prediction-centric AI to generate actionable, agentic clinical guidance. For each dimension, we reframe the problem and propose a path forward, culminating in a
vision of physician-centered AI solutions with agentic
reasoning that extend, rather than replace, clinical judgment.
Read the full abstract ▾
All authors Arman Rahmim, Nourhan Bayasi, Xiaoxiao Li, Babak Saboury, Fereshteh Yousefirizi
Artificial Intelligence · Cross-listed from cs.CL
Selecting Open-Weight Language Models for Zero-Shot Intent Classification: A Systematic Evaluation of 41 Models
Parishruthi Ganesh, Gerry Dozier and Cheryl Seals
Intent classification is a core component of task-oriented dialogue systems, yet practitioners have limited systematic guidance for selecting deployable open-weight
language models under compute, latency, and robustness constraints. We present a systematic zero-shot evaluation of 41 open-weight
language models spanning 15 families and the 135M–9B parameter range across eight English single-label intent-classification
datasets. A ninth
dataset, ATIS, uses five labeled demonstrations and is reported as an auxiliary five-shot result. The evaluation includes standard
benchmarks, a large-scale voice-assistant corpus, and production-derived e-commerce
datasets. Beyond exact-match accuracy, we analyze confidence calibration, robustness to realistic input perturbations, statistical reliability of model rankings, deployment efficiency, and
benchmark saturation. Our results show that instruction-tuned 3B models can outperform several evaluated 7B base models, that differences among leading models on MASSIVE are statistically indistinguishable under pairwise McNemar tests, and that widely used
benchmarks such as SNIPS have become saturated and no longer meaningfully discriminate among current open-weight models. Instruction tuning's effect on confidence calibration is inconsistent rather than uniformly harmful. These findings provide practical guidance for selecting and evaluating open-weight
language models for intent classification.
Read the full abstract ▾
Artificial Intelligence
Dimensionality and Measurement Precision in HLE's Multiple-Choice Subset
Mayank Sharma, Savira Nadela and Tyler Matteson
Humanity's Last Exam (HLE) is widely used to evaluate frontier
language models. HLE organizes its questions into eight subject-domain categories, whose subscores are often interpreted as evidence of distinct capabilities. However, no study has assessed whether these labels correspond to empirically separable latent constructs, nor whether the
benchmark effectively differentiates between models of similar ability. We evaluate 29 LLMs on the text-only multiple-choice subset of HLE and apply psychometric methods to assess both the dimensionality of the
benchmark and the distribution of its measurement precision. Fitting a two-parameter logistic IRT model, we find that HLE is dominated by a single general
reasoning factor: domain labels explain only 3.5% of variance in the leading principal components of item responses, and within- and between-domain residual correlations are nearly identical (Cohen's $d = 0.016$). A simulation study with the same number of models ($N = 29$) shows that our analyses would detect clearly distinct domain abilities, so their absence is informative; however, small domain-specific differences cannot be ruled out, and model rankings vary across domains somewhat more than a single factor predicts. A separate analysis of the test information function reveals that measurement precision concentrates at moderate ability levels and drops sharply above $\theta = 0$, where the strongest models sit. These findings suggest that the subset's domain subscores do not warrant distinct capability interpretations and that its ability to discriminate among the strongest models might be limited.
Read the full abstract ▾
Artificial Intelligence
Bridging Inference-Time Scaling and Episodic Memory with Action-Centric Graphs
Xu Zheng, Chaohao Lin, Zhuomin Chen et al. (+4 more)
Recent advancements in inference-time scaling have significantly unlocked the complex
reasoning capabilities of Large
Language Models~(LLMs). However, for
agents, these approaches suffer from a critical inefficiency, operating in a stateless manner and engaging in redundant search processes. Existing
memory mechanisms largely rely on the
reasoning capabilities of LLMs, leading to prohibitive computational costs. In this paper, we propose a novel framework, GAMER~(Graph-based Action-centric
Memory with Episodic
Reasoning), that bridges the gap between inference scaling and episodic
memory. Our approach models historical
reasoning as a dynamic Action-Centric Graph. By decoupling the
memory mechanism from LLMs, our method can save
token/money usage by providing less
memory context than
memory mechanism baselines. To extract knowledge from the graph effectively, we use a dual-stream Temporal Difference learning mechanism to estimate the positive~(suggestion) and negative~(avoidance) value of action nodes based on past successes and failures. During the inference phase, this learned value function optimizes decision-making bi-directionally, so that positive values provide action suggestions, while negative values indicate high-risk actions. By performing efficient searches on the graph, our method significantly improves the efficiency of inference scaling. Experiments on multiple
benchmarks demonstrate that GAMER achieves superior performance by 20.81%/6.17% for success/progress rate compared to vanilla baselines.
Read the full abstract ▾
All authors Xu Zheng, Chaohao Lin, Zhuomin Chen, Weijieying Ren, Haifeng Chen, Wei Cheng, Dongsheng Luo
Artificial Intelligence · Cross-listed from cs.SE
SWE-NFI: Studying and Benchmarking Coding Agents for Non-Functional Improvements
Pengyu Xue, He Yang Yuan, Xin Wang et al. (+6 more)
Although coding
agents have achieved impressive performance on correctness-oriented
benchmarks, their ability to make behavior-preserving non-functional improvements (NFIs) remains underexplored. In real-world software development, developers continuously improve software quality without changing observable behavior, yet existing
benchmarks primarily evaluate functional correctness and provide limited support for assessing these non-functional improvements. In this paper, we present SWE-NFI, a
benchmark for evaluating coding
agents on NFIs beyond functional correctness. SWE-NFI contains 188 tasks constructed from real merged pull requests in open-source Python projects. We operationalize five developer-oriented NFI aspects into 92 executable rules and develop an evaluation suite that first applies task-specific functional preservation checks to assess whether key properties of the original code are preserved and then performs rule-based NFI evaluation. For outputs that pass these checks, we measure NFI improvement as the change in the normalized rule score for the task's target aspect relative to the original code. We evaluate eight commercial and open-source
agent configurations and run each configuration five times on every task. At most 70.0% of valid
agent outputs satisfy all task-specific functional preservation checks. Among outputs that pass these checks, NFI performance varies substantially across aspects. In particular, all evaluated configurations achieve mean Logic Patterns improvements between 0.0 and 1.3, below the Human Reference mean of 1.5. Our
benchmark and findings provide a reproducible foundation for evaluating coding
agents on non-functional improvements and identifying the NFI aspects where further advances are needed.
Read the full abstract ▾
All authors Pengyu Xue, He Yang Yuan, Xin Wang, Junkai Chen, Haonan Zhang, Boyuan Chen, Zishuo Ding, Zhenhao Li, Weiyi Shang
Artificial Intelligence · Cross-listed from cs.CL
Benchmarking LLM Competence on Logical Inference over Probability Operators
Nayera Hasan, Jack Greff and Alvin Grissom II
Both expressions of uncertainty and inferences are ubiquitous in natural
language, and valid inferences over natural-
language expressions of uncertainty are necessary for not only everyday conversations but also for high-stakes domains such as medicine and law. While large
language models are increasingly evaluated on logical
reasoning tasks, disentangling principled, symbolic
reasoning from clever surface-level pattern matching is fraught with difficulty. We introduce a
benchmark for
reasoning over probability operators–inference over sentences with gradable epistemic modals (e.g., probably, might, must) containing 14,320 procedurally-generated English
prompts across fifteen inference templates, systematically varying question form, negation strategy, and surface content. Evaluating 29 models, we find that most show answer biases independent of the logical form, a systematic preference for Yes or No. We summarize this with a competence floor: the worse of a model's accuracy on Yes-correct and No-correct items. Only 9 of 29 models exceed random chance. We also test variations in question form, verb phrases/activity, and both the gender and origin of names used in the
prompts, finding biases across every axis.
Read the full abstract ▾
Artificial Intelligence · Cross-listed from cs.LG
ECG-InterpBench: Benchmarking the Interpretability of ECG Foundation Models with Matched-Scale Sparse Autoencoders
Yixuan Duan and Wei Qiu
Existing
benchmarks for electrocardiogram foundation models primarily evaluate downstream predictive performance, providing limited insight into whether their internal representations can be faithfully decomposed, clinically interpreted, or reproduced across independent analyses. We introduce ECG-InterpBench, a
benchmark designed to systematically evaluate the interpretability of ECG foundation-model representations. ECG-InterpBench uses sparse autoencoders as standardized measurement instruments and matches their capacity across models to enable controlled comparisons. We evaluate six frozen ECG foundation models across five standardized encoder depths, five matched dictionary widths, and three random seeds, producing a 450-
cell interpretability atlas comprising 75 exactly matched six-model comparison blocks. The
benchmark evaluates complementary dimensions of representation interpretability, including sparse reconstruction fidelity, single-feature accessibility and coverage of 49 clinically meaningful ECG measurements, and cross-seed feature reproducibility. The evaluation further quantifies patient-sampling uncertainty, depth- and seed-dependent variation, and sensitivity to the sparsity parameterization. The
benchmark reveals that ECG foundation models exhibit distinct interpretability profiles. A matched replication on MIMIC-IV-ECG confirms that reconstruction fidelity and clinical accessibility identify different leading models. The
benchmark is accompanied by executable evaluation code, standardized manifests,
cell-level metrics, and reproducibility audits. ECG-InterpBench complements performance-centered ECG
benchmarks by providing a capacity-controlled and reproducible framework for comparing ECG foundation models across distinct dimensions of representation interpretability.
Read the full abstract ▾
Artificial Intelligence · Cross-listed from cs.CL
AHA-Memes: A Fine-Grained Multimodal Benchmark for Understanding Hate in Arabic Memes
Mohamed Bayan Kmainasi, Ali Ezzat Shahroor, Abul Hasnat et al. (+3 more)
Hateful memes are a growing form of
multimodal online harm, where hostile intent is often conveyed through the joint interpretation of images, text, cultural references, and implicit targets. While hateful meme detection has advanced in high-resource
languages, Arabic remains underexplored, with existing meme resources focusing mainly on propaganda or coarse harmful-content labels. We introduce AHA-Memes (Arabic HAteful Memes), which is, to our knowledge, the first large-scale Arabic hateful meme
benchmark with fine-grained, multi-label annotations. The
dataset includes 5K manually annotated memes using a taxonomy that captures hate types, i.e., attack strategies. We further provide ~66K silver-labeled memes to support future studies. We
benchmark text-only, image-only, and late-fusion
multimodal models, as well as few-shot in-context learning (ICL) and open- and closed-weight
Vision-
Language Models (VLMs) under zero-shot and fine-tuning settings. Our results establish strong baselines and highlight key challenges in culturally grounded Arabic hateful meme detection. We release the
dataset, annotation guidelines, and evaluation scripts to support future research. WARNING: This paper contains examples that may be disturbing to readers.
Read the full abstract ▾
All authors Mohamed Bayan Kmainasi, Ali Ezzat Shahroor, Abul Hasnat, Md. Rafiul Biswas, Wajdi Zaghouani, Firoj Alam