Artificial Intelligence
Graph-Structured Rubrics: Compiling Rubrics into Typed Evaluation Graphs for LLM Judges
Xi Chen, Jie Mu, Mo Xuan et al. (+1 more)
Rubric-based evaluators commonly treat rubrics as
prompt context or flat criteria: they specify what to judge but leave criterion composition implicit, even when natural-
language rules state it. We introduce Graph-Structured Rubrics (GSR), which compiles a rubric into a response-independent typed evaluation graph before observing responses. Criterion nodes elicit judgments; transformation, reduction, and gating operators compose them through named ports; and a task-specific output mapping, termed Readout, converts the unique sink into a score or preference. Compilation rejects malformed or type-incompatible graphs. Pointwise evaluation judges rubric dimensions separately before graph aggregation; pairwise evaluation reuses the graph with one judgment for each candidate under every criterion. Under GPT-OSS-120B, GSR improves exact score agreement by 0.62–6.75 percentage points over Prometheus-style scoring on four pointwise
datasets and achieves the numerically highest end-to-end pairwise accuracy on two preference
benchmarks under native tie and abstention policies.
Read the full abstract ▾
All authors Xi Chen, Jie Mu, Mo Xuan, Qun Shao
Artificial Intelligence · Cross-listed from cs.LG
Faithful, Sufficient and Understandable: Rethinking Graph Counterfactual Explanations via Discrete Diffusion Inversion
David Bechtoldt and Sidney Bender
Graph Neural Networks (GNNs) achieve strong predictive performance on graph-structured data across domains such as chemistry, biology, and network analysis, yet they provide no intrinsic explanation of their predictions. This limits their adoption in high-stakes and safety-critical settings. Counterfactual explanations address this by revealing the minimal structural modifications that would change a model's prediction. On graphs, however, such a modification is hard to produce. The search space is discrete and combinatorial, and a valid answer must respect categorical node and edge types together with domain rules such as chemical valency in the case of molecular graphs. Existing explainers give up one of two things. Either edits are not held on the data
manifold, or the search does not span the full edit space. We propose Graph
Diffusion Counterfactual Explanation via Inversion (GDCE-I), which gives up neither. A discrete denoising
diffusion model with a novel discrete inversion scheme enables distribution-aware edits leveraging the whole domain edit space. We further address the incomplete and inconsistent evaluation of graph counterfactuals by deriving a framework of explanation desiderata and applying it to every method under one shared protocol. Across four
benchmarks, GDCE-I outperforms related work by a large margin on the defined framework. For the molecular domain, we further qualitatively show that GDCE-I attains interpretable in-distribution solutions.
Read the full abstract ▾
Artificial Intelligence · Cross-listed from cs.CV
Better Slots, Better Worlds: Representation Quality & Robustness in Object-Centric World Models
Shukrullo Nazirjonov, Sai Prasanna, Anna Manasyan et al. (+1 more)
Learning world models from offline trajectories enables
agents to accomplish different tasks through planning. Object-centric (OC) representations, which decompose a scene into a set of slots that bind to its objects, have been proposed as an inductive bias for world models that are more sample-efficient and generalize better. Yet prior object-centric world models (OCWMs) take the slot encoder as given and evaluate only in-distribution, leaving open whether the object-centric bias actually delivers for planning and what within the OCWM drives it. We conduct a controlled study of OCWMs for visual model-predictive control along two axes: object-centric representation quality and generalization under distribution shift relative to scene-centric models. We find that (i) planning success correlates positively with unsupervised slot-quality metrics (FG-ARI, mBO), though the gains saturate at high slot quality; (ii) with well-bound slots, the auxiliary proprioception inputs and masking inductive bias that prior methods relied on become unnecessary; and (iii) under unseen distribution shifts, the OCWM with well-bound slots is more robust overall than the end-to-end trained scene-centric LeWM, while DINO-WM, built on similar frozen pretrained features, remains competitive – pointing to pretrained features as a key contributor to robustness.
Read the full abstract ▾
All authors Shukrullo Nazirjonov, Sai Prasanna, Anna Manasyan, Georg Martius
Artificial Intelligence · Cross-listed from cs.RO
Learning Loco-Manipulation From SMPC Demonstrations With Sparse Offline-to-Online RL
Martin Schuck, Maks Sorokin, Simone Manni et al. (+5 more)
Integrating locomotion and manipulation is essential for
robot autonomy, but scaling standard Reinforcement Learning (RL) to complex tasks is severely bottlenecked by the slow, manual process of dense reward shaping. To bypass this limitation, we leverage Sample-based Model Predictive Control (SMPC) entirely in simulation as an automated, rapidly tunable expert to generate massive offline
datasets. Because this data solves the fundamental exploration problem, we can train an off-policy RL
agent using purely sparse task rewards, drastically reducing the time required to learn new skills and eliminating the need for manual tuning. Integrating this high-level
agent with a low-level dynamic stability controller yields more optimal behaviors that strictly align with true task objectives, ultimately allowing the learned policies to surpass the original optimal control teacher. We validate the robustness of this sim-to-real framework by successfully deploying complex loco-manipulation skills across different morphologies, including an arm-equipped Spot quadruped and a G1
humanoid.
Read the full abstract ▾
All authors Martin Schuck, Maks Sorokin, Simone Manni, Duy Ta, Angela P. Schoellig, Marco Hutter, Simon Le Cleac'H, Jan Brüdigam
Artificial Intelligence · Cross-listed from cs.CL
Preference Tree Optimization: Enhancing Goal-Oriented Dialogue with Look-Ahead Simulations
Lior Baruch, Moshe Butman, Kfir Bar et al. (+1 more)
Developing dialogue systems capable of engaging in multi-turn, goal-oriented conversations remains a significant challenge, especially in specialized domains with limited data. This research proposes a novel framework called Preference Tree Optimization (PTO), designed to iteratively improve
agent models in such dialogue systems, by generating preference data using a method called Preference Tree with Look-Ahead. Focusing on Motivational Interviewing (MI) – a counseling technique aimed at facilitating behavioral change – we leverage virtual patients and an oracle evaluator to simulate conversations and generate rich preference
datasets. By combining this method with Direct Preference Optimization (DPO), we aim to enhance the
agent's decision-making capabilities over iterative training cycles. The proposed framework addresses data scarcity and advances the development of more nuanced and effective dialogue systems in goal-oriented domains. Experimental evaluations demonstrate that the PTO framework enhances dialogue
agents' performance in goal-oriented conversations within the domain of Motivational Interviewing (MI). Models trained with PTO consistently outperformed the baseline in key metrics such as session satisfaction and working alliance. Additionally, incorporating look-ahead simulations led to improved long-term planning and more effective conversational strategies, with deeper look-ahead configurations yielding the most stable and high-scoring results.
Read the full abstract ▾
All authors Lior Baruch, Moshe Butman, Kfir Bar, Doron Friedman
Artificial Intelligence
Mechanist: AI as a Scientific Instrument for Discovering the Mechanisms of Intelligence
Mengru Wang, Junfeng Fang, Shuofei Qiao et al. (+17 more)
AI models are increasingly used in scientific discovery and human decision-making. Yet how AI models work and what risks they pose remain poorly understood. As AI development becomes faster and more automated, research on the mechanisms underlying AI remains largely manual. To bridge this gap, we introduce Mechanist, an agentic system that uses AI as a scientific instrument for the autonomous discovery of mechanisms underlying AI. To ground novel mechanism hypotheses, we construct a scientific knowledge graph of 13,000 studies on AI mechanisms, alongside a multidisciplinary database of 43 million papers spanning 26 fields. For reliable experiment execution, we curate a library of 32 foundational methods for mechanism analysis. Compared with Claude Code and existing AI-scientist systems, Mechanist generates higher-quality mechanism hypotheses and executes experiments more reliably. Across four case studies, Mechanist autonomously discovers new model behaviors and their underlying mechanisms, and translates these discoveries into mechanism-guided interventions and interdisciplinary design. Specifically, Mechanist first uncovers a counterintuitive safety risk in scientific laboratories, showing that unsafe traits can transfer to fine-tuned student models through apparently safe training data and emerge across modalities. Mechanist then develops a mechanism theory of belief, revealing how models represent world knowledge, form beliefs, infer the beliefs of others, and how these mechanisms emerge during pretraining. Building on this theory, Mechanist develops targeted interventions that improve model performance across diverse scenarios. Finally, Mechanist can also advance interdisciplinary discovery through mechanistic design, providing an alternative to the computationally intensive generate-and-rerank paradigm.
Read the full abstract ▾
All authors Mengru Wang, Junfeng Fang, Shuofei Qiao, Zhenqian Xu, Haoming Xu, Haoxiong Wang, Shumin Deng, Linyi Yang, Xin Xu, Yunzhi Yao, Dan Zhang, Fei Shen, Zhixiang Cui, Buqiang Xu, Haozhe Luo, Yunxiang Wei, Ningyu Zhang, Julian McAuley, Tat Seng Chua, Huajun Chen
Artificial Intelligence · Cross-listed from cs.CV
How Far from Clinical Deployment? Evaluating the Complete Unsupervised Domain Adaptation Pipeline in Medical Imaging
Yiheng Xiong, Luisa Gallée, Daniel Santak Wolf et al. (+2 more)
Deploying unsupervised domain adaptation (UDA) in clinical practice requires choosing which
algorithm to use and which of its trained models to ship. However, the deployment (target) domain is unlabeled, so models cannot be evaluated directly on it, leaving it unclear which to select. We address this by evaluating the complete UDA pipeline, considering both adaptation and label-free selection together. Our study covers eleven clinically relevant cross-domain scenarios from nine medical imaging
datasets, with ten UDA
algorithms and 13 label-free selection methods (validators), evaluating over 80,000 trained models in total. By this, we find that a capable adapted model usually exists, but identifying it without target labels is difficult: the validator-selected models leave a large and structural target performance gap to the best available one, with no evaluated validator consistently reliable. Towards closing it, we explore two strategies, ensembling and a small target-labeling budget; both narrow this gap but do not close it entirely. Overall, deployable UDA depends on the complete pipeline; addressing the less explored selection step could bring much of current UDA closer to clinical use.
Read the full abstract ▾
All authors Yiheng Xiong, Luisa Gallée, Daniel Santak Wolf, Heiko Hillenhagen, Michael Götz
Artificial Intelligence · Cross-listed from cs.CV
LoSA: Near-Lossless Sparse Attention for Training-Free Video Diffusion Acceleration
Enhuai Liu, Yunke Wang, Yutong Wang et al. (+2 more)
Video diffusion transformers are costly to sample: every denoising step applies self-
attention over a long 3D
token sequence, a quadratic cost that dominates as resolution and duration grow. Sparse
attention reduces this cost without retraining, but existing methods pursue aggressive sparsity, where further speedup costs disproportionately more
attention fidelity. We target the opposite end of this trade-off: fix near-lossless fidelity by construction, and remove as much computation as this constraint permits. Two observations make this regime practical: roughly 40% of block interactions can be removed while retaining 99% of the
attention mass, and the high-mass support remains stable across denoising steps. We propose LoSA, a training-free sparse-
attention method that fixes a retained-mass threshold of 99% rather than a sparsity ratio: it measures exact block
attention masses at one early dense step, keeps, for each head and query block, the smallest key/value block set meeting the threshold, and reuses the frozen block indices for all remaining steps. On Wan2.1-1.3B, LoSA alone gives a $1.36\times$ speedup with a 0.06-point VBench Overall drop. The benefit is largest under composition: combined with feature caching, LoSA reaches a $3.2\times$ speedup on HunyuanVideo at a 0.02-point drop, versus 0.32 points for the strongest sparse baseline at comparable speed. Across three
video diffusion transformers and speedups up to $3.2\times$, LoSA consistently achieves the best training-free speed-quality trade-off.
Read the full abstract ▾
All authors Enhuai Liu, Yunke Wang, Yutong Wang, Changming Sun, Chang Xu
Artificial Intelligence · Cross-listed from cs.LG
Uncertainty-Aware Probabilistic Constrained Clustering from Entangled Pairwise Supervision
Shaojie Zhang and Ke Chen
Pairwise constrained clustering typically relies on hard must-link/cannot-link labels, whereas realistic pairwise supervision may be real-valued and entangle intrinsic ambiguity, expert judgment, and
stochastic corruption. Existing deep constrained clustering (DCC) methods mainly target hard, expert-agnostic constraints, treating soft labels mostly numerically rather than semantically. We formalize this setting as uncertainty-aware probabilistic constrained clustering (UPCC), defining a canonical aleatoric relation through a heterogeneous observation process and analyzing its conditional identifiability. We introduce ProbPair, an angular pairwise objective for probabilistic relations, and build ECI-PP, an
estimator–corrector–integrator framework that refines imperfect supervision via belief estimation, correction, and reliability-aware integration. Across challenging probabilistic supervision settings, experiments on diverse
benchmarks show that ECI-PP outperforms state-of-the-art DCC methods and remains robust with a shared default configuration.
Read the full abstract ▾
Artificial Intelligence · Cross-listed from cs.SE
From Safety Documentation to Safety Knowledge Support: An Evidence-Grounded LLM Framework for Medical Devices
Tuhinangshu Gangopadhyay, Rasmus Adler, Peter Liggesmeyer et al. (+1 more)
Medical devices are becoming more software-intensive, connected, and AI-enabled. Their development requires risk-management evidence aligned with ISO 14971 and, for software, IEC 62304. This evidence must be kept consistent across requirements, design decisions, software changes, verification results, complaints, and post-
market data. These tasks are costly and depend on scarce safety and domain experts. Large
language models (LLMs) may reduce parts of this effort because medical-device safety work is highly document-based. However, current LLM-based safety-engineering studies often address isolated methods, rely on generic prompting or public examples, and provide limited support for source links, traceability, uncertainty handling, lifecycle updates, and recorded expert review. This limits their use in regulated medical-device development. This paper argues that the central research problem is not safety-text generation, but source-linked safety-knowledge support. We propose an evidence-grounded framework that connects device artifacts, controlled knowledge storage and
retrieval, method-specific generation of candidate safety items, critique and uncertainty checks, and recorded expert review. The framework prepares, links, checks, and updates candidate safety artifacts for expert decision-making. It does not decide whether a device is safe and does not provide regulatory approval. We also outline an evaluation strategy using non-public or newly built medical-device case studies and expert reference analyses to assess coverage, correctness, relevance, traceability, duplicate rate, unsupported claims, and review effort.
Read the full abstract ▾
All authors Tuhinangshu Gangopadhyay, Rasmus Adler, Peter Liggesmeyer, Jan Reich
Artificial Intelligence · Cross-listed from cs.LG
Dual-Model Sentiment Analysis of Consumer Reviews in the Retail Coffee Sector Using Machine Learning and Deep Learning Approaches
Muntasir Hasan Kanchan, Md. Alamgir Hossain, Md. Samiul Islam et al. (+1 more)
Consumer reviews play an important role in shaping brand perception and business strategies, particularly in service-driven industries such as retail coffee. This study presents a comparative sentiment analysis framework for Starbucks customer reviews using classical machine learning and deep learning approaches. The
dataset, collected from ConsumerAffairs, contains more than 700 reviews and was analyzed through preprocessing and exploratory data analysis to identify temporal and geographic patterns. Sentiment labels were generated by binarizing
star ratings, with ratings of 4 and 5 classified as positive and ratings of 1 to 3 as negative. The resulting
dataset was substantially imbalanced toward negative sentiment. Five machine learning classifiers, including Logistic Regression, Support Vector Machine (SVM), Decision Tree, Random Forest, and Naive Bayes, were evaluated alongside five deep learning models: LSTM, RNN, Bidirectional LSTM, GRU, and CNN. Model performance was assessed using accuracy, precision, recall, and F1-score. SVM achieved the highest accuracy among the machine learning models at 91.0 percent, while Bidirectional LSTM showed the strongest performance among the deep learning models and demonstrated good generalization on unseen data. The findings also show that class imbalance negatively affected positive sentiment recall across several models. Overall, this study provides a comparative evaluation of machine learning and deep learning approaches for real-world consumer sentiment analysis and highlights the importance of appropriate model selection and preprocessing for customer experience analytics in the retail coffee sector.
Read the full abstract ▾
All authors Muntasir Hasan Kanchan, Md. Alamgir Hossain, Md. Samiul Islam, Muhammad Masud Tarek
Artificial Intelligence · Cross-listed from cs.SE
RealisticTritonBench: A Benchmark for Triton-Kernel Generation in Real-World AI Frameworks
Jinjun Huang, Zhongzhen Wen, Tongtong Xu et al. (+3 more)
In modern AI frameworks, GPU kernels are key to overall system performance. Combining usability, portability, and near-handwritten CUDA performance, Triton is widely adopted for implementing GPU kernels. Recent advances show the potential of large
language models (LLMs) to automatically generate Triton kernels, reducing the manual effort required from expert kernel developers. Several
benchmarks evaluate LLM-generated Triton kernels. However, they suffer from three key limitations: (1) they restrict tasks to PyTorch-to-Triton translation, failing to reflect the diversity and complexity of real-world Triton tasks; (2) they evaluate only individual-kernel performance rather than end-to-end performance, the core criterion for real-world deployment in AI frameworks; and (3) they rely on manually written evaluation scripts for individual kernels, which may contain flaws that models can exploit to bypass correctness checks and obtain inflated scores. To address these limitations, we introduce RealisticTritonBench, the first
benchmark to derive Triton kernel generation tasks from real-world pull requests in popular AI frameworks, enabling realistic, production-like evaluation. RealisticTritonBench systematically extracts PRs that modify Triton kernels from popular open-source AI frameworks and transforms them into generation tasks with concrete engineering contexts. Each task takes a natural
language requirement as input and requires a corresponding Triton kernel implementation, with a complete and reproducible evaluation environment. Unlike prior
benchmarks focused on isolated kernel performance, RealisticTritonBench integrates generated kernels into their original frameworks and evaluates them using end-to-end tests, enabling a more faithful assessment. We evaluate leading LLMs on RealisticTritonBench and find that they still struggle with real-world Triton kernel generation tasks.
Read the full abstract ▾
All authors Jinjun Huang, Zhongzhen Wen, Tongtong Xu, Meng Yan, Xin Xia, Zhongxin Liu
Artificial Intelligence
CTBench: Evaluating Troubleshooting Capabilities of AI Agents in Realistic Telecom Network Operations
Xingyu Yan, Tingting Dai, Antonio De Domenico et al. (+16 more)
While autonomous
agents show promise for network operations, existing telecom troubleshooting evaluations fail to capture the partial observability and heterogeneity of real-world telecom environments. We introduce CTBench, a public
benchmark evaluating
agent troubleshooting capabilities in root cause analysis (RCA) and path restoration. Constructed by domain experts, CTBench features rich metadata and golden evidence trajectories, utilizing metrics that assess both final accuracy and diagnostic
reasoning. Our evaluation of state-of-the-art
agents reveals that while they excel at path endpoint identification, they severely underperform in RCA. Crucially, even when
agents produce correct answers, they frequently fail to gather the requisite diagnostic evidence demanded in operational practice. Finally, we demonstrate that increased computational expenditure does not guarantee improved diagnostic competence, highlighting critical gaps in current
agent capabilities.
Read the full abstract ▾
All authors Xingyu Yan, Tingting Dai, Antonio De Domenico, Mohamed Sana, Nicola Piovesan, Changchang Li, Bowen Liu, Kun Jiang, Mengjie Zhang, Dingcheng Shan, Jing-Cheng Pang, Chenwei Wu, Sijie Wu, Lianying Chao, Haoran Cai, Jiantao Ye, Xubin Li, Simon Mark Lucas, Xin Chen
Artificial Intelligence · Cross-listed from cs.LG
Remote Sensing and Machine Learning-Based Analysis of Land Use and Vegetation Change in Dhaka District, Bangladesh
Muhammad Masud Tarek, Md. Alamgir Hossain, Md. Samiul Islam et al. (+1 more)
Rapid urbanization in Dhaka District, Bangladesh has triggered substantial alterations in land use and environmental conditions, necessitating systematic monitoring for informed urban planning and ecological sustainability. This study employs remote sensing data and machine learning techniques to analyze spatiotemporal changes in land cover and vegetation dynamics between 2019 and 2024. High-resolution satellite imagery from Sentinel-2 MSI and Landsat 8 was utilized to classify land cover types and compute spectral indices including the Normalized Difference Vegetation Index (NDVI), Normalized Difference Built-up Index (NDBI), and Normalized Difference Water Index (NDWI). A supervised machine learning approach incorporating Decision Tree, K-Nearest Neighbors (KNN), and Random Forest classifiers was applied using labeled geospatial training points within Google Earth Engine. Accuracy assessments were conducted using confusion
matrices and kappa statistics. Results indicate a 59.5% increase in urban built-up areas and a significant decline in vegetation (-8.46%) and water bodies (-7.77%) over the five-year period. Land conversion from vegetated and aquatic areas to urban infrastructure was identified as a dominant trend. Among the models, Random Forest demonstrated the highest classification accuracy. These findings underscore the growing environmental pressures driven by unregulated urban expansion in Dhaka. The study highlights the potential of remote sensing and machine learning tools in providing timely, actionable data to support sustainable urban development, land-use regulation, and ecosystem conservation policies.
Read the full abstract ▾
All authors Muhammad Masud Tarek, Md. Alamgir Hossain, Md. Samiul Islam, Muntasir Hasan Kanchan
Artificial Intelligence
Claim-Level Reliability Assessment for Efficient Test-Time Reasoning
Sen Xu, Wei Wang, Shixi Liu et al. (+5 more)
We propose claim-level falsification as a principle for test-time scaling and instantiate it through Claim-Level Reliability Assessment (CLR), a training-free framework that reallocates test-time compute from additional solution sampling to targeted verification. Since whole-trace evaluation often obscures decisive errors due to signal dilution from routine
tokens, CLR condenses each
reasoning trace into a compact set of decision-critical claims, thereby isolating its logical anchors. Furthermore, recognizing the inherent difficulty of generating entirely correct solutions under fixed model capabilities, CLR shifts the focus to semantic falsification. This approach exploits a fundamental asymmetry between solution construction and claim refutation. Constructing a valid solution requires a flawless
reasoning path, whereas refuting an incorrect claim requires identifying only a single decisive flaw. This targeted search for negative evidence systematically compresses the survival space of high-confidence incorrect traces, effectively suppressing erroneous consensus via nonlinear reliability scoring. Across four LLMs and four
reasoning benchmarks under matched budgets, CLR generally improves upon pass@1 and self-consistency. On GPT-OSS-20B/CMIMC25, for instance, CLR exceeds pass@1 by 27.15 percentage-points and raises self-consistency accuracy from 77.50% to 82.19% with 37.0% fewer
tokens.
Read the full abstract ▾
All authors Sen Xu, Wei Wang, Shixi Liu, Jixin Min, Yingwei Dai, Zhibin Yin, Yirong Chen, Junlin Zhang
Artificial Intelligence · Cross-listed from cs.IR
Learning from Unreachable Rewards: Hint-Conditioned Reinforcement Learning for Generative Recommendation
Kangning Zhang, Haotian Fang, Xukun Luo et al. (+6 more)
Semantic-ID generative recommenders represent each item as a short sequence of discrete semantic
tokens and predict the next item by autoregressively generating this
token sequence. This paradigm enables a unified generation interface for item IDs, histories, and item text, but it also creates a structured optimization bottleneck during reward-based post-training: when an early semantic
token enters the wrong branch of the item-
token space, finite rollout groups rarely reach the ground-truth item, so group-relative optimization receives identical zero rewards and produces no useful advantage. We propose Hint-Conditioned Generative Recommendation (HCGRec), a semantic-ID generative recommendation framework that recovers learning signal for such hard training instances. HCGRec diagnoses each instance with checkpoint rollouts and supplies a minimal target-prefix hint only when the current generator cannot reach the correct item. The model then generates the unhinted suffix under the hinted semantic branch, turning zero-reward groups into informative comparisons over item-
token completions. Hinting also changes
token identity: hinted prefix
tokens are oracle-provided item context, while unhinted suffix
tokens are sampled generation actions. We therefore introduce hint-aware credit decomposition, using supervised learning to preserve item-semantic and prefix-structure
alignment for hinted
tokens and GRPO to optimize the sampled suffix. Experiments on sequential recommendation
benchmarks show that HCGRec substantially improves over supervised fine-tuning and vanilla reward-based post-training, while reducing zero-advantage training samples from over 70% to below 20%. The code is accessible at https://github.com/WncFht/GRec.
Read the full abstract ▾
All authors Kangning Zhang, Haotian Fang, Xukun Luo, Hao Yin, Yang Gao, Peng Yan, Weiwen Liu, Weinan Zhang, Yong Yu
Artificial Intelligence
Bench2Robust: Scenario-Controlled Recovery for Tool-Using Agents
Chaoran Chen, Vy Nguyen, Ziji Zhang et al. (+7 more)
Tool-using LLM
agents are typically developed and post-trained in controlled environments with reliable tools, but deployment exposes them to tool failures, API and schema drift, and third-party services outside the developer's control. We introduce Bench2Robust, a framework for scenario-controlled solvability that augments existing
benchmarks with episodes requiring retrying the original path, switching to a fallback-equivalent tool, or recognizing that no viable path remains. By controlling recoverability, Bench2Robust separates recovery-policy quality from randomness in whether failures persist or resolve. Across 7 models from 4 families and 10 task/domain slices spanning customer-service and function-calling
benchmarks, 69 of 70 model-subset pairs degrade under tool failures, by up to 46.7 percentage points. We propose
Bayesian Tool
Memory (BTM) for structured runtime recovery context and use reinforcement learning (RL) for learned recovery behavior. On held-out Retail tasks using the
benchmark's original tool set, BTM improves robustness by 16.8 percentage points without retraining, while RL retains a 6.3-point gain without inference-time BTM. Combining RL with BTM yields a 20.7-point improvement over the base model while preserving clean performance. These results show that robust tool use benefits from combining structured runtime recovery context with learned recovery behavior.
Read the full abstract ▾
All authors Chaoran Chen, Vy Nguyen, Ziji Zhang, Abhinav Gullapalli, Ziyi Wang, Yuxuan Lu, Dakuo Wang, Jing Huang, Zhou Yu, Jin Lai
Artificial Intelligence · Cross-listed from cs.LG
LoongReflect: Boosting Long-Horizon Reflection in Search Agents via Global Perspective Distillation
Zhixin Zhang, Xinke Jiang, Zhibang Yang et al. (+5 more)
Large
language model
agents increasingly rely on long-horizon
reasoning to solve complex tasks involving planning, tool use, and
memory. A critical capability in such settings is reflection: assessing trajectory progress, identifying missing evidence and unreliable intermediate states, and deciding whether to continue, revise, or abandon the current branch. Learning effective reflection, however, is challenging because reflection is performed locally within the current branch, whereas its utility can only be determined by its contribution to the final trajectory outcome. This local-global mismatch makes outcome-based reinforcement learning provide only local, sparse and delayed supervision for reflective decisions. To solve these, we propose LoongReflect, a training framework that formulates reflection as a
memory-control policy. The
agent operates over a reversible trajectory tree using explicit reflect and backtrack actions. Reflection consolidates verified facts, missing evidence, and branch-specific risks into working
memory, while backtracking removes an unreliable branch from the active context and preserves a concise corrective lesson. To learn this policy, LoongReflect combines two complementary signals through a look-ahead, extragradient-style coordination mechanism. A fast channel distills globally informed reflective behavior from a privileged teacher, with supervision restricted to reflection and backtracking
tokens. A slow channel optimizes complete trajectories using outcome-based GRPO, aligning local control decisions with final task success. Experiments on multi-hop
retrieval-augmented generation and mathematical
reasoning benchmarks demonstrate consistent improvements over outcome-only reinforcement learning and self-
distillation baselines.
Read the full abstract ▾
All authors Zhixin Zhang, Xinke Jiang, Zhibang Yang, Weixuan Xu, Guohong Qiu, Xu Chu, Junfeng Zhao, Yasha Wang
Artificial Intelligence · Cross-listed from stat.ME
Causal inference for group-contaminated structured outcomes: observable quotients, lossless reduction and exact randomization inference
Usef Faghihi and Amir Saki
Structured potential outcomes such as microscopy images may be recorded after an unknown, unit-specific transformation. If that transformation can depend on treatment,
covariates or the intrinsic outcome, raw-coordinate analyses may mix biological effects with acquisition geometry. We study the unrestricted observation model X = {\Gamma} . Y(A) and characterize its observable information: a target is uniformly recoverable exactly when it is constant on group orbits, while a Borel maximal invariant retains every measurable invariant target. We then distinguish observability from statistical losslessness. A quotient-faithful reconstruction
theorem shows that quotient reduction is sufficient for the full transformed experiment exactly when the conditional law of the raw observation given treatment,
covariates and the quotient has a parameter-free version. Conditional Haar contamination on a compact group yields Blackwell equivalence as a special case; it is not imposed in the main model. We also separate independent site-specific product actions from shared diagonal actions and show why componentwise canonicalization can discard relative cross-site information. Under explicit metric and kernel regularity, an approximate-contamination
theorem bounds quotient-law Wasserstein error and the induced perturbation of population maximum mean discrepancy. For finite-support multichannel
lattice images, we construct a maximal invariant under integer translations and quarter turns, combine its characteristic Gaussian kernel with a complete paired-swap test, and retain the original simulations and RxRx1 HUVEC study. Under the sharp null, the quotient test rejected in 0.052 of simulation replicates; at unit effect strength its power was 0.992. The primary RxRx1 contrast had an enumerated paired-swap p-value of 0.0078.
Read the full abstract ▾
Artificial Intelligence · Cross-listed from cs.LG
TailBooster: A Dual-Layer Generative Framework for Extreme Value Augmentation with Operational Validity Enforcement
Karim Aly, Alexei Sharpanskykh and Jacco Hoekstra
Extreme events in air transport, such as severe arrival delays and abnormal air times, cause cascading network disruptions with substantial operational, economic, and safety costs. Such events are rare in historical records, leaving insufficient training signal for machine learning models. Synthetic data augmentation offers a principled solution, but conventional generative models under-represent distributional tails and give no guarantee against operationally infeasible instances, such as a short air time paired with a long flight distance. No existing approach addresses both limitations for mixed-type tabular records. We propose TailBooster, a dual-layer generative framework combining generative modelling with two anomaly detection layers. A statistical layer extracts extremes via the interquartile range, supplying tail-concentrated training signal to dedicated generative models, here a Tabular Variational Autoencoder. A deep learning layer then applies autoencoder-based cleaning, discarding synthetic records that violate the operational envelope learned from historical data. The framework was evaluated on US flight records across five dimensions: diversity, statistical similarity, fidelity, operational validity, and utility, the latter two being the primary improvement targets. Data-driven cleaning markedly improved operational validity, while targeted augmentation enhanced utility for extreme-event prediction. Across six regression
algorithms, training on the framework's records reduced Mean Absolute Error by 47-49% on extreme air time and 29-57% on extreme arrival delay prediction relative to conventional synthetic data, with comparable gains when real records were enriched with synthetic extremes. Being fully data-driven and model-agnostic, TailBooster extends to domains where extreme-event prediction is critical and domain-specific rules are unavailable.
Read the full abstract ▾
Artificial Intelligence
ExRole: From Team Trajectories to Executable Roles in Multi-Agent Language Models
Zhou Liu, Chaoyang Han, Zewei Pan et al. (+2 more)
Roles provide an interpretable interface for organizing
language-model
agents, yet most multi-
agent systems treat them as hand-written
prompt labels disconnected from learned behavior and parameter updates. We argue that a useful role should instead be an executable control variable: it should summarize behavior predictive of future utility, guide subsequent interaction, and identify the trainable capacity responsible for that behavior. We introduce ExRole, a trajectory-to-role framework that learns future-aware role prototypes from prefix-local team traces, resolves them into readable instructions and
token-aligned role markers, and optionally routes shared LoRA rank slots with turn-aligned credit. Across MuSiQue and 2WikiMultiHopQA, ExRole improves over single-
agent search by 15.0/14.4 and 13.5/16.1 EM/F1 points, respectively. Against the strongest non-ExRole controls, the corresponding gains remain 11.5/11.6 and 7.7/9.7 points. Across both
benchmarks, the controlled results consistently favor trajectory-induced role conditioning over role-free, manual, random, and shuffled alternatives. Role-
Agent-Turn interventions further show that the induced roles capture transferable behavioral specialization beyond fixed
agent identities or turn positions.
Read the full abstract ▾
All authors Zhou Liu, Chaoyang Han, Zewei Pan, Zeli Su, Wentao Zhang
Artificial Intelligence · Cross-listed from cs.CL
A, B, C, or Stable? Accuracy and Order Sensitivity Diverge in LLM Multiple-Choice Evaluation
Karl Hanna and Chen Feng
Multiple-choice question (MCQ)
benchmarks are widely used to evaluate large
language models, yet a single reported accuracy number is not enough to distinguish a model's latent ability from order sensitivity. Debiasing methods tend to be judged on one axis, which implicitly assumes that fixing order sensitivity and improving accuracy go together. We test this assumption with two proprietary models and two open-weight models on MMLU and ARC-Challenge, across four families of strategies. We measure order sensitivity with flip rate, capturing per-question instability, and recall standard deviation (RStd), capturing aggregate positional imbalance. We find accuracy and order sensitivity do not move together consistently: two-stage prompting reduces both flip rate and RStd for GPT on MMLU without a significant change in accuracy. The two sensitivity metrics also diverge from each other. Adding
reasoning to direct answering leaves Llama's MMLU flip rate statistically indistinguishable while RStd falls sharply. A $2\times2$ ablation across generation-then-matching, crossing option visibility with matcher type, shows that withholding options mainly harms semantic matching, which achieves zero flip rate by construction, yet loses significantly in accuracy; the effect is smaller when using an LLM matcher. We recommend reporting flip rate and RStd along with accuracy when evaluating MCQ strategies.
Read the full abstract ▾
Artificial Intelligence
Tom Adamczewski
We construct OEIS Open, a
benchmark based on 492 open mathematical
conjectures from the OEIS, formalized in Lean by Tsoukalas et al. Whereas these
conjectures had previously been attempted only with a bespoke
agent, our open-source evaluation code runs any generic
language model (LM) against them, and is secure against LM cheating attempts. We find that LMs equipped with a minimal set of tools resolve 147 of these
conjectures with a budget of \$50 per attempt, scoring 30% on OEIS Open. OEIS Open Lite is a random subset of 100 conjectures for cheaper evaluation. When evaluated with a budget of \$200 per attempt, the best current LM scores 44% on OEIS Open Lite. Giving LMs access to the mathematics literature via 476,000 papers from arXiv did not increase performance on OEIS Open Lite, and nor did using more sophisticated
agent loops. The
conjectures covered in this work are of uncertain mathematical significance, and most have likely received little previous
attention. Nevertheless, our results show that LMs can resolve open research
conjectures autonomously and at modest cost.
Read the full abstract ▾
Artificial Intelligence · Cross-listed from quant-ph
Hamilton-Zero: A Neural Tensor-Network Foundation Model for Ground States of Arbitrary Quadratic Qubit Hamiltonians
Timothy Heightman, Elena Orlova, Philip Mantrov et al. (+1 more)
A central promise of useful
quantum advantage is the ability to compute ground states of
Hamiltonian systems beyond the reach of classical simulation methods. Here we demonstrate that this problem can be effectively amortized across an arbitrary and universal set of
Hamiltonians by a foundation model with $\sim0.5$B variational parameters, trained with contemporary techniques from large
language models and deep reinforcement learning. To do this, we formulate $\text{spin-}1/2$
quantum ground-state learning as
manifold variational optimisation over centrally odd scalar functions on $\mathrm{SU}(2)^N$. This replaces explicit
Hilbert-space vector amplitudes with
manifold functions on which the
Hamiltonian acts through Lie derivatives, evaluated by custom automatic differentiation primitives. We prove that the resulting variational principle on this
manifold preserves the $\text{spin-}1/2$ sector's ground-state upper bound using the Peter-Weyl
theorem and justify the choice of such a representation with a no-go
theorem for pure state foundation NQS. We then pre-train our foundation model on a
dataset of hundreds of thousands of different
Hamiltonian systems, varying the connection
topology, system size, interaction types and strengths, bringing together a century of many-body literature. Using a novel $\mathrm{SU}(2)$ replica-exchange Langevin sampler and sharded natural-gradient optimisation, we train our model with our own extension of the Kronecker-Factored Approximate
Curvature (KFAC) optimiser on system sizes up to 64
qubits. On a held-out generalisation
dataset, we fine-tune our model on system sizes of up to 1024
qubits, and evaluate on systems up to 8100
qubits.
Read the full abstract ▾
All authors Timothy Heightman, Elena Orlova, Philip Mantrov, Aleksei Ustimenko
Artificial Intelligence · Cross-listed from cs.CV
A Model-Internal Protocol for Assessing Multimodal Models as Integrated Systems
Hao Zhang, Jiaxin Qi, Zhijiang Tang et al. (+1 more)
As Large
Vision-
Language Models increasingly aim to integrate visual generation and understanding within a single parameter space, evaluating such structural unification in a cohesive manner remains a critical challenge. Current evaluation protocols predominantly treat generative and discriminative capabilities as separate tasks, leaving a gap in system-level evaluation for unified
multimodal models (UMMs). In this work, we propose Self-Generative-Understanding (SGU), a novel, annotation-free evaluation framework that probes the integrated capabilities of unified models through a semantic closed-loop challenge. Without requiring new annotations, SGU leverages the dual understanding-and-generation abilities of UMMs by asking them to first perceive an image and produce a textual description, subsequently reconstruct a visual context based on that description, and finally perform
reasoning over the self-generated output. This pipeline provides a zero-cost testbed that yields an integrated performance score specifically tailored for evaluating UMMs as unified systems. Extensive experiments show that even high-performing UMMs often struggle to reason over their own generated contexts, revealing limitations that are not captured by separate evaluations of understanding or generation alone. Our work provides a complementary holistic evaluation framework and offers a foundation for benchmarking the development of next-generation unified
multimodal models.
Read the full abstract ▾
All authors Hao Zhang, Jiaxin Qi, Zhijiang Tang, Jianqiang Huang
Artificial Intelligence
Policy-as-logic for robust reasoning over rules
Rahul Nair, Bastian Lipka and Elizabeth Daly
In many practical applications of generative AI systems, from tax rules to airline baggage allowance, responses to natural
language queries must respect written policies or rules. We present a hybrid symbolic approach that expresses policies in formal logic and at inference time exploits the representation power of
language models for fact extraction to ground predicates, and an answer set solver for
reasoning such that responses are interpretable, auditable, and as we show, accurate and robust under input perturbations. Specifically, we show this separation of extraction and
reasoning steps outperforms policy-as-
prompt and policy-as-code methods in most cases with ~10x reduction in
token usage. The results point to the value of structured
reasoning and symbolic solvers in conjunction with generative models to make robust decisions involving objective criteria.
Read the full abstract ▾
Artificial Intelligence · Cross-listed from cs.CY
Benchmark-Based Comparative Assessment of Publicly Benchmarked Indian Foundation Models: A Capability and Evaluation-Maturity Framework
Avinash Agarwal and Vridhi Jain
Purpose: Governments increasingly fund indigenous foundation models to strengthen national AI capability, digital sovereignty, and
multilingual computing. This paper assesses India's foundation-model ecosystem and examines whether apparent capability gaps in public
benchmark evidence may also reflect gaps in evaluation maturity. Approach: The paper presents a structured,
benchmark-based comparative assessment of Indian foundation models against global frontier and comparable-scale models across eight capability domains: general-purpose
reasoning, coding and software engineering, agentic AI and computer use, cybersecurity,
vision and image understanding,
video and
multimodal understanding, scientific research, and Indic
language capability. Using only publicly reported results, it proposes an exploratory four-dimension
Benchmark Maturity Index (BMI), scoring each domain on standardization, participation, independent verification, and national Findings: Indian models achieve strong scores on established
benchmarks such as MMLU and MATH-500. However, these are now widely regarded as saturated, and frontier developers no longer report them. Indian models participate far less frequently in newer, agentic, and domain-specialized evaluations, and participation is highly uneven across organizations. Sarvam AI reports the broadest coverage by a substantial margin. The BMI refines, and in some cases revises, the maturity judgments a purely descriptive review would produce. Practical implications: Many apparent capability gaps cannot be distinguished, on available evidence, from evaluation-ecosystem gaps, with direct implications for how national AI programs should design monitoring and funding criteria. Originality: The paper proposes BMI as a reusable instrument for scoring evaluation-ecosystem maturity at the domain level and demonstrates its application to the Indian foundation-model ecosystem.
Read the full abstract ▾
Artificial Intelligence · Cross-listed from cs.DB
DexterSQL: Deep Schema Exploration and Rule-based Correction for Text-to-SQL Generation
Anik Pramanik, Murat Kantarcioglu, Vincent Oria et al. (+1 more)
Prompting-based (i.e., non-fine-tuning) Text-to-SQL methods, where underlying large
language model parameters are not changed for the task, face three problems: (i) relying on coarse-grained schema information that may not reveal the fine-grained relationships needed to distinguish ambiguous columns, (ii) failing to capture recurring SQL-generation failures, and (iii) suffering from omission or
hallucination of components in complex questions. This paper develops DexterSQL, a prompting/non-fine-tuning-based Text-to-SQL system that improves SQL generation with three novel components: (i) deep schema explorator that identifies ambiguous columns, analyzes their individual and joint data distributions to uncover their relationships and the distinct role of each, (ii) database-agnostic rule creator that mines mismatches between generated and gold SQL only on the training database and converts them into database-agnostic corrective rules that capture recurring LLM failure patterns; and (iii) multi-path SQL generation that introduces a dependency-tree-based intermediate representation that uses the question's sentence structure to guide its decomposition into an SQL skeleton for final SQL generation. DexterSQL achieves a higher accuracy compared to the state-of-the-art using both open-source/weight and closed-source/weight models. Particularly, DexterSQL shows a high improvement of at least 5.5% using an open-weight model (GPT-OSS-120B) on BIRDDev, with total accuracy 70.4%. DexterSQL also shows better improvement of at least 1.4% using closed-weight models, with total accuracy 72.1% and 72.9% on BIRD-Dev with GPT-4o and GPT-5.2.
Read the full abstract ▾
All authors Anik Pramanik, Murat Kantarcioglu, Vincent Oria, Shantanu Sharma
Artificial Intelligence
Agent Skills Can Be Harmful: An Empirical Study of Skill-Induced Failures in LLM Agents
Gen Dong, Yanjie Gao, Liqun Li et al. (+3 more)
Agent skills are the de facto mechanism for extending LLM
agents with reusable guidance. A skill can shape the
agent's task execution, including planning, tool use, problem-solving, and validation. Prior work reported mixed results of
agent skills: some skills improve task success rates, while others have no effect, increase
token use and execution time, and even reduce success rates. This paper presents a comprehensive analysis of skill-induced
agent failures by attributing task failures and cost regressions to specific loaded skills. We introduce a differential analysis framework that attributes a failure or regression to a skill by comparing a target skill-guided run against a no-skill or semantically matched skill reference run that solves the same task, or solves it more cheaply. We instantiate this framework on SkillsBench and SWE-Skills-Bench, yielding 307 skill-induced failures, including 125 functional failures and 182 efficiency regressions. We also build SkillTriage, a taxonomy-guided attribution tool that normalizes paired cases, extracts differential evidence, and produces triage reports. Our major findings include: (1) Skill induced functional failures are rarely caused by obviously irrelevant skills; instead, seemingly relevant skills often make the
agent incorrectly implement or omit task-required implementation elements. (2) Skill-induced efficiency regressions are not explained by
prompt length alone. (3) The largest sources within Excessive Procedure are excessive verification and heavy implementation pipelines, contributing 67 and 30 cases, respectively. This shows that skills often turn validation checklists and construction recipes into mandatory work. Based on our findings, we propose research topics and tooling improvements for safer and more cost-aware skill reuse.
Read the full abstract ▾
All authors Gen Dong, Yanjie Gao, Liqun Li, Tianyin Xu, Yu Hua, Fan Yang
Artificial Intelligence · Cross-listed from quant-ph
CoQui: A Coordinate-Conditioned Quantum Implicit Generative Adversarial Network for End-to-End Image Generation
Xue Yang, Rigui Zhou, ShiZheng Jia et al. (+7 more)
Quantum generative
adversarial networks (QGANs) have attracted increasing
attention for image generation using parameterized
quantum circuits. Existing amplitude-based approaches face two key limitations: pixel locations are typically encoded by computational-basis indices or address
qubits, causing
quantum resources to grow with image resolution; meanwhile, jointly decoding many pixels from normalized
quantum states introduces probability competition among pixels and limits precise pixel-wise control. To address these issues, we reformulate
quantum image generation as coordinate-conditioned implicit function learning. Our method takes spatial coordinates and latent variables as inputs, uses a classical
embedding network to generate input-dependent
circuit parameters, and evaluates a variational
quantum circuit at each coordinate. Pixel intensities are directly obtained from the expectation value of a dedicated color
qubit, and a complete image is generated by querying all spatial coordinates. This design decouples image resolution from address-
qubit requirements and avoids shared probability-normalization constraints across pixels. We further design a specialized variational
quantum circuit to provide structural inductive bias for coordinate-conditioned generation. Simulated experiments on two
benchmark datasets show that our method outperforms FRQI-based generation and PQWGAN in visual and quantitative quality while using fewer
qubits, and also achieves better generation quality than the corresponding classical baseline.
Read the full abstract ▾
All authors Xue Yang, Rigui Zhou, ShiZheng Jia, Dax Enshan Koh, Siong Thye Goh, Young-Wook Cho, YaoChong Li, Xuezhi Ma, Hongyu Chen, Xin Wang