Machine Learning · Cross-listed from cs.CV
Quantifying and Inducing Shape Bias in CNNs via Max-Pool Dilation
Takito Sawada, Akinori Iwata and Masahiro Okuda
Convolutional Neural Networks (CNNs) exhibit a well-known texture bias, prioritizing local patterns over global shapes - a tendency inherent to their convolutional architecture. While this bias is beneficial for texture-rich natural images, it often degrades performance on shape-dominant data such as illustrations and sketches. Although prior work has proposed shape-biased models to mitigate this issue, these approaches lack a quantitative metric for identifying which
datasets would actually benefit from such modifications. To address this limitation, we propose a data-driven metric that quantifies the shape-texture balance within a
dataset by computing the Structural Similarity Index (SSIM) between an image's luminance (Y) channel and its L0-smoothed counterpart. Building on this metric, we introduce a computationally efficient adaptation method that promotes shape bias by modifying the dilation of max-pooling operations while keeping convolutional weights frozen. Experimental results demonstrate consistent accuracy improvements on shape-dominant
datasets, particularly in low-data regimes where full fine-tuning is impractical, requiring training only the final classification layer.
Read the full abstract ▾
Machine Learning
Good Allocations from Bad Estimates
Sílvia Casacuberta and Moritz Hardt
Conditional average treatment effect (CATE) estimation is the de facto gold standard for targeting a treatment to a heterogeneous population. The method estimates treatment effects up to an error $\epsilon > 0$ in each of $M$ different strata of the population, targeting individuals in decreasing order of estimated treatment effect until the budget runs out. In general, this method requires $O(M/\epsilon^2)$ samples. This is best possible if the goal is to estimate all treatment effects up to an $\epsilon$ error. In this work, we show how to achieve the same total treatment effect as CATE with only $O(M/\epsilon)$ samples for natural distributions of treatment effects. The key insight is that coarse estimates suffice for near-optimal treatment allocations. In addition, we show that budget flexibility can further reduce the sample complexity of allocation. Finally, we evaluate our
algorithm on various real-world RCT
datasets. In all cases, it finds nearly optimal treatment allocations with surprisingly few samples. Our work highlights the fundamental distinction between treatment effect estimation and treatment allocation: the latter requires far fewer samples.
Read the full abstract ▾
Machine Learning
PaCoRe: Learning to Scale Test-Time Compute with Parallel Coordinated Reasoning
Jingcheng Hu, Yinmin Zhang, Shijie Shang et al. (+17 more)
We introduce Parallel Coordinated
Reasoning (PaCoRe), a training-and-inference framework designed to overcome a central limitation of contemporary
language models: their inability to scale test-time compute (TTC) far beyond sequential
reasoning under a fixed context window. PaCoRe departs from the traditional sequential paradigm by driving TTC through massive parallel exploration coordinated via a message-passing architecture in multiple rounds. Each round launches many parallel
reasoning trajectories, compacts their findings into context-bounded messages, and synthesizes these messages to guide the next round and ultimately produce the final answer. Trained end-to-end with large-scale, outcome-based reinforcement learning, the model masters the synthesis abilities required by PaCoRe and scales to multi-million-
token effective TTC without exceeding context limits. The approach yields strong improvements across diverse domains, and notably pushes
reasoning beyond frontier systems in mathematics: an 8B model reaches 94.5% on HMMT 2025, surpassing GPT-5's 93.2% by scaling effective TTC to roughly two million
tokens. We open-source model checkpoints, training data, and the full inference pipeline to accelerate follow-up work.
Read the full abstract ▾
All authors Jingcheng Hu, Yinmin Zhang, Shijie Shang, Xiaobo Yang, Yue Peng, Zhewei Huang, Hebin Zhou, Xin Wu, Jie Cheng, Fanqi Wan, Xiangwen Kong, Chengyuan Yao, Kaiwen Yan, Ailin Huang, Hongyu Zhou, Qi Han, Zheng Ge, Daxin Jiang, Xiangyu Zhang, Heung-Yeung Shum
Machine Learning · Cross-listed from cs.IR
Autoregressive Ranking: Bridging the Gap Between Dual and Cross Encoders
Benjamin Rozonoyer, Chong You, Michael Boratko et al. (+5 more)
The success of Large
Language Models (LLMs) has motivated a shift toward generative approaches to
retrieval and ranking, aiming to supersede classical Dual Encoders (DEs) and Cross Encoders (CEs). A prominent paradigm is pointwise Autoregressive Ranking (ARR), where an LLM generates document identifiers (docIDs)
token-by-
token to enable ranking via beam search. ARR offers the promise of superior expressivity compared to DEs while avoiding the prohibitive computational cost of CEs. However, a formal theoretical foundation for this expressive power has been missing. Moreover, the standard next-
token prediction loss is rank-agnostic and inappropriate for finetuning an LLM for ranking tasks. In this paper, we first prove that the expressive capacity of ARR is strictly superior to DEs. While a DE requires an
embedding dimension that grows linearly with corpus size to achieve arbitrary rankings, ARR can solve it with a constant hidden dimension. We then propose SToICaL (Simple
Token-Item Calibrated Loss), a generalized rank-aware training loss for LLM finetuning. By using item-level reweighting and prefix-tree marginalization, we distribute probability mass over valid docID
tokens based on their ground-truth relevance. Experiments on WordNet and ESCI
datasets verify that our loss suppresses invalid docID generations and significantly improves ranking metrics beyond top-1
retrieval.
Read the full abstract ▾
All authors Benjamin Rozonoyer, Chong You, Michael Boratko, Himanshu Jain, Nilesh Gupta, Srinadh Bhojanapalli, Andrew McCallum, Felix Yu
Machine Learning
Poisson Hyperplane Processes with Rectified Linear Units
Shufei Ge, Shijia Wang and Lloyd Elliott
Neural networks have shown state-of-the-art performances in various classification and regression tasks. Rectified linear units (ReLU) are often used as activation functions for the hidden layers in a neural network model. In this article, we establish the connection between the Poisson hyperplane processes (PHP) and two-layer ReLU neural networks. We show that the PHP with a Gaussian prior is an alternative probabilistic representation to a two-layer ReLU neural network. In addition, we show that a two-layer neural network constructed by PHP is scalable to large-scale problems via the decomposition propositions. Finally, we propose an annealed sequential Monte Carlo
algorithm for
Bayesian inference. Our numerical experiments demonstrate that our proposed method outperforms the classic two-layer ReLU neural network. The implementation of our proposed model is available at https://github.com/ShufeiGe/Pois_Relu.git.
Read the full abstract ▾
Machine Learning
Learn to Evolve: Self-supervised Neural JKO Operator for Wasserstein Gradient Flow
Xue Feng, Li Wang, Deanna Needell et al. (+1 more)
The Jordan-Kinderlehrer-Otto (JKO) scheme provides a stable variational framework for computing Wasserstein gradient flows, but its practical use is often limited by the high computational cost of repeatedly solving the JKO subproblems. We propose a self-supervised approach for learning a JKO solution operator without requiring numerical solutions of any JKO trajectories. The learned operator maps an input density directly to the minimizer of the corresponding JKO subproblem, and can be iteratively applied to efficiently generate the gradient-flow
evolution. A key challenge is that only a number of initial densities are typically available for training. To address this, we introduce a Learn-to-Evolve
algorithm that jointly learns the JKO operator and its induced trajectories by alternating between trajectory generation and operator updates. As training progresses, the generated data increasingly approximates true JKO trajectories. Meanwhile, this Learn-to-Evolve strategy serves as a natural form of data augmentation, significantly enhancing the generalization ability of the learned operator. Numerical experiments demonstrate the accuracy, stability, and robustness of the proposed method across various choices of energies and initial conditions.
Read the full abstract ▾
All authors Xue Feng, Li Wang, Deanna Needell, Rongjie Lai
Machine Learning · Cross-listed from cond-mat.stat-mech
Autonomous Discovery of the Ising Model's Critical Parameters with Reinforcement Learning
Hai Man, Chaobo Wang, Jia-Rui Li et al. (+2 more)
Traditional methods for determining critical parameters are often influenced by human factors. This research introduces a physics-inspired adaptive reinforcement learning framework that enables
agents to autonomously interact with physical environments, simultaneously identifying both the critical temperature and various types of critical exponents in the Ising model with precision. Interestingly, our
algorithm exhibits search behavior reminiscent of phase transitions, efficiently converging to target parameters regardless of initial conditions. Experimental results demonstrate that this method significantly outperforms traditional approaches, particularly in environments with strong perturbations. This study not only incorporates physical concepts into machine learning to enhance
algorithm interpretability but also establishes a new paradigm for scientific exploration, transitioning from manual analysis to autonomous AI discovery.
Read the full abstract ▾
All authors Hai Man, Chaobo Wang, Jia-Rui Li, Yuping Tian, Shu-Gang Chen
Machine Learning
Buffered AUC maximization for scoring systems via mixed-integer optimization
Moe Shiina, Shunnosuke Ikeda and Yuichi Takano
A scoring system is a linear classifier composed of a small number of explanatory variables, each assigned a small integer coefficient. This system is highly interpretable and allows predictions to be made with simple manual calculations without the need for a calculator. Several previous studies have used mixed-integer optimization (MIO) techniques to develop scoring systems for binary classification; however, they have not focused on directly maximizing AUC (i.e., area under the receiver operating characteristic curve), even though AUC is recognized as an essential evaluation metric for scoring systems. Our goal herein is to establish an effective MIO framework for constructing scoring systems that directly maximize the buffered AUC (bAUC) as the tightest concave lower bound on AUC. Our optimization model is formulated as a mixed-integer linear optimization (MILO) problem that maximizes bAUC subject to a group sparsity constraint for limiting the number of questions in the scoring system. Computational experiments using publicly available real-world
datasets demonstrate that our MILO method can build scoring systems with superior AUC values compared to the baseline methods based on regularization and stepwise regression. This research contributes to the advancement of MIO techniques for developing highly interpretable classification models.
Read the full abstract ▾
Machine Learning · Cross-listed from cs.SE
Understanding LLM-Driven Test Oracle Generation
Adam Bodicoat, Gunel Jahangirova and Valerio Terragni
Automated unit test generation aims to improve software quality while reducing the time and effort required for creating tests manually. However, existing techniques primarily generate regression oracles that predicate on the implemented behavior of the class under test. They do not address the oracle problem: the challenge of distinguishing correct from incorrect program behavior. With the rise of Foundation Models (FMs), particularly Large
Language Models (LLMs), there is a new opportunity to generate test oracles that reflect intended behavior. This positions LLMs as enablers of Promptware, where software creation and testing are driven by natural-
language prompts. This paper presents an empirical study on the effectiveness of LLMs in generating test oracles that expose software failures. We investigate how different prompting strategies and levels of contextual input impact the quality of LLM-generated oracles. Our findings offer insights into the strengths and limitations of LLM-based oracle generation in the FM era, improving our understanding of their capabilities and fostering future research in this area.
Read the full abstract ▾
Machine Learning
Scalable Heterogeneous Graph Learning via Heterogeneous-aware Orthogonal Prototype Experts
Wei Zhou, Hong Huang, Ruize Shi et al. (+1 more)
Heterogeneous Graph Neural Networks(HGNNs) have advanced mainly through better encoders, yet their decoding/projection stage still relies on a single shared linear head, assuming it can map rich node
embeddings to labels. We call this the Linear Projection Bottleneck: in heterogeneous graphs, contextual diversity and long-tail shifts make a global head miss fine semantics, overfit hub nodes, and underserve tail nodes. While Mixture-of-Experts(MoE) could help, naively applying it clashes with structural imbalance and risks expert collapse. We propose a Heterogeneous-aware Orthogonal Prototype Experts framework named HOPE, a plug-and-play replacement for the standard prediction head. HOPE uses learnable prototype-based routing to assign instances to experts by similarity, letting expert usage follow the natural long-tail distribution, and adds expert orthogonalization to encourage diversity and prevent collapse. Experiments on four real
datasets show consistent gains across SOTA HGNN backbones with minimal overhead.
Read the full abstract ▾
All authors Wei Zhou, Hong Huang, Ruize Shi, Bang Liu
Machine Learning · Cross-listed from q-bio.GN
DNATokenizer: A GPU-First Byte-to-Identifier Tokenizer for High-Throughput DNA Language Models
Eliatan Niktab and Hardip Patel
Tokenization sits at the boundary between high-throughput genomic input and GPU compute, posing challenges in both
algorithm design and system throughput. Overlapping k-mer tokenization can introduce information leakage under masked
language modeling (MLM) and may degrade downstream accuracy. Single-nucleotide tokenization avoids leakage and preserves per-base fidelity, but it greatly increases sequence length for
attention-based architectures. Non-overlapping k-mers and byte-pair encoding (BPE) provide compression and avoid leakage, at the cost of boundary sensitivity or reduced interpretability. Empirically, the choice of tokenization interacts strongly with model architecture and task requirements. At the system level, however, standard string tokenizers and host-bound vocabulary lookups dominate wall-clock time once inputs reach billions of bases, regardless of the tokenization
algorithm. We present DNATok, a high-performance, GPU-first tokenization system that replaces general-purpose string processing with byte lookup table (LUT)-based identifier streaming and an overlapped host-to-device (H2D)/compute pipeline using pinned
memory and architectural parallelism. DNATok is vocabulary-agnostic: it accelerates single-nucleotide, non-overlapping k-mer, and BPE tokenization, and integrates as a drop-in systems layer beneath genomic foundation models. DNATok achieves 84-95x higher encoding throughput than optimized Hugging Face baselines and up to 1.9x higher H2D throughput. End-to-end streaming reaches 1.27-1.84e8
tokens/s depending on configuration, effectively removing tokenization as a bottleneck for production-scale training and inference.
Read the full abstract ▾
Machine Learning · Cross-listed from cond-mat.mtrl-sci
Autonomous Probe Microscopy with Robust Bag-of-Features Multi-Objective Bayesian Optimization: Pareto-Front Mapping of Nanoscale Structure-Property Trade-Offs
Kamyar Barakati, Haochen Zhu, C Charlotte Buchanan et al. (+3 more)
Combinatorial materials libraries are an efficient route to generate large families of candidate compositions, but their impact is often limited by the speed and depth of characterization and by the difficulty of extracting actionable structure-property relations from complex characterization data. Here we develop an autonomous scanning probe microscopy (SPM) framework that integrates automated atomic force and
magnetic force microscopy (AFM/MFM) to rapidly explore
magnetic and structural properties across combinatorial spread libraries. To enable automated exploration of systems without a clear optimization target, we introduce a combination of a static physics-informed bag-of-features (BoF) representation of measured surface morphology and
magnetic structure with multi-objective
Bayesian optimization (MOBO) to discover the relative significance and robustness of features. The resulting closed-loop workflow selectively samples the compositional gradient and reconstructs feature landscapes consistent with dense grid "ground truth" measurements. The resulting Pareto structure reveals where multiple nanoscale objectives are simultaneously optimized, where trade-offs between roughness, coherence, and
magnetic contrast are unavoidable, and how families of compositions cluster into distinct functional regimes, thereby turning multi-feature imaging data into interpretable maps of competing structure-property trends. While demonstrated for Au-Co-Ni and AFM/MFM, the approach is general and can be extended to other combinatorial systems, imaging modalities, and feature sets, illustrating how feature-based MOBO and autonomous SPM can transform microscopy images from static data products into active feedback for real-time, multi-objective materials discovery.
Read the full abstract ▾
All authors Kamyar Barakati, Haochen Zhu, C Charlotte Buchanan, Dustin A Gilbert, Philip Rack, Sergei V. Kalinin
Machine Learning
DeMa: Dual-Path Delay-Aware Mamba for Efficient Multivariate Time Series Analysis
Rui An, Haohao Qu, Wenqi Fan et al. (+2 more)
Accurate and efficient multivariate time series (MTS) analysis is increasingly critical for a wide range of intelligent applications. Within this realm,
Transformers have emerged as the predominant architecture due to their strong ability to capture pairwise dependencies. However,
Transformer-based models suffer from quadratic computational complexity and high
memory overhead, limiting their scalability and practical deployment in long-term and large-scale MTS modeling. Recently, Mamba has emerged as a promising linear-time alternative with high expressiveness. Nevertheless, directly applying vanilla Mamba to MTS remains suboptimal due to three key limitations: (i) the lack of explicit cross-variate modeling, (ii) difficulty in disentangling the entangled intra-series temporal dynamics and inter-series interactions, and (iii) insufficient modeling of latent time-lag interaction effects. These issues constrain its effectiveness across diverse MTS tasks. To address these challenges, we propose DeMa, a dual-path delay-aware Mamba backbone. DeMa preserves Mamba's linear-complexity advantage while substantially improving its suitability for MTS settings. Specifically, DeMa introduces three key innovations: (i) it decomposes the MTS into intra-series temporal dynamics and inter-series interactions; (ii) it develops a temporal path with a Mamba-SSD module to capture long-range dynamics within each individual series, enabling series-independent, parallel computation; and (iii) it designs a variate path with a Mamba-DALA module that integrates delay-aware linear
attention to model cross-variate dependencies. Extensive experiments on five representative tasks, long- and short-term
forecasting, data imputation, anomaly detection, and series classification, demonstrate that DeMa achieves state-of-the-art performance while delivering remarkable computational efficiency.
Read the full abstract ▾
All authors Rui An, Haohao Qu, Wenqi Fan, Xuequn Shang, Qing Li
Machine Learning · Cross-listed from cs.AI
Explainable AI: Learning from the Learners
Ricardo Vinuesa, Steven L. Brunton and Gianmarco Mengaldo
Artificial intelligence now outperforms humans in several scientific and engineering tasks, yet its internal representations often remain opaque. In this Perspective, we argue that explainable artificial intelligence (XAI), combined with causal
reasoning, enables learning from the learners. Focusing on discovery, optimization and certification, we show how the combination of foundation models and explainability methods allows the extraction of causal mechanisms, guides robust design and control, and supports trust and accountability in high-stakes applications. We discuss challenges in faithfulness, generalization and usability of explanations, and propose XAI as a unifying framework for human-AI collaboration in science and engineering.
Read the full abstract ▾
Machine Learning
Toward an Integrated Cross-Urban Accident Prevention System: A Multi-Task Spatial-Temporal Learning Framework for Urban Safety Management
Jiayu Fang, Zhiqi Shao, Haoning Xi et al. (+2 more)
The development of a cross-city accident prevention system is particularly challenging due to the heterogeneity, inconsistent reporting, and inherently clustered, sparse, cyclical, and noisy nature of urban accident data. These intrinsic data properties, combined with fragmented governance and incompatible reporting standards, have long hindered the creation of an integrated, cross-city accident prevention framework. To address this gap, we propose the Mamba Local-ttention Spatial-Temporal Network MLA-STNet, a unified system that formulates accident risk prediction as a multi-task learning problem across multiple cities. MLA-STNet integrates two complementary modules: (i)the Spatio-Temporal Geographical Mamba-
Attention (STG-MA), which suppresses unstable spatio-temporal fluctuations and strengthens long-range temporal dependencies; and (ii) the Spatio-Temporal Semantic Mamba-
Attention (STS-MA), which mitigates cross-city heterogeneity through a shared-parameter design that jointly trains all cities while preserving individual semantic representation spaces. We validate the proposed framework through 75 experiments under two
forecasting scenarios, full-day and high-frequency accident periods, using real-world
datasets from New York City and Chicago. Compared with the state-of-the-art baselines, MLA-STNet achieves up to 6% lower RMSE, 8% higher Recall, and 5% higher MAP, while maintaining less than 1% performance variation under 50% input noise. These results demonstrate that MLA-STNet effectively unifies heterogeneous urban
datasets within a scalable, robust, and interpretable Cross-City Accident Prevention System, paving the way for coordinated and data-driven urban safety management.
Read the full abstract ▾
All authors Jiayu Fang, Zhiqi Shao, Haoning Xi, Boris Choy, Junbin Gao
Machine Learning
Over-Searching in Search-Augmented Large Language Models
Roy Xie, Deepak Gopinath, David Qiu et al. (+4 more)
Search-augmented large
language models (LLMs) excel at knowledge-intensive tasks by integrating external
retrieval. However, they often over-search – unnecessarily invoking search tool even when it does not improve response quality, which leads to computational inefficiency and
hallucinations by incorporating irrelevant context. In this work, we conduct a systematic evaluation of over-searching across multiple dimensions, including query types, model categories,
retrieval conditions, and multi-turn conversations. Our finding shows: (i) search generally improves answer accuracy on answerable queries but harms abstention on unanswerable ones; (ii) over-searching is more pronounced in complex
reasoning models and deep research systems, is exacerbated by noisy
retrieval, and compounds across turns in multi-turn conversations; and (iii) the composition of retrieved evidence is crucial, as the presence of negative evidence improves abstention. To quantify over-searching, we introduce
Tokens Per Correctness (TPC), an evaluation metric that captures the performance-cost trade-off for search-augmented LLMs. Lastly, we investigate mitigation approaches at both the query and
retrieval levels and release the OverSearchQA to foster continued research into efficient search-augmented LLMs.
Read the full abstract ▾
All authors Roy Xie, Deepak Gopinath, David Qiu, Dong Lin, Haitian Sun, Saloni Potdar, Bhuwan Dhingra
Machine Learning
Hi-ZFO: Hierarchical Zeroth- and First-Order LLM Fine-Tuning via Importance-Guided Tensor Selection
Feihu Jin and Ying Tan
Fine-tuning large
language models (LLMs) using standard first-order (FO) optimization often drives training toward sharp, poorly generalizing minima. Conversely, zeroth-order (ZO) methods offer stronger exploratory behavior without relying on explicit gradients, yet suffer from slow convergence. More critically, our analysis reveals that in generative tasks, the vast output and search space significantly amplify estimation variance, rendering ZO methods both noisy and inefficient. To address these challenges, we propose Hi-ZFO (Hierarchical Zeroth- and First-Order optimization), a hybrid framework designed to synergize the precision of FO gradients with the exploratory capability of ZO estimation. Hi-ZFO adaptively partitions the model through layer-wise importance profiling, applying precise FO updates to critical layers while leveraging ZO optimization for less sensitive ones. Notably, ZO in Hi-ZFO is not merely a
memory-saving surrogate; it is intentionally introduced as a source of "beneficial stochasticity" to help the model escape the local minima where pure FO optimization tends to stagnate. Validated across diverse generative, mathematical, and code
reasoning tasks, Hi-ZFO consistently achieves superior performance while significantly reducing the training time. These results demonstrate the effectiveness of hierarchical hybrid optimization for LLM fine-tuning.
Read the full abstract ▾
Machine Learning · Cross-listed from cs.CV
Prompt-Free SAM-Based Multi-Task Framework for Breast Ultrasound Lesion Segmentation and Classification
Samuel E. Johnny, Bernes L. Atabonfack, Israel Alagbe et al. (+1 more)
Accurate tumor
segmentation and classification in breast ultrasound (BUS) imaging remain challenging due to low contrast, speckle noise, and diverse lesion morphology. This study presents a multi-task deep learning framework that jointly performs lesion
segmentation and diagnostic classification using
embeddings from the Segment Anything Model (SAM)
vision encoder. Unlike
prompt-based SAM variants, our approach employs a
prompt-free, fully supervised adaptation where high-dimensional SAM features are decoded through either a lightweight convolutional head or a UNet-inspired decoder for pixel-wise
segmentation. The classification branch is enhanced via mask-guided
attention, allowing the model to focus on lesion-relevant features while suppressing background artifacts. Experiments on the PRECISE 2025 breast ultrasound
dataset, split per class into 80 percent training and 20 percent testing, show that the proposed method achieves a Dice Similarity Coefficient (DSC) of 0.887 and an accuracy of 92.3 percent, ranking among the top entries on the PRECISE challenge leaderboard. These results demonstrate that SAM-based representations, when coupled with
segmentation-guided learning, significantly improve both lesion delineation and diagnostic prediction in breast ultrasound imaging.
Read the full abstract ▾
All authors Samuel E. Johnny, Bernes L. Atabonfack, Israel Alagbe, Assane Gueye
Machine Learning
MaxCode: A Max-Reward Reinforcement Learning Framework for Automated Code Optimization
Jiefu Ou, Sapana Chaudhary, Kaj Bostrom et al. (+4 more)
Large
Language Models (LLMs) demonstrate strong capabilities in general coding tasks but encounter two key challenges when optimizing code: (i) the complexity of writing optimized code (such as performant CUDA kernels and competition-level CPU code) requires expertise in systems,
algorithms and specific
languages and (ii) requires interpretation of performance metrics like timing and device utilization beyond binary correctness. In this work, we explore inference-time search
algorithms that guide the LLM to discover better solutions through iterative refinement based on execution feedback. Our approach, called MaxCode unifies existing search methods under a max-reward reinforcement learning framework, making the observation and action-value functions modular for modification. To enhance the observation space, we integrate a natural
language critique model that converts raw execution feedback into diagnostic insights about errors and performance bottlenecks, and the best-discounted reward seen so far. Together, these provide richer input to the code proposal function. To improve exploration during search, we train a generative reward-to-go model using action values from rollouts to rerank potential solutions. Testing on the KernelBench (CUDA) and PIE (C++) optimization
benchmarks shows that MaxCode improves optimized code performance compared to baselines, achieving 20.3% and 10.1% relative improvements in absolute speedup value and relative speedup ranking, respectively.
Read the full abstract ▾
All authors Jiefu Ou, Sapana Chaudhary, Kaj Bostrom, Nathaniel Weir, Shuai Zhang, Huzefa Rangwala, George Karypis
Machine Learning
Efficient Differentiable Causal Discovery via Reliable Super-Structure Learning
Pingchuan Ma, Qixin Zhang, Shuai Wang et al. (+1 more)
Recently, differentiable causal discovery has emerged as a promising approach to improve the accuracy and efficiency of existing methods. However, when applied to high-dimensional data or data with latent confounders, these methods, often based on off-the-shelf continuous optimization
algorithms, struggle with the vast search space, the complexity of the objective function, and the nontrivial nature of graph-theoretical constraints. As a result, there has been a surge of interest in leveraging super-structures to guide the optimization process. Nonetheless, learning an appropriate super-structure at the right level of granularity, and doing so efficiently across various settings, presents significant challenges. In this paper, we propose ALVGL, a novel and general enhancement to the differentiable causal discovery pipeline. ALVGL employs a sparse and low-rank decomposition to learn the precision
matrix of the data. We design an ADMM procedure to optimize this decomposition, identifying components in the precision
matrix that are most relevant to the underlying causal structure. These components are then combined to construct a super-structure that is provably a superset of the true causal graph. This super-structure is used to initialize a standard differentiable causal discovery method with a more focused search space, thereby improving both optimization efficiency and accuracy. We demonstrate the versatility of ALVGL by instantiating it across a range of structural causal models, including both Gaussian and non-Gaussian settings, with and without unmeasured confounders. Extensive experiments on synthetic and real-world
datasets show that ALVGL not only achieves state-of-the-art accuracy but also significantly improves optimization efficiency, making it a reliable and effective solution for differentiable causal discovery.
Read the full abstract ▾
All authors Pingchuan Ma, Qixin Zhang, Shuai Wang, Dacheng Tao
Machine Learning · Cross-listed from cs.AI
ART: Adaptive Reasoning Trees for Explainable Claim Verification
Sahil Wadhwa, Himanshu Kumar, Guanqun Yang et al. (+4 more)
Large
Language Models (LLMs) are powerful candidates for complex decision-making, leveraging vast encoded knowledge and remarkable zero-shot abilities. However, their adoption in high-stakes environments is hindered by their opacity; their outputs lack faithful explanations and cannot be effectively contested to correct errors, undermining trustworthiness. In this paper, we propose ART (Adaptive
Reasoning Trees), a hierarchical method for claim verification. The process begins with a root claim, which branches into supporting and attacking child arguments. An argument's strength is determined bottom-up via a pairwise tournament of its children, adjudicated by a judge LLM, allowing a final, transparent and contestable verdict to be systematically derived which is missing in methods like Chain-of-Thought (CoT). We empirically validate ART on multiple
datasets, analyzing different argument generators and comparison strategies. Our findings show that ART's structured
reasoning outperforms strong baselines, establishing a new
benchmark for explainable claim verification which is more reliable and ensures clarity in the overall decision making step.
Read the full abstract ▾
All authors Sahil Wadhwa, Himanshu Kumar, Guanqun Yang, Abbaas Alif Mohamed Nishar, Pranab Mohanty, Swapnil Shinde, Yue Wu
Machine Learning
RingSQL: Schema-Independent Synthetic Data Generation for Text-to-SQL Reinforcement Learning
Marko Sterbentz, Kevin Cushing, Cameron Barrie et al. (+1 more)
Recent advances in text-to-SQL have been driven by larger models, better
datasets, and new training methods like RLVR. However, progress remains limited by scarce high-quality training data, a problem RLVR is especially sensitive to since noisy data can produce spurious rewards. Manual data creation is expensive, and existing synthetic methods trade off reliability for scalability: template-based approaches guarantee correct SQL but need schema-specific templates and lack diversity, while LLM-based generation scales easily but lacks quality guarantees. We introduce RingSQL, a hybrid framework for generating question-SQL pairs that combines schema-independent query templates with LLM-based paraphrasing of natural
language questions. By grounding question generation in complete template questions, RingSQL preserves question-query correctness across all levels of query complexity, a property purely LLM-based methods fail to maintain. RingSQL also produces the only synthetic
dataset that improves RLVR training performance across all tested model architectures and
benchmarks, achieving 69.8% average accuracy and surpassing both the next-best synthetic
dataset by 2.1% and human-annotated data from Spider and BIRD. Code and data are available at https://github.com/nu-c3lab/RingSQL.
Read the full abstract ▾
All authors Marko Sterbentz, Kevin Cushing, Cameron Barrie, Kristian J. Hammond
Machine Learning · Cross-listed from cs.CR
Knowledge-Driven Multi-Turn Jailbreaking on Large Language Models
Songze Li, Ruishi He, Xiaojun Jia et al. (+2 more)
Large
Language Models (LLMs) face a significant threat from multi-turn jailbreak attacks, where adversaries progressively steer conversations to elicit harmful outputs. However, the practical effectiveness of existing attacks is undermined by several critical limitations: they struggle to maintain a coherent progression over long interactions, often losing track of what has been accomplished and what remains to be done; they rely on rigid or pre-defined patterns, and fail to adapt to the LLM's dynamic and unpredictable conversational state. To address these shortcomings, we introduce Mastermind, a multi-turn jailbreak framework that adopts a dynamic and self-improving approach. Mastermind operates in a closed loop of planning, execution, and reflection, enabling it to autonomously build and refine its knowledge of model vulnerabilities through interaction. It employs a hierarchical planning architecture that decouples high-level attack objectives from low-level tactical execution, ensuring long-term focus and coherence. This planning is guided by a knowledge repository that autonomously discovers and refines effective attack patterns by reflecting on interactive experiences. Mastermind leverages this accumulated knowledge to dynamically recombine and adapt attack vectors, dramatically improving both effectiveness and resilience. We conduct comprehensive experiments against state-of-the-art models, including GPT-5 and Claude 3.7 Sonnet. The results demonstrate that Mastermind significantly outperforms existing baselines, achieving substantially higher attack success rates and harmfulness ratings. Moreover, our framework exhibits notable resilience against multiple advanced defense mechanisms.
Read the full abstract ▾
All authors Songze Li, Ruishi He, Xiaojun Jia, Jun Wang, Zhihui Fu
Machine Learning · Cross-listed from math.ST
What Functions Does XGBoost Learn?
Dohyeong Ki and Adityanand Guntuboyina
This paper establishes a rigorous theoretical foundation for the function class implicitly learned by XGBoost, bridging the gap between its empirical success and our theoretical understanding. We introduce an infinite-dimensional function class $\mathcal{F}^{d, s}_{\infty-\text{ST}}$ that extends finite ensembles of bounded-depth regression trees, together with a complexity measure $V^{d, s}_{\infty-\text{XGB}}(\cdot)$ that generalizes the $L^1$ regularization penalty used in XGBoost. We show that every optimizer of the XGBoost objective is also an optimizer of an equivalent penalized regression problem over $\mathcal{F}^{d, s}_{\infty-\text{ST}}$ with penalty $V^{d, s}_{\infty-\text{XGB}}(\cdot)$, providing an interpretation of XGBoost as implicitly targeting a broader function class. We also develop a smoothness-based interpretation of $\mathcal{F}^{d, s}_{\infty-\text{ST}}$ and $V^{d, s}_{\infty-\text{XGB}}(\cdot)$ in terms of Hardy–Krause variation. We prove that the least squares
estimator over $\{f \in \mathcal{F}^{d, s}_{\infty-\text{ST}}: V^{d, s}_{\infty-\text{XGB}}(f) \le V\}$ achieves a nearly minimax-optimal rate of convergence $n^{-2/3} (\log n)^{4(\min(s, d) - 1)/3}$, thereby avoiding the curse of dimensionality. Our results provide the first rigorous characterization of the function space underlying XGBoost, clarify its connection to classical notions of variation, and identify an important open problem: whether the XGBoost
algorithm itself achieves minimax optimality over this class.
Read the full abstract ▾
Machine Learning · Cross-listed from stat.ML
A brief note on learning problem with global perspectives
Getachew K. Befekadu
This brief note considers the problem of learning with dynamic-optimizing principal-
agent setting, in which the
agents are allowed to have global perspectives about the learning process, i.e., the ability to view things according to their relative importances or in their true relations based-on some aggregated information shared by the principal. Whereas, the principal, which is exerting an influence on the learning process of the
agents in the aggregation, is primarily tasked to solve a high-level optimization problem posed as an empirical-likelihood
estimator under conditional moment restrictions model that also accounts information about the
agents' predictive performances on out-of-samples as well as a set of private
datasets available only to the principal. In particular, we present a coherent mathematical argument which is necessary for characterizing the learning process behind this abstract principal-
agent learning framework, although we acknowledge that there are a few conceptual and theoretical issues still need to be addressed.
Read the full abstract ▾