Machine Learning
ML-driven detection and reduction of ballast information in multi-modal datasets
Yaroslav Solovko
Modern
datasets often contain ballast as redundant or low-utility information that increases dimensionality, storage requirements, and computational cost without contributing meaningful analytical value. This study introduces a generalized,
multimodal framework for ballast detection and reduction across structured, semi-structured, unstructured, and sparse data types. Using diverse
datasets,
entropy, mutual information, Lasso, SHAP, PCA, topic modelling, and
embedding analysis are applied to identify and eliminate ballast features. A novel Ballast Score is proposed to integrate these signals into a unified, cross-modal
pruning strategy. Experimental results demonstrate that significant portions of the feature space as often exceeding 70% in sparse or semi-structured data, can be pruned with minimal or even improved classification performance, along with substantial reductions in training time and
memory footprint. The framework reveals distinct ballast typologies (e.g. statistical, semantic, infrastructural), and offers practical guidance for leaner, more efficient machine learning pipelines.
Read the full abstract ▾
Machine Learning
Position: A Dynamical Systems Perspective is Needed to Advance Time Series Modeling
Daniel Durstewitz, Christoph Jürgen Hemmer, Florian Hess et al. (+2 more)
Time series (TS) modeling has come a long way from early statistical, mainly linear, approaches to the current trend in TS foundation models. With a lot of hype and industrial demand in this field, it is not always clear how much progress there really is. To advance TS
forecasting and analysis to the next level, here we argue that the field needs a dynamical systems (DS) perspective. TS of observations from natural or engineered systems almost always originate from some underlying DS, and arguably access to its governing equations would yield theoretically optimal forecasts. This is the promise of DS reconstruction (DSR), a class of ML/AI approaches that aim to infer surrogate models of the underlying DS from data. But models based on DS principles offer other profound advantages: Beyond short-term forecasts, they enable to predict the long-term statistics of an observed system, which in many practical scenarios may be the more relevant quantities. DS theory furthermore provides domain-independent theoretical insight into mechanisms underlying TS generation, and thereby will inform us, e.g., about upper bounds on performance of any TS model, generalization into unseen regimes as in tipping points, or potential control strategies. After reviewing some of the central concepts, methods, measures, and models in DS theory and DSR, we will discuss how insights from this field can advance TS modeling in crucial ways, enabling better
forecasting with much lower computational and
memory footprints. We conclude with a number of specific suggestions for translating insights from DSR into TS modeling.
Read the full abstract ▾
All authors Daniel Durstewitz, Christoph Jürgen Hemmer, Florian Hess, Charlotte Ricarda Doll, Lukas Eisenmann
Machine Learning
On the Mechanism and Dynamics of Modular Addition: Fourier Features, Lottery Ticket, and Grokking
Jianliang He, Leda Wang, Siyu Chen et al. (+1 more)
We present a comprehensive analysis of how two-layer neural networks learn features to solve the modular addition task. Our work provides a full mechanistic interpretation of the learned model and a theoretical explanation of its training dynamics. While prior work has identified that individual
neurons learn single-frequency
Fourier features and phase
alignment, it does not fully explain how these features combine into a global solution. We bridge this gap by formalizing a diversification condition that emerges during training when overparametrized, consisting of two parts: phase
symmetry and frequency diversification. We prove that these properties allow the network to collectively approximate a flawed indicator function on the correct logic for the modular addition task. While individual
neurons produce noisy signals, the phase
symmetry enables a majority-voting scheme that cancels out noise, allowing the network to robustly identify the correct sum. Furthermore, we explain the emergence of these features under random initialization via a lottery ticket mechanism. Our gradient flow analysis proves that frequencies compete within each
neuron, with the "winner" determined by its initial spectral magnitude and phase
alignment. From a technical standpoint, we provide a rigorous characterization of the layer-wise phase coupling dynamics and formalize the competitive landscape using the ODE comparison lemma. Finally, we use these insights to demystify grokking, characterizing it as a three-stage process involving memorization followed by two generalization phases, driven by the competition between loss minimization and weight decay.
Read the full abstract ▾
All authors Jianliang He, Leda Wang, Siyu Chen, Zhuoran Yang
Machine Learning
What is the Value of Censored Data? An Exact Analysis for the Data-driven Newsvendor
Rachitesh Kumar and Omar Mouchtaki
We study the offline data-driven newsvendor problem when historical demand is censored by past inventory levels. We introduce a class of policies for which worst-case expected regret over all bounded demand distributions admits an exact finite-dimensional characterization, valid for any sample size and historical inventory design. The dimension depends on the number of distinct inventory levels, rather than the number of observations. This framework yields exact analyses of the sales-as-demand heuristic, the Kaplan-Meier policy, and recent policies from the censored-newsvendor literature. We leverage this reduction to derive sharp insights on the achievable performance of standard inventory policies under demand censoring. In particular, our analysis of the Kaplan-Meier policy shows that while demand censoring fundamentally limits what can be learned from passive sales data, just a small amount of targeted exploration at high inventory levels can substantially improve worst-case guarantees. Finally, we further leverage our reduction to design new policies that push the frontier and substantially improve upon the worst-case regret of known policies, reducing it by up to 40%.
Read the full abstract ▾
Machine Learning
Training Large Reasoning Models Efficiently via Progressive Thought Encoding
Zeliang Zhang, Xiaodong Liu, Hao Cheng et al. (+3 more)
Large
reasoning models (LRMs) excel on complex problems but face a critical barrier to efficiency: reinforcement learning (RL) training requires long rollouts for outcome-based rewards, where autoregressive decoding dominates time and
memory usage. While sliding-window cache strategies can bound
memory, they disrupt long-context
reasoning and degrade performance. We introduce Progressive Thought Encoding, a parameter-efficient fine-tuning method that enables LRMs to reason effectively under fixed-size caches. By progressively encoding intermediate
reasoning into fixed-size vector representations, our approach eliminates the need to backpropagate through full-cache rollouts, thereby reducing
memory usage, while maintaining constant
memory during inference. Experiments on three models, including Qwen2.5-3B-Instruct, Qwen2.5-7B-Instruct, and DeepSeek-R1-Distill-Llama-8B, on six widely used challenging mathematical
benchmarks show consistent gains: our method achieves +19.3% improvement over LoRA-based fine-tuning and +29.9% over LRMs without fine-tuning on average, with up to +23.4 accuracy improvement on AIME2024/2025 under the same tight cache budgets. These results demonstrate that Progressive Thought Encoding not only improves
reasoning accuracy but also makes RL training of LRMs substantially more efficient and scalable under real-world
memory constraints.
Read the full abstract ▾
All authors Zeliang Zhang, Xiaodong Liu, Hao Cheng, Hao Sun, Chenliang Xu, Jianfeng Gao
Machine Learning
A Structural Theory of Position Bias in Transformers
Hanna Herasimchyk, Robin Labryga, Tomislav Prusina et al. (+1 more)
Transformer models systematically favor certain
token positions, yet the architectural origins of this position bias remain poorly understood. This bias is closely connected to the Lost-in-the-Middle phenomenon, where models underutilize information placed in the middle of the context. We show that causal
Transformer architectures induce a systematic position-bias prior that can help explain the influence profiles associated with Lost-in-the-Middle. To characterize this prior, we develop a structural theory based on cumulative
attention rollout with layerwise residual mixing. At finite depth, we isolate the competing positional drifts induced by causal masking, residual mixing, and positional encoding. Their interaction produces broad, often U-shaped profiles. At infinite depth, prior work predicts inevitable collapse under
attention-only theory and identifies the discrepancy with observed
Transformer behavior as an open problem. Under the stated boundedness assumptions, our residual-aware analysis replaces inevitable collapse with a dichotomy governed by cumulative
attention mixing. Empirically, across 30 model-
dataset settings, our adaptive residual-aware rollout closely matches measured input-
token influence and reduces the average profile discrepancy by approximately 85% relative to both the
attention-only and fixed-residual rollout baselines.
Read the full abstract ▾
All authors Hanna Herasimchyk, Robin Labryga, Tomislav Prusina, Sören Laue
Machine Learning · Cross-listed from cs.CR
NeST: Neuron Selective Tuning for LLM Safety
Sasha Behrouzi, Lichao Wu, Mohamadreza Rostami et al. (+1 more)
Safety
alignment is essential for the responsible deployment of Large
Language Models (LLMs). Yet, existing approaches often rely on heavyweight fine-tuning that is costly to update, audit, and maintain across model families. Full fine-tuning incurs substantial computational and storage overhead, while parameter-efficient methods, e.g., Low-Rank Adaptation (LoRA), trade efficiency for inconsistent safety gains and sensitivity to design choices. Safety intervention mechanisms reduce unsafe outputs without modifying model weights, but do not directly shape or preserve the internal representations that govern safety behavior. We present NeST, a
Neuron-Selective Tuning framework for efficient post-hoc safety
alignment. NeST identifies safety-relevant feed-forward
neurons via activation probing on vanilla harmful and benign
prompts, clusters
neurons with similar activation profiles, and trains shared cluster-level updates while freezing the rest of the model. Importantly, NeST is trained only on vanilla malicious
prompts, without using jailbreak-specific attack data, yet generalizes robustly to diverse jailbreaks. The learned updates are then folded into the original weights, incurring no inference-time overhead. Evaluated on 14 open-weight
language and
multimodal models, NeST outperforms lightweight baselines and approaches full fine-tuning robustness with significantly fewer trainable parameters. On text-only models, NeST reduces average jailbreak attack success rate from 44.5% to 1.1% while training only 0.4M parameters on average. Across
multimodal settings, it reduces ASR from 55.3% to 1.1%, and for downstream fine-tuned variants, it restores safety by reducing ASR from 53.8% to 0.8%. These results show that robust, maintainable safety
alignment can be achieved by concentrating adaptation on localized, functionally coherent safety structures.
Read the full abstract ▾
All authors Sasha Behrouzi, Lichao Wu, Mohamadreza Rostami, Ahmad-Reza Sadeghi
Machine Learning
VAM: Verbalized Action Masking for Controllable Exploration in RL Post-Training – A Chess Case Study
Zhicheng Zhang, Ziyan Wang, Yali Du et al. (+1 more)
Exploration remains a key bottleneck for reinforcement learning (RL) post-training of large
language models (LLMs), where sparse feedback and large action spaces can lead to premature collapse into repetitive behaviors. We propose Verbalized Action Masking (VAM), which verbalizes an action mask in the
prompt and enforces that the model outputs an action from the masked set. Building on this interface, we introduce iterative action-space
pruning: if the target action is not sampled, we remove valid sampled actions from the mask and resample under the reduced candidate set, repeating until the target is sampled or a fixed budget is exhausted. We study VAM in chess and evaluate it under two training regimes: an engine-play regime that generates states via play against an engine opponent and a fixed-
dataset regime that trains from a fixed
dataset of positions with verifier scores. Across held-out chess puzzles and full-game play measured by average centipawn loss (ACPL), VAM improves learning efficiency and final performance over strong baselines, highlighting verbalized masking as a practical mechanism for controllable exploration in LLM RL post-training.
Read the full abstract ▾
All authors Zhicheng Zhang, Ziyan Wang, Yali Du, Fei Fang
Machine Learning · Cross-listed from stat.AP
The Impact of Formations on Football Matches Using Double Machine Learning. Is it worth parking the bus?
Genís Ruiz-Menárguez and Llorenç Badiella
This study addresses a central tactical dilemma for football coaches: whether to employ a defensive strategy, colloquially known as "parking the bus", or a more offensive one. Using an advanced Double Machine Learning (DML) framework, this project provides a robust and interpretable tool to estimate the causal impact of different formations on key match outcomes such as goal difference, possession, corners, and disciplinary actions. Leveraging a
dataset of over 22,000 matches from top European leagues, formations were categorized into six representative types based on tactical structure and expert consultation. A major methodological contribution lies in the adaptation of DML to handle categorical treatments, specifically formation combinations, through a novel
matrix-based residualization process, allowing for a detailed estimation of formation-versus-formation effects that can inform a coach's tactical decision-making. Results show that while offensive formations like 4-3-3 and 4-2-3-1 offer modest statistical advantages in possession and corners, their impact on goals is limited. Furthermore, no evidence supports the idea that defensive formations, commonly associated with parking the bus, increase a team's winning potential. Additionally, red cards appear unaffected by formation choice, suggesting other behavioral factors dominate. Although this approach does not fully capture all aspects of playing style or team strength, it provides a valuable framework for coaches to analyze tactical efficiency and sets a precedent for future research in sports analytics.
Read the full abstract ▾
Machine Learning
Learning under noisy supervision is governed by a feedback-truth gap
Elan Schonfeld and Elias Wisnia
When feedback is absorbed faster than task structure can be evaluated, the learner will favor feedback over truth. A two-timescale model shows this feedback-truth gap is inevitable whenever the two rates differ and vanishes only when they match. We test this prediction across neural networks trained with noisy labels (30
datasets, 2,700 runs), human probabilistic reversal learning (N = 292), and human reward/punishment learning with concurrent
EEG (N = 25). In each system, truth is defined operationally: held-out labels, the objectively correct option, or the participant's pre-feedback expectation - the only non-circular reference decodable from post-feedback
EEG. The gap appeared universally but was regulated differently: dense networks accumulated it as memorization; sparse-residual scaffolding suppressed it; humans generated transient over-commitment that was actively recovered. Neural over-commitment (~0.04-0.10) was amplified tenfold into behavioral commitment (d = 3.3-3.9). The gap is a fundamental constraint on learning under noisy supervision; its consequences depend on the regulation each system employs.
Read the full abstract ▾
Machine Learning
HiVAE: Hierarchical Latent Variables for Scalable Theory of Mind
Nigel Doering, Rahath Malladi, Arshia Sangwan et al. (+2 more)
Theory of mind (ToM) enables AI systems to infer
agents' hidden goals and mental states, but existing approaches focus mainly on small human understandable gridworld spaces. We introduce HiVAE, a hierarchical variational architecture that scales ToM
reasoning to realistic spatiotemporal domains. Inspired by the belief-desire-intention structure of human cognition, our three-level VAE hierarchy achieves substantial performance improvements on a 3,185-node campus
navigation task. However, we identify a critical limitation: while our hierarchical structure improves prediction, learned latent representations lack explicit grounding to actual mental states. We propose self-supervised
alignment strategies and present this work to solicit community feedback on grounding approaches.
Read the full abstract ▾
All authors Nigel Doering, Rahath Malladi, Arshia Sangwan, David Danks, Tauhidur Rahman
Machine Learning
Formal Mechanistic Interpretability: Automated Circuit Discovery with Provable Guarantees
Itamar Hadad, Guy Katz and Shahaf Bassan
*Automated
circuit discovery* is a central tool in mechanistic interpretability for identifying the internal components of neural networks responsible for specific behaviors. While prior methods have made significant progress, they typically depend on heuristics or approximations and do not offer provable guarantees over continuous input domains for the resulting
circuits. In this work, we leverage recent advances in neural network verification to propose a suite of automated
algorithms that yield
circuits with *provable guarantees*. We focus on three types of guarantees: (1) *input domain robustness*, ensuring the
circuit agrees with the model across a continuous input region; (2) *robust patching*, certifying
circuit alignment under continuous patching perturbations; and (3) *minimality*, formalizing and capturing a wide array of various notions of succinctness. Interestingly, we uncover a diverse set of novel theoretical connections among these three families of guarantees, with critical implications for the convergence of our
algorithms. Finally, we conduct experiments with state-of-the-art verifiers on various
vision models, showing that our
algorithms yield
circuits with substantially stronger robustness guarantees than standard
circuit discovery methods, establishing a principled foundation for provable
circuit discovery.
Read the full abstract ▾
Machine Learning
TopoFlow: Topography-aware Pollutant Flow Learning for High-Resolution Air Quality Prediction
Ammar Kheder, Helmi Toropainen, Wenqing Peng et al. (+4 more)
We propose TopoFlow (Topography-aware pollutant Flow learning), a physics-guided neural network for efficient, high-resolution air quality prediction. To explicitly embed physical processes into the learning framework, we identify two critical factors governing pollutant dynamics: topography and wind direction. Complex terrain can channel, block, and trap pollutants, while wind acts as a primary driver of their transport and dispersion. Building on these insights, TopoFlow leverages a
vision transformer architecture with two novel mechanisms: topography-aware
attention, which explicitly models terrain-induced flow patterns, and wind-guided patch reordering, which aligns spatial representations with prevailing wind directions. Trained on six years of high-resolution reanalysis data assimilating observations from over 1,400 surface monitoring stations across China, TopoFlow achieves a PM2.5 RMSE of 9.71 ug/m3, representing a 71-80% improvement over operational
forecasting systems and a 13% improvement over state-of-the-art AI baselines. Forecast errors remain well below China's 24-hour air quality threshold of 75 ug/m3 (GB 3095-2012), enabling reliable discrimination between clean and polluted conditions. These performance gains are consistent across all four major pollutants and forecast lead times from 12 to 96 hours, demonstrating that principled integration of physical knowledge into neural networks can fundamentally advance air quality prediction.
Read the full abstract ▾
All authors Ammar Kheder, Helmi Toropainen, Wenqing Peng, Samuel Antão, Jia Chen, Michael Boy, Zhi-Song Liu
Machine Learning · Cross-listed from cs.SE
Hybrid-Gym: Training Coding Agents to Generalize Across Tasks
Yiqing Xie, Emmy Liu, Gaokai Zhang et al. (+7 more)
When assessing the quality of coding
agents, predominant
benchmarks focus on solving single issues on GitHub, such as SWE-Bench. In contrast, in real use, these
agents solve more various and complex tasks that involve other skills such as exploring codebases, testing software, and designing architecture. In this paper, we first characterize some transferable skills that are shared across diverse tasks by decomposing trajectories into fine-grained components, and derive a set of principles for designing auxiliary training tasks to teach
language models these skills. Guided by these principles, we propose a training environment, Hybrid-Gym, consisting of a set of scalable synthetic tasks, such as function localization and dependency search. Experiments show that
agents trained on our synthetic tasks effectively generalize to diverse real-world tasks that are not present in training, improving a base model by 25.4% absolute gain on SWE-Bench Verified, 7.9% on SWT-Bench Verified, and 5.1% on Commit-0 Lite. Hybrid-Gym also complements
datasets built for the downstream tasks (e.g., improving SWE-Play by 4.9% on SWT-Bench Verified). Code available at: https://github.com/yiqingxyq/Hybrid-Gym.
Read the full abstract ▾
All authors Yiqing Xie, Emmy Liu, Gaokai Zhang, Nachiket Kotalwar, Shubham Gandhi, Sathwik Acharya, Xingyao Wang, Carolyn Rose, Graham Neubig, Daniel Fried
Machine Learning · Cross-listed from cs.AI
Simple Baselines are Competitive with Code Evolution
Yonatan Gideoni, Sebastian Risi and Yarin Gal
Code
evolution is a family of techniques that rely on large
language models to search through possible computer programs by evolving existing code. While code
evolution pipelines have shown impressive performance across domains, many are highly complex and are typically not compared to simpler alternatives. To prevent bad comparisons from stagnating the field, we propose two simple baselines and compare them to popular pipelines across several domains: finding better mathematical bounds, designing agentic scaffolds, developing heuristic combinatorial optimization
algorithms, and machine learning competitions. Surprisingly, the baselines match or outperform existing methods across all domains, prompting a closer look at which factors drive performance in different settings. For mathematical bounds, we find that the search space matters far more than the exact search
algorithm, with both simple and complex methods performing similarly given the same search space. Thus, expanding the search space is what matters for improving performance, and does so across all pipelines. Moreover, different domain knowledge embedded in
prompts can significantly affect the search's efficiency, potentially leading to overestimating a pipeline's sample efficiency. For automated agentic scaffold design, we show that typical high-variance evaluations lead to all automated search methods picking subpar scaffolds relative to a simple majority vote. To mitigate this, we propose improved evaluation procedures that reduce stochasticity while keeping the search economically feasible. Overall, these results indicate that further improving code
evolution's performance depends more on the setup surrounding the search pipeline than on the pipeline itself. We hope these insights will enable developing new code
evolution methods and help spur meaningful progress in the field.
Read the full abstract ▾
Machine Learning · Cross-listed from cs.CL
References Improve LLM Alignment in Non-Verifiable Domains
Kejian Shi, Yixin Liu, Peifeng Wang et al. (+3 more)
While Reinforcement Learning with Verifiable Rewards (RLVR) has shown strong effectiveness in
reasoning tasks, it cannot be directly applied to non-verifiable domains lacking ground-truth verifiers, such as LLM
alignment. In this work, we investigate whether reference-guided LLM-evaluators can bridge this gap by serving as soft "verifiers". First, we design evaluation protocols that enhance LLM-based evaluators for LLM
alignment using reference outputs. Through comprehensive experiments, we show that a reference-guided approach substantially improves the accuracy of less capable LLM-judges using references from frontier models; stronger LLM-judges can also be enhanced by high-quality (i.e., human-written) references. Building on these improved judges, we demonstrate the utility of high-quality references in
alignment tuning, where LLMs guided with references are used as judges to self-improve. We show that reference-guided self-improvement yields clear gains over both direct SFT on reference outputs and self-improvement with reference-free judges, achieving performance comparable to training with ArmoRM, a strong finetuned reward model. Specifically, our method achieves 73.1% and 58.7% on AlpacaEval and Arena-Hard with Llama-3-8B-Instruct, and 70.0% and 74.1% with Qwen2.5-7B, corresponding to average absolute gains of +20.2 / +17.1 points over SFT
distillation and +5.3 / +3.6 points over reference-free self-improvement on AlpacaEval / Arena-Hard. These results highlight the potential of using reference-guided LLM-evaluators to enable effective LLM post-training in non-verifiable domains.
Read the full abstract ▾
All authors Kejian Shi, Yixin Liu, Peifeng Wang, Alexander R. Fabbri, Shafiq Joty, Arman Cohan
Machine Learning · Cross-listed from cs.CR
Large-scale online deanonymization with LLMs
Simon Lermen, Daniel Paleka, Joshua Swanson et al. (+3 more)
We show that large
language models can be used to perform at-scale deanonymization. With full Internet access, our
agent can re-identify Hacker News users and Anthropic Interviewer participants at high precision, given pseudonymous online profiles and conversations alone, matching what would take hours for a dedicated human investigator. We then design attacks for the closed-world setting. Given two databases of pseudonymous individuals, each containing unstructured text written by or about that individual, we implement a scalable attack pipeline that uses LLMs to: (1) extract identity-relevant features, (2) search for candidate matches via semantic
embeddings, and (3) reason over top candidates to verify matches and reduce false positives. Compared to classical deanonymization work (e.g., on the Netflix prize) that required structured data, our approach works directly on raw user content across arbitrary platforms. We construct three
datasets with known ground-truth data to evaluate our attacks. The first links Hacker News to LinkedIn profiles, using cross-platform references that appear in the profiles. Our second
dataset matches users across Reddit movie discussion communities; and the third splits a single user's Reddit history in time to create two pseudonymous profiles to be matched. In each setting, LLM-based methods substantially outperform classical baselines, achieving up to 68% recall at 90% precision compared to near 0% for the best non-LLM method. Our results show that the practical obscurity protecting pseudonymous users online no longer holds and that threat models for online
privacy need to be reconsidered.
Read the full abstract ▾
All authors Simon Lermen, Daniel Paleka, Joshua Swanson, Michael Aerni, Nicholas Carlini, Florian Tramèr
Machine Learning
Efficient Tail-Aware Generative Optimization via Flow Model Fine-Tuning
Zifan Wang, Riccardo De Santi, Xiaoyu Mo et al. (+3 more)
Fine-tuning pre-trained
diffusion and flow models to optimize downstream utilities is central to real-world deployment. Existing
entropy-regularized methods primarily maximize expected reward, providing no mechanism to shape tail behavior. However, tail control is often essential: the lower tail determines reliability by limiting low-reward failures, while the upper tail enables discovery by prioritizing rare, high-reward outcomes. In this work, we present Tail-aware Flow Fine-Tuning (TFFT), a principled and efficient distributional fine-tuning
algorithm based on the Conditional Value-at-Risk (CVaR). We address two distinct tail-shaping goals: right-CVaR for seeking novel samples in the high-reward tail and left-CVaR for controlling worst-case samples in the low-reward tail. Unlike prior approaches that rely on non-linear optimization, we leverage the variational dual formulation of CVaR to decompose it into a decoupled two-stage procedure: a lightweight one-dimensional threshold optimization step, and a single
entropy-regularized fine-tuning process via a specific pseudo-reward. This decomposition achieves CVaR fine-tuning efficiently with computational cost comparable to standard expected fine-tuning methods. We demonstrate the effectiveness of TFFT across illustrative experiments, high-dimensional text-to-image generation, and molecular design.
Read the full abstract ▾
All authors Zifan Wang, Riccardo De Santi, Xiaoyu Mo, Michael M. Zavlanos, Andreas Krause, Karl H. Johansson
Machine Learning · Cross-listed from stat.ML
Beyond Procedure: Substantive Fairness in Conformal Prediction
Pengqi Liu, Zijun Yu, Mouloud Belbahri et al. (+3 more)
Conformal prediction (CP) offers distribution-free uncertainty quantification for machine learning models, yet its interplay with
fairness in downstream decision-making remains underexplored. Moving beyond CP as a standalone operation (procedural
fairness), we analyze the holistic decision-making pipeline to evaluate substantive
fairness-the equity of downstream outcomes. Theoretically, we derive an upper bound that decomposes prediction-set size disparity into interpretable components, clarifying how label-clustered CP helps control method-driven contributions to unfairness. To facilitate scalable empirical analysis, we introduce an LLM-in-the-loop evaluator that approximates human assessment of substantive
fairness across diverse modalities. Our experiments show that label-clustered CP often provides a favorable balance between utility and substantive
fairness, while reducing set-size disparities in line with our theory. Finally, we empirically show that equalized set sizes, rather than coverage, strongly correlate with improved substantive
fairness, enabling practitioners to design more fair CP systems. Our code is available at https://github.com/layer6ai-labs/llm-in-the-loop-conformal-
fairness.
Read the full abstract ▾
All authors Pengqi Liu, Zijun Yu, Mouloud Belbahri, Arthur Charpentier, Masoud Asgharian, Jesse C. Cresswell
Machine Learning
Escaping the Cognitive Well: Efficient Competition Math with Off-the-Shelf Models
Xingyu Dang, Rohit Agarwal, Rodrigo Porto et al. (+3 more)
In the past year, custom and unreleased math
reasoning models reached gold medal performance on the International Mathematical Olympiad (IMO). Similar performance was then reported using large-scale inference on publicly available models but at prohibitive costs (e.g., 3000 USD per problem). In this work, we present an inference pipeline that attains best-in-class performance on IMO-style math problems at an average inference cost orders of magnitude below competing methods while using only general-purpose off-the-shelf models. Our method relies on insights about grader failure in solver-grader pipelines, which we call the Cognitive Well (iterative refinement converging to a wrong solution that the solver as well as the pipeline's internal grader consider to be basically correct). Our pipeline addresses these failure modes through
conjecture extraction, wherein candidate lemmas are isolated from generated solutions and independently verified alongside their negations in a fresh environment (context detachment). On IMO-ProofBench Advanced (PB-Adv), our pipeline achieves 67.1 percent performance using Gemini 3.0 Pro with an average cost per question of approximately 31 USD. At the time of evaluation, this represented the state-of-the-art on PB-Adv among both public and unreleased models, and more than doubles the success rate of the next best publicly accessible pipeline, all at a fraction of the cost.
Read the full abstract ▾
All authors Xingyu Dang, Rohit Agarwal, Rodrigo Porto, Anirudh Goyal, Liam H Fowl, Sanjeev Arora
Machine Learning
Better Think Thrice: Learning to Reason Causally with Double Counterfactual Consistency
Victoria Lin, Xinnuo Xu, Rachel Lawrence et al. (+4 more)
Despite their strong performance on
reasoning benchmarks, large
language models (LLMs) have proven brittle when presented with counterfactual questions, suggesting weaknesses in their causal
reasoning ability. While recent work has demonstrated that labeled counterfactual tasks can be useful
benchmarks of LLMs' causal
reasoning, producing such data at the scale required to cover the vast potential space of counterfactuals is limited. In this work, we introduce double counterfactual consistency (DCC), a lightweight inference-time method for measuring and guiding the ability of LLMs to reason causally. Without requiring labeled counterfactual data, DCC verifies a model's ability to execute two important elements of causal
reasoning: causal intervention and counterfactual prediction. Using DCC, we evaluate the causal
reasoning abilities of various leading LLMs across a range of
reasoning tasks and interventions. Moreover, we demonstrate the effectiveness of DCC as a training-free test-time rejection sampling criterion and show that it can directly improve performance on
reasoning tasks across multiple model families.
Read the full abstract ▾
All authors Victoria Lin, Xinnuo Xu, Rachel Lawrence, Risa Ueno, Amit Sharma, Javier Gonzalez, Niranjani Prasad
Machine Learning
Omitted Variable Bias in Language Models Under Distribution Shift
Victoria Lin, Louis-Philippe Morency and Eli Ben-Michael
Despite their impressive performance on a wide variety of tasks, modern
language models remain susceptible to distribution shifts, exhibiting brittle behavior when evaluated on data that differs in distribution from their training data. In this paper, we describe how distribution shifts in
language models can be separated into observable and unobservable components, and we discuss how established approaches for dealing with distribution shift address only the former. Importantly, we identify that the resulting omitted variable bias from unobserved variables can compromise both evaluation and optimization in
language models. To address this challenge, we introduce a framework that maps the strength of the omitted variables to bounds on the worst-case generalization performance of
language models under distribution shift. In empirical experiments, we show that using these bounds directly in
language model evaluation and optimization provides more principled measures of out-of-distribution performance, improves true out-of-distribution performance relative to standard distribution shift adjustment methods, and further enables inference about the strength of the omitted variables when target distribution labels are available.
Read the full abstract ▾
Machine Learning
Machine Learning Argument of Latitude Error Model for LEO Satellite Orbit and Covariance Correction
Alex Moody, Penina Axelrad and Rebecca Russell
Low Earth orbit (LEO) satellites are leveraged to support new position,
navigation, and timing (PNT) service alternatives to GNSS. These alternatives require accurate propagation of satellite position and velocity with a realistic quantification of uncertainty. It is commonly assumed that the propagated uncertainty distribution is Gaussian; however, the validity of this assumption can be quickly compromised by the mismodeling of atmospheric drag. We develop a machine learning approach that corrects error growth in the argument of latitude for a diverse set of LEO satellites. The improved orbit propagation accuracy extends the applicability of the Gaussian assumption and modeling of the errors with a corrected mean and covariance. We compare the performance of a time-conditioned neural network and a Gaussian Process on
datasets computed with an open source orbit propagator and publicly available Vector Covariance Message (VCM) ephemerides. The learned models predict the argument of latitude error as a Gaussian distribution given parameters from a single VCM epoch and reverse propagation errors. We show that this one-dimensional model captures the effect of mismodeled drag, which can be mapped to the Cartesian state space. The correction method only updates information along the dimensions of dominant error growth, while maintaining the physics-based propagation of VCM covariance in the remaining dimensions. We therefore extend the utility of VCM ephemerides to longer time horizons without modifying the functionality of the existing propagator.
Read the full abstract ▾
Machine Learning
Attending to Routers Aids Indoor Wireless Localization
Ayush Roy, Tahsin Fuad Hassan, Roshan Ayyalasomayajula et al. (+1 more)
Modern machine learning-based wireless localization using Wi-Fi signals continues to face significant challenges in achieving groundbreaking performance across diverse environments. A major limitation is that most existing
algorithms do not appropriately weight the information from different routers during aggregation, resulting in suboptimal convergence and reduced accuracy. Motivated by traditional weighted triangulation methods, this paper introduces the concept of
attention to routers, ensuring that each router's contribution is weighted differently when aggregating information from multiple routers for triangulation. We demonstrate, by incorporating
attention layers into a standard machine learning localization architecture, that emphasizing the relevance of each router can substantially improve overall performance. We have also shown through evaluation over the open-sourced
datasets and demonstrate that
Attention to Routers outperforms the
benchmark architecture by over 30% in accuracy.
Read the full abstract ▾
All authors Ayush Roy, Tahsin Fuad Hassan, Roshan Ayyalasomayajula, Vishnu Suresh Lokhande
Machine Learning · Cross-listed from q-bio.QM
U-FedTomAtt: Ultra-lightweight Federated Learning with Attention for Tomato Disease Recognition
Romiyal George, Sathiyamohan Nishankar, Selvarajah Thuseethan et al. (+4 more)
Federated learning has emerged as a
privacy-preserving and efficient approach for deploying intelligent agricultural solutions. Accurate edge-based diagnosis across geographically dispersed farms is crucial for recognising tomato diseases in sustainable farming. Traditional centralised training aggregates raw data on a central server, leading to communication overhead,
privacy risks and latency. Meanwhile, edge devices require lightweight networks to operate effectively within limited resources. In this paper, we propose U-FedTomAtt, an ultra-lightweight
federated learning framework with
attention for tomato disease recognition in resource-constrained and distributed environments. The model comprises only 245.34K parameters and 71.41 MFLOPS. First, we propose an ultra-lightweight neural network with dilated bottleneck (DBNeck) modules and a linear
transformer to minimise computational and
memory overhead. To mitigate potential accuracy loss, a novel local-global residual
attention (LoGRA) module is incorporated. Second, we propose the
federated dual adaptive weight aggregation (FedDAWA)
algorithm that enhances global model accuracy. Third, our framework is validated using three
benchmark datasets for tomato diseases under simulated
federated settings. Experimental results show that the proposed method achieves 0.9910% and 0.9915% Top-1 accuracy and 0.9923% and 0.9897% F1-scores on SLIF-Tomato and PlantVillage tomato
datasets, respectively.
Read the full abstract ▾
All authors Romiyal George, Sathiyamohan Nishankar, Selvarajah Thuseethan, Chathrie Wimalasooriya, Yakub Sebastian, Roshan G. Ragel, Zhongwei Liang
Machine Learning
LiveClin: A Live Clinical Benchmark without Leakage
Xidong Wang, Shuqi Guo, Yue Shen et al. (+6 more)
The reliability of medical LLM evaluation is critically undermined by data contamination and knowledge obsolescence, leading to inflated scores on static
benchmarks. To address these challenges, we introduce LiveClin, a live
benchmark designed for approximating real-world clinical practice. Built from contemporary, peer-reviewed case reports and updated biannually, LiveClin ensures clinical currency and resists data contamination. Using a verified AI-human workflow involving 239 physicians, we transform authentic patient cases into complex,
multimodal evaluation scenarios that span the entire clinical pathway. The
benchmark currently comprises 1,407 case reports and 6,605 questions. Our evaluation of 26 models on LiveClin reveals the profound difficulty of these real-world scenarios, with the top-performing model achieving a Case Accuracy of just 35.7%. In benchmarking against human experts, Chief Physicians achieved the highest accuracy, followed closely by Attending Physicians, with both surpassing most models. LiveClin thus provides a continuously evolving, clinically grounded framework to guide the development of medical LLMs towards closing this gap and achieving greater reliability and real-world utility. Our data and code are publicly available at https://github.com/AQ-MedAI/LiveClin.
Read the full abstract ▾
All authors Xidong Wang, Shuqi Guo, Yue Shen, Junying Chen, Jian Wang, Jinjie Gu, Ping Zhang, Lei Liu, Benyou Wang
Machine Learning
Low-Dimensional and Transversely Curved Optimization Dynamics in Grokking
Yongzhong Xu
Grokking – the delayed transition from memorization to generalization in small algorithmic tasks – remains poorly understood. We present a
geometric analysis of optimization dynamics in
transformers trained on modular arithmetic. PCA of
attention weight trajectories reveals that training evolves predominantly within a low-dimensional execution subspace, with a single principal component capturing 68-83% of trajectory variance. To probe loss-landscape geometry, we measure commutator defects – the non-commutativity of successive gradient steps – and project them onto this learned subspace. We find that
curvature grows sharply in directions orthogonal to the execution subspace while the trajectory remains largely confined to it. Importantly,
curvature growth consistently precedes generalization across learning rates and hyperparameter regimes, with the lead time obeying a power law in the grokking timescale. Causal intervention experiments show that motion along the learned subspace is necessary for grokking, while artificially increasing
curvature is insufficient. Together, these results support a
geometric account in which grokking reflects escape from a metastable regime characterized by low-dimensional confinement and transverse
curvature accumulation. All findings replicate across this learning-rate range, a qualitatively different slow regime (lr=5e-5, wd=0.1, 3 layers), and three random seeds, though
alignment dynamics differ quantitatively between regimes. Causal intervention experiments establish that orthogonal gradient flow is necessary but not sufficient for grokking: suppressing it prevents generalization with a monotonic dose-response across four operations, while artificially boosting
curvature defects has no effect.
Read the full abstract ▾
Machine Learning
PETS: A Principled Framework Towards Optimal Trajectory Allocation for Efficient Test-Time Self-Consistency
Zhangyi Liu, Huaizhi Qu, Xiaowei Yin et al. (+4 more)
Test-time scaling can improve model performance by aggregating
stochastic reasoning trajectories. However, achieving sample-efficient test-time self-consistency under a limited budget remains an open challenge. We introduce PETS (Principled and Efficient Test-TimeSelf-Consistency), which initiates a principled study of trajectory allocation through an optimization framework. Central to our approach is the self-consistency rate, a new measure defined as agreement with the infinite-budget majority vote. This formulation makes sample-efficient test-time allocation theoretically grounded and amenable to rigorous analysis. We study both offline and online settings. In the offline regime, where all questions are known in advance, we connect trajectory allocation to crowdsourcing, a classic and well-developed area, by modeling
reasoning traces as workers. This perspective allows us to leverage rich existing theory, yielding theoretical guarantees and an efficient majority-voting-based allocation
algorithm. In the online streaming regime, where questions arrive sequentially and allocations must be made on the fly, we propose a novel method inspired by the offline framework. Our approach adapts budgets to question difficulty while preserving strong theoretical guarantees and computational efficiency. Experiments show that PETS consistently outperforms uniform allocation. On GPQA, PETS achieves perfect self-consistency in both settings while reducing the sampling budget by up to 75% (offline) and 55% (online) relative to uniform allocation. Code is available at https://github.com/ZDCSlab/PETS.
Read the full abstract ▾
All authors Zhangyi Liu, Huaizhi Qu, Xiaowei Yin, He Sun, Yanjun Han, Tianlong Chen, Zhun Deng
Machine Learning
DeepVision-103K: A Visually Diverse, Broad-Coverage, and Verifiable Mathematical Dataset for Multimodal Reasoning
Haoxiang Sun, Lizhen Xu, Bing Zhao et al. (+5 more)
Reinforcement Learning with Verifiable Rewards (RLVR) has been shown effective in enhancing the visual reflection and
reasoning capabilities of Large
Multimodal Models (LMMs). However, existing
datasets are predominantly derived from either small-scale manual construction or recombination of prior resources, which limits data diversity and coverage, thereby constraining further gains in model performance. To this end, we introduce DeepVision-103K, a comprehensive
dataset for RLVR training that covers diverse K12 mathematical topics, extensive knowledge points, and rich visual elements. Models trained on DeepVision achieve strong performance on
multimodal mathematical
benchmarks, and generalize effectively to general
multimodal reasoning tasks. Further analysis reveals enhanced visual perception, reflection and
reasoning capabilities in trained models, validating DeepVision's effectiveness for advancing
multimodal reasoning. Data: \href{https://huggingface.co/
datasets/skylenage/DeepVision-103K}{this url}.
Read the full abstract ▾
All authors Haoxiang Sun, Lizhen Xu, Bing Zhao, Wotao Yin, Wei Wang, Boyu Yang, Rui Wang, Hu Wei
Machine Learning · Cross-listed from cs.CR
Can Adversarial Code Comments Fool AI Security Reviewers – Large-Scale Empirical Study of Comment-Based Attacks and Defenses Against LLM Code Analysis
Scott Thornton
AI-assisted code review is widely used to detect vulnerabilities before production release. Prior work shows that
adversarial prompt manipulation can degrade large
language model (LLM) performance in code generation. We test whether similar comment-based manipulation misleads LLMs during vulnerability detection. We build a 100-sample
benchmark across Python, JavaScript, and Java, each paired with eight comment variants ranging from no comments to
adversarial strategies such as authority spoofing and technical deception. Eight frontier models, five commercial and three open-source, are evaluated in 9,366 trials.
Adversarial comments produce small, statistically non-significant effects on detection accuracy (McNemar exact p > 0.21; all 95 percent confidence intervals include zero). This holds for commercial models with 89 to 96 percent baseline detection and open-source models with 53 to 72 percent, despite large absolute performance gaps. Unlike generation settings where comment manipulation achieves high attack success, detection performance does not meaningfully degrade. More complex
adversarial strategies offer no advantage over simple manipulative comments. We test four automated defenses across 4,646 additional trials (14,012 total). Static analysis cross-referencing performs best at 96.9 percent detection and recovers 47 percent of baseline misses. Comment stripping reduces detection for weaker models by removing helpful context. Failures concentrate on inherently difficult vulnerability classes, including race conditions, timing side channels, and complex authorization logic, rather than on
adversarial comments.
Read the full abstract ▾