Machine Learning
A Fine-Grained Understanding of Uniform Convergence for Halfspaces
Aryeh Kontorovich and Kasper Green Larsen
We study the fine-grained uniform convergence behavior of halfspaces beyond worst-case VC bounds. For inhomogeneous halfspaces in $\mathbb{R}^d$ with $d\ge 2$, we show that standard first-order VC bounds are essentially tight: even consistent hypotheses can incur population error $\Theta(d\ln(n/d)/n)$, and in the agnostic setting the deviation scales as $\sqrt{\tau\ln(1/\tau)}$ at true error $\tau$. In contrast, homogeneous halfspaces in $\mathbb{R}^2$ exhibit a markedly different behavior. In the realizable case, every hypothesis consistent with the sample has error $O(1/n)$. In the agnostic case, we prove a bandwise, log-free deviation bound on each dyadic risk band via a critical-wedge localization argument. Unioning over bands incurs only a $\ln\ln n$ overhead, and we establish a matching lower bound showing this overhead is unavoidable. Together, these results give a fine-grained and nearly complete picture of uniform convergence for halfspaces, revealing sharp dimensional and structural thresholds.
Read the full abstract ▾
Machine Learning · Cross-listed from stat.ML
Gaussian mixture models in Hilbert spaces via kernel methods
Daniel López-Montero, Antonio Álvarez-López and Marcos Matabuena
Modern
datasets across many disciplines increasingly consist of time-evolving, potentially infinite-dimensional random objects, such as dynamic functional data, which are naturally modeled in
Hilbert spaces. In these settings, characterizing probability measures, for example, through densities, can be ill-defined or technically challenging. Motivated by clustering applications, we propose a Gaussian mixture framework for
Hilbert-space-valued data based on kernel mean
embeddings and develop efficient optimization
algorithms for estimation. We establish theoretical guarantees showing that the proposed
algorithm is well-defined and that the model yields a dense class of approximations in infinite-dimensional spaces. We evaluate the framework through extensive experiments on diverse data geometries, including $L^2$-functional data and graph signals arising in modern medical applications.
Read the full abstract ▾
Machine Learning
DiBA: Diagonal and Binary Matrix Approximation for Neural Network Weight Compression
Nobutaka Ono
In this paper, we propose DiBA (Diagonal and Binary
Matrix Approximation), a compact
matrix factorization for neural network weight compression. Many components of modern networks, including linear layers, $1\times1$ convolutions,
attention projections, and
embedding layers, have dense
matrix weights. DiBA approximates $A\in\mathbb{R}^{m\times n}$ by $\widehat A=D_1B_1D_2B_2D_3$, where $D_1,D_2,D_3$ are diagonal
matrices and $B_1,B_2$ are $0/1$ binary
matrices. The intermediate dimension $k$ controls the trade-off between theoretical storage and approximation accuracy. For
matrix-vector products, DiBA decomposes dense multiplication into three element-wise scaling operations and two binary mixing operations, reducing the floating-point multiplication count from $mn$ to $m+k+n$. For optimization, we introduce DiBA-Greedy, an alternating solver that combines closed-form least-squares updates for the diagonal factors with exact one-bit improvement tests for the binary factors. We also introduce DiBARD (DiBA with Retuning only Diagonal factors), which replaces dense-
matrix layers by DiBA factors, freezes the binary
matrices, and retunes only the diagonal entries on downstream data. This preserves compact binary mixing without discrete search during adaptation. On 40 dense weight
matrices extracted from public pretrained models, DiBA-Greedy yields consistent SNR improvements as the theoretical storage ratio increases. After DiBA replacement in two component-replacement studies, DiBARD improves DistilBERT/WikiText masked-
token accuracy from 0.4447 to 0.5210 and
Speech Commands test accuracy for an Audio Spectrogram
Transformer from 0.7684 to 0.9781 without reoptimizing the binary factors.
Read the full abstract ▾
Machine Learning · Cross-listed from stat.ML
TabCF: Distributional Control Function Estimation with Tabular Foundation Models
Geping Chen, Chunlin Li, Tianzhong Yang et al. (+2 more)
Instrumental variable (IV) and control function (CF) methods are powerful tools for causal effect estimation in the presence of unmeasured confounding, yet most existing approaches target only mean effects and/or demand substantial fitting and tuning effort. In this paper, we introduce a simple method, TabCF, for control function regression using tabular foundation models, which enables accurate, fast, identification-transparent, and tuning-light causal estimation of distributional quantities, such as interventional means and
quantiles; we also propose a copula-based approximation for multivariate outcomes. TabCF performs favorably against representative methods across a broad range of small- to medium-sized synthetic and real data scenarios. The central message is two-fold: for practitioners, it highlights that TabCF is an effective tool for distributional causal inference; for researchers, it suggests that the proposed approach could be considered a strong baseline for future method development. Code is available at https://github.com/GepingChen/TabCF.
Read the full abstract ▾
All authors Geping Chen, Chunlin Li, Tianzhong Yang, Zhengyuan Zhu, Jing Zhou
Machine Learning
Towards Steering without Sacrifice: Principled Training of Steering Vectors for Prompt-only Interventions
Yuntai Bao, Qinfeng Li, Xinyan Yu et al. (+6 more)
Recently, steering vectors (SVs) have emerged as an effective and lightweight approach to steer behaviors of large
language models (LLMs), among which fine-tuned SVs are more effective than optimization-free ones. However, current approaches to fine-tuned SVs suffer from two limitations. First, they require careful selection of steering factors on a per-SV basis to balance steering effectiveness and generation quality at inference time. Second, they operate as full-sequence SVs (FSSVs), which can sacrifice generation quality regardless of factor selection due to excessive intervention on the model generation process. To address the first limitation, we propose joint training of steering factors and directions, such that post-hoc factor selection is no longer required. Using neural network scaling theory, we find that moderately large initialization sizes and learning rates for steering factors are essential for stability and efficiency of joint training. To tackle the second limitation, we draw inspiration from representation fine-tuning and introduce
Prompt-only SV (PrOSV), an SV that intervenes only on a few
prompt tokens. Our empirical results show that PrOSV outperforms traditional FSSVs on AxBench when using our joint training scheme. We also find that PrOSV achieves a better tradeoff between general model utility and
adversarial robustness than FSSV.
Read the full abstract ▾
All authors Yuntai Bao, Qinfeng Li, Xinyan Yu, Ge Su, Wenqi Zhang, Liu Yan, Haiqin Weng, Jianwei Yin, Xuhong Zhang
Machine Learning
Physical Fidelity Reconstruction via Improved Consistency-Distilled Flow Matching for Dynamical Systems
Sicheng Ma, Tianyue Yang, Xiuzhe Wu et al. (+1 more)
Reconstructing high-fidelity flow fields from low-fidelity observations is a central problem in scientific machine learning, yet recent
diffusion and flow-matching models typically rely on iterative sampling, making them costly for latency-sensitive workflows such as ensemble
forecasting, real-time visualization, and simulation-in-the-loop inference. We study whether a high-fidelity flow-matching generative model can be compressed into a compact one-step model for fast scientific flow reconstruction. Our approach distills an optimal-transport flow-matching teacher into a one-step consistency model. Low-fidelity observations are incorporated at inference by initializing the generative trajectory from a noised observation along the transport path, allowing an unconditional high-fidelity flow model to perform conditional reconstruction without retraining the teacher. We evaluate this
distillation strategy on three fluid
benchmarks, Smoke Buoyancy, Turbulent Channel Flow, and Kolmogorov Flow, using coarse-to-fine reconstruction as a controlled testbed at field sizes up to $256 \times 256$. Across these settings, the distilled student retains similar performance of the teacher's model on spectrum metrics, while using roughly half as many parameters and achieving a $12\times$ inference speedup over the flow-matching teacher. Under the same training budget, the distilled student also outperforms a one-step consistency model trained directly from scratch by $23.1\%$ in SSIM, showing that teacher
distillation improves training efficiency rather than merely accelerating sampling. These results suggest a promising route for turning future high-capacity scientific generative models into compact reconstruction models that are faster to train, cheaper to run, and easier to deploy.
Read the full abstract ▾
All authors Sicheng Ma, Tianyue Yang, Xiuzhe Wu, Xiao Xue
Machine Learning · Cross-listed from stat.ML
Towards Reliable LLM Evaluation: Correcting the Winner's Curse in Adaptive Benchmarking
Jiefu Zhang, Yang Xu, Haixiang Sun et al. (+3 more)
Adaptive
prompt and program search makes LLM evaluation selection-sensitive. Once
benchmark items are reused inside tuning, the observed winner's score need not estimate the fresh-data performance of the full tune-then-deploy procedure. We study inference for this procedure-level target under explicit tuning budgets. We propose SIREN, a selection-aware repeated-split reporting protocol that freezes the post-search shortlist, separates splitwise selection from held-out evaluation, and uses an item-level Gaussian multiplier bootstrap for uncertainty quantification. In a fixed-shortlist regime with smooth stabilized selection, the
estimator admits a first-order item-level representation, and the bootstrap yields valid simultaneous inference on a finite budget grid. This supports confidence intervals for procedure-performance curves and pre-specified equal-budget and cross-budget comparisons. Controlled simulations and MMLU-Pro tuning experiments show that winner-based reporting can be optimistic and can change deployment conclusions, while SIREN remains close to the finite-sample reporting target.
Read the full abstract ▾
All authors Jiefu Zhang, Yang Xu, Haixiang Sun, Zihan Zhou, Tianyu Cao, Vaneet Aggarwal
Machine Learning
Training Transformers for KV Cache Compressibility
Yoav Gelberg, Yam Eitan, Michael Bronstein et al. (+2 more)
Long-context
language modeling is increasingly constrained by the Key-Value (KV) cache, whose
memory and decode-time access costs scale linearly with the prefix length. This bottleneck has motivated a range of context-compression methods, from
token-level summarization to recent optimization-based KV compression methods. These post-hoc methods operate on the KV cache of a fixed pretrained model, so their effectiveness is fundamentally limited by how well the model's internal representations can be compressed. In this work, we formalize the notion of KV compressibility and show that it is a property of the learned representations, rather than of the context alone. We prove that almost any sequence-to-vector function admits both highly compressible and inherently non-compressible
transformer implementations, highlighting the need to guide
transformers toward compressible representations during training. Motivated by this, we propose KV-Compression Aware Training (KV-CAT), a continued pretraining procedure that incentivizes the emergence of compressible representations. We introduce a train-time KV sparsification policy that masks KV slots during training. This forces the model to use fewer KV slots and encourages it to learn representations amenable to post-hoc compression. Empirically, we show that KV-CAT improves the quality-budget tradeoff of downstream compression methods across
retrieval, long-context question answering, and perplexity-based evaluation of compressed-prefix continuation.
Read the full abstract ▾
All authors Yoav Gelberg, Yam Eitan, Michael Bronstein, Yarin Gal, Haggai Maron
Machine Learning
Sharper Guarantees for Misspecified Kernelized Bandit Optimization
Davide Maran and Csaba Szepesvári
Existing guarantees for misspecified kernelized bandit optimization pay for misspecification through kernel complexity: in generic offline bounds, the misspecification level $\varepsilon$ is multiplied by $\sqrt{d_\mathrm{eff}}$, where $d_\mathrm{eff}$ is the kernel effective dimension, while in online regret bounds, the corresponding penalty is $\sqrt{\gamma_n}\,n\varepsilon$, where $\gamma_n$ is the maximum information gain after $n$ rounds of interaction. In this work, we show that, for a large class of kernels, the misspecification amplification can be reduced to logarithmic or polylogarithmic growth. In the offline setting, we first prove high-probability simple-regret bounds whose misspecification term is governed by a spectral Lebesgue constant. This yields logarithmic amplification for one-dimensional monotone spectra and polylogarithmic amplification for multivariate
Fourier-diagonal product kernels. In the online setting, we modify a domain-splitting
algorithm and prove a cumulative regret bound of $\widetilde{\mathcal O}(\sqrt{\gamma_n n}+n\varepsilon)$ under mild localized eigendecay assumptions, removing the extra $\sqrt{\gamma_n}$ factor from the misspecification term. The common principle is localization: spectral localization controls the Lebesgue constant of the offline approximation operator, while domain splitting implements the spatial analogue of this mechanism in the online setting, preventing local misspecification errors from being amplified globally.
Read the full abstract ▾
Machine Learning
Selective Successor Eligibility Traces for RLVR
Chaoli Mou, Xinning Chen, Zhan Zhuang et al. (+3 more)
Reinforcement Learning with Verifiable Rewards (RLVR) has become a key approach for improving the
reasoning abilities of large
language models. However, widely used critic-free
algorithms such as Group Relative Policy Optimization (GRPO) suffer from coarse-grained temporal credit assignment, as they assign the same trajectory-level advantage to all
tokens and fail to distinguish the contributions of different
reasoning steps. To address this challenge, we propose Selective Successor Eligibility Traces (S-trace), an critic-free eligibility trace method that enables more effective credit assignment. S-trace introduces successor importance reweighting to recover downstream credit signals under PPO-style clipping and propagates them to selected historical
tokens through
entropy-based
token selection. Theoretically, we establish an upper bound on its discrepancy from the raw eligibility traces-based policy gradient
estimator, jointly controlled by the KL divergence and the trace decay factor $\lambda$. Experiments show that S-trace achieves absolute gains in average pass@16 of 1.80% and 6.17% over GRPO, and 2.32% and 0.87% over trace-style GRPO($\lambda$), on Qwen3-1.7B and Qwen3-4B, respectively, while maintaining robust gains on Qwen3-8B and producing shorter responses.
Read the full abstract ▾
All authors Chaoli Mou, Xinning Chen, Zhan Zhuang, Qinghe Zheng, Deyuan Kong, Yu Zhang
Machine Learning
Uncertainty Estimation via Hyperspherical Confidence Mapping
Eunseo Choi, Ho-Yeon Kim, Jaewon Lee et al. (+3 more)
Quantifying uncertainty in neural network predictions is essential for high-stakes domains such as autonomous driving, healthcare, and manufacturing. While existing approaches often depend on costly sampling or restrictive distributional assumptions, we propose Hyperspherical Confidence Mapping (HCM), a simple yet principled framework for sampling-free and distribution-free uncertainty estimation. HCM decomposes outputs into a magnitude and a normalized direction vector constrained to lie on the unit hypersphere, enabling a novel interpretation of uncertainty as the degree of violation of this
geometric constraint. This yields deterministic and interpretable estimates applicable to both regression and classification. Experiments across diverse
benchmarks and real-world industrial tasks demonstrate that HCM matches or surpasses ensemble and evidential approaches, with far lower inference cost and stronger confidence-error
alignment. Our results highlight the power of
geometric structure in uncertainty estimation and position HCM as a versatile alternative to conventional techniques.
Read the full abstract ▾
All authors Eunseo Choi, Ho-Yeon Kim, Jaewon Lee, Taeyong jo, Myungjun lee, Heejin Ahn
Machine Learning · Cross-listed from cs.AI
From Coordinate Matching to Structural Alignment: Rethinking Prototype Alignment in Heterogeneous Federated Learning
Xinghao Wu, Jianwei Niu, Guogang Zhu et al. (+3 more)
Heterogeneous
federated learning (HtFL) aims to enable collaboration among clients that differ in both data distributions and model architectures. Prototype-based methods, which communicate class-level feature centers (prototypes) instead of full model parameters, have recently shown strong potential for HtFL. Existing prototype-based HtFL methods typically reuse the MSE-based or cosine-based
alignment mechanism developed for homogeneous FL when aligning client-specific representations with global prototypes. These approaches are essentially coordinate
alignment, where representations of clients are forced to match the global prototypes in the
embedding space in an element-wise manner. Such
alignment implicitly assumes that all clients should map their representations into the feature subspace defined by the global prototypes. This assumption is reasonable in homogeneous FL, where all clients share the same feature extractor. However, it becomes problematic in HtFL, since heterogeneous feature extractors naturally induce client-specific feature subspaces, and forcing all clients to optimize within a single global subspace unnecessarily suppresses their learning capacity. We observe that coordinate
alignment implicitly couples two distinct objectives: aligning inter-class semantic structure, which is directly beneficial for classification, and enforcing a shared feature basis, which is unnecessary and even harmful under model heterogeneity. Building on this insight, we design FedSAF, which shifts the
alignment objective from absolute coordinates to inter-class relational structure. We demonstrate that structural
alignment consistently outperforms coordinate
alignment in heterogeneous settings. Experiments on multiple
benchmarks show that our structural
alignment outperforms state-of-the-art prototype-based HtFL methods by up to 3.52%.
Read the full abstract ▾
All authors Xinghao Wu, Jianwei Niu, Guogang Zhu, Xuefeng Liu, Shaojie Tang, Jiayuan Zhang
Machine Learning
Knowing but Not Correcting: Routine Task Requests Suppress Factual Correction in LLMs
Zixuan Chen, Hao Lin, Zizhe Chen et al. (+6 more)
LLMs reliably correct false claims when presented in isolation, yet when the same claims are embedded in task-oriented requests, they often comply rather than correct. We term this failure mode correction suppression and construct a
benchmark of 300 false premises to systematically evaluate it across eight models. Suppression rates range from 19% to 90%, with four models exceeding 80%, establishing correction suppression as a prevalent and severe phenomenon. Mechanistic analysis reveals that suppression is not a knowledge failure: the model registers the error internally but task context diverts early-layer
attention from the false claim as output intent crystallizes toward compliance at middle layers. We characterize this as knowing but not correcting – suppression occurs at response selection rather than knowledge encoding. Guided by this mechanism, we propose two training-free interventions. Correction Direction Steering (CDS) estimates a correction-compliance direction from matched pairs and injects it at middle layers before output intent crystallizes. Dynamic Payload Amplification (DPA) localizes payload
tokens via
attention divergence between early and late layers and amplifies their representation at the final layer, requiring no calibration data. Experiments on Qwen3.5-9B and LLaMA3.1-8B show both methods substantially improve factual strictness. CDS achieves the highest correction rate on Qwen3.5-9B (0%$\to$58.2%). DPA is the only method that preserves or improves
reasoning capability on both models. These findings introduce factual strictness – the willingness to uphold accuracy against contextual pressures – as a new dimension of model reliability.
Read the full abstract ▾
All authors Zixuan Chen, Hao Lin, Zizhe Chen, Yizhou Tian, Garry Yang, Depeng Wang, Ya Guo, Huijia Zhu, James Cheng
Machine Learning · Cross-listed from quant-ph
Architecture Shape Governs QNN Trainability: Jacobian Null Space Growth and Parameter Efficiency
Michael Poppel, David Bucher, Maximilian Zorn et al. (+5 more)
Variational
quantum circuits with angle encoding implement truncated
Fourier series, and architectures arranging $N$
qubits with $L$ encoding layers each – sharing encoding budget $E = NL$ – generate identical frequency spectra, identical frequency redundancy, and require the same minimum parameter count for coefficient control. Despite this equivalence, trainability varies substantially with architecture shape $(N,L)$ at fixed $E$. We identify structural rank deficiency of the coefficient matching Jacobian $J$ as the mechanism responsible. For serial single-
qubit architectures, we prove $\mathrm{rank}(J) \leq 2L+1$ regardless of parameter count $P$, with $\dim(\ker J) \geq P-(2L+1)$ growing without bound – a phenomenon we term structural gradient starvation: a growing fraction of parameters become structurally decoupled from the loss as $P$ increases at fixed $L$. Parallel architectures avoid this via independent phase trajectories, ensuring $\sigma_{\min}(J^{(\mathrm{par})}) > 0$ generically for $P \leq 2E+1$, so no parameter lies in $\ker J$. For practitioners, we further show that the two natural routes to increasing parameter count have fundamentally different effects: adding feature map (FM) layers monotonically strengthens the Jacobian QFIM
eigenvalue spectrum and achieves $R^2 \geq 0.95$ with $1.6$–$2.2\times$ fewer parameters than adding trainable blocks across all tested architectures, while trainable blocks improve training only through the classical interpolation mechanism with no
quantum-specific benefit.
Read the full abstract ▾
All authors Michael Poppel, David Bucher, Maximilian Zorn, Markus Baumann, Sebastian Wölckert, Claudia Linnhoff-Popien, Philipp Altmann, Jonas Stein
Machine Learning
Near-Policy: Accelerating On-Policy Distillation via Asynchronous Generation and Selective Packing
Miao Rang, Zhenni Bi, Hang Zhou et al. (+6 more)
Standard knowledge
distillation for autoregressive models often suffers from distribution mismatch. While on-policy methods mitigate this by leveraging student-generated outputs, they rely on computationally expensive Reinforcement Learning (RL) frameworks. To improve efficiency, we propose Near-Policy
Distillation (NPD), an asynchronous approach that decouples student generation from training. This reformulation enables Supervised Fine-Tuning (SFT) with sequence packing. However, asynchronous updates inevitably introduce policy lag and sample noise, which can cause the behavior to drift from near-policy toward off-policy. To counteract this without sacrificing efficiency, NPD integrates sparse student updates and the $\Delta$-IFD filtering mechanism, a heuristic sample selection mechanism that empirically stabilizes the optimization trajectory. By filtering extreme out-of-distribution samples, $\Delta$-IFD prevents noise from dominating the gradients, ensuring updates remain within a safe proximal learning zone. Empirically, the NPD framework achieves a 8.1x speedup over on-policy baselines and outperforms SFT by 8.09%. Crucially, by effectively narrowing the exploration space for subsequent RL, our method enables openPangu-Embedded-1B to reach a state-of-the-art score of 68.73%, outperforming the substantially larger Qwen3-1.7B. Codes will be released soon.
Read the full abstract ▾
All authors Miao Rang, Zhenni Bi, Hang Zhou, Kai Han, Xuechun Wang, An Xiao, Xinghao Chen, Yunhe Wang, Hanting Chen
Machine Learning · Cross-listed from quant-ph
Borja Aizpurua, Sukhbinder Singh, Augustine Kshetrimayum et al. (+2 more)
Large
language models (LLMs) have transformed artificial intelligence, yet classical architectures impose a fundamental constraint: every trainable parameter demands classical
memory that scales unfavourably with model size.
Quantum computing offers a qualitatively different pathway, but practical demonstrations on real hardware have remained elusive for models of practical relevance. Here we show that Cayley-parameterised
unitary adapters –
quantum circuit blocks inserted into the frozen projection layers of pre-trained LLMs and executed on a 156-
qubit IBM
Quantum System Two superconducting processor – improve the perplexity of Llama 3.1 8B, an 8-billion-parameter model in widespread use, by 1.4% with only 6,000 additional parameters and end-to-end inference validated on real
Quantum Processing Unit (QPU). A systematic study on SmolLM2 (135M parameters), chosen for its tractability, reveals monotonically improving perplexity with
unitary block dimension, 83% recovery of compression-induced degradation, and correct answers to questions that both classical baselines fail – with a sharp noise-expressivity phase transition identifying the concrete path to
quantum utility at larger
qubit scales.
Read the full abstract ▾
All authors Borja Aizpurua, Sukhbinder Singh, Augustine Kshetrimayum, Saeed S. Jahromi, Roman Orus
Machine Learning
From Drops to Grid: Noise-Aware Spatio-Temporal Neural Process for Rainfall Estimation
Rafael Pablos Sarabia, Joachim Nyborg, Morten Birk et al. (+1 more)
High-resolution rainfall observations are crucial for weather
forecasting, water management, and hazard mitigation. Traditional operational measurements are often biased and low-resolution, limiting their ability to capture local rainfall. Accurate high-resolution rainfall maps require integrating sparse surface observations, yet existing deep learning densification methods are hindered by rainfall's skewed, localized nature, noise, and limited spatio-temporal fusion. We present DropsToGrid, a Neural Process-based method that generates dense rainfall fields by fusing temporal sequences from noisy, irregularly distributed private weather stations with spatial context from radar. Leveraging multi-scale feature extraction, temporal
attention, and multi-modal fusion, the model produces
stochastic, continuous rainfall estimates and explicitly quantifies uncertainty. Evaluations on real-world
datasets demonstrate that DropsToGrid outperforms both operational and deep learning baselines, generating accurate high-resolution rainfall maps with well-calibrated uncertainty, even when only few stations are available and in cross-regional scenarios.
Read the full abstract ▾
All authors Rafael Pablos Sarabia, Joachim Nyborg, Morten Birk, Ira Assent
Machine Learning · Cross-listed from cs.AI
PREFER: Personalized Review Summarization with Online Preference Learning
Millend Roy, Agostino Capponi and Vineet Goyal
Product reviews significantly influence purchasing decisions on e-commerce platforms. However, the sheer volume of reviews can overwhelm users, obscuring information most relevant to their individual needs. Existing e-commerce summarization systems typically produce generic, static summaries that fail to account for the fact that (i) different users prioritize different product characteristics, and (ii) these preferences may evolve with interactions. To address these challenges, we propose an online learning framework that generates personalized summaries for each user by representing user interests in a latent aspect space and selecting preference-aligned review evidence. Our system then iteratively refines its understanding of user preferences by incorporating feedback on the generated summaries directly over time. We evaluate our framework on Amazon Reviews'23 using controlled feedback simulations and robustness checks with held-out-history, independent-
embedding, and external LLM-based evaluators. These experiments show improvements in preference-aligned evidence selection under the evaluated feedback settings. A separate human-based study finds that scenario-conditioned summaries are preferred over generic (uniformly-aggregated) alternatives in 69.9% of paired comparisons.
Read the full abstract ▾
Machine Learning
Quadratic Objective Perturbation: Curvature-Based Differential Privacy
Daniel Cortild and Coralia Cartis
Objective perturbation is a standard mechanism in differentially private empirical risk minimization. In particular, Linear Objective Perturbation (LOP) enforces
privacy by adding a random linear term, while strong convexity and stability are ensured by an additional deterministic quadratic term. However, this approach requires the strong assumption of bounded gradients of the loss function, which excludes many modern machine learning models. In this work, we introduce Quadratic Objective Perturbation (QOP), which perturbs the objective with a random quadratic form. This perturbation induces strong convexity and enforces stability of the problem through
curvature, thereby enabling
privacy and allowing sensitivity to be controlled through spectral properties of the perturbation rather than assumptions on the gradients. As a result, we obtain $(\varepsilon, \delta)$-differential
privacy under weaker \red{gradient} assumptions. Furthermore, we extend the analysis to account for approximate solutions, showing that
privacy guarantees are preserved under inexact solves. Additionally, we derive utility guarantees in terms of empirical excess risk, and provide a theoretical and numerical comparison to LOP, highlighting the advantages of
curvature-based perturbations. Finally, we discuss algorithmic aspects and show that the resulting problems can be solved efficiently using modern splitting schemes.
Read the full abstract ▾
Machine Learning
VisMMOE: Exploiting Visual-Expert Affinity for Efficient Visual-Language MoE Offloading
Cheng Xu, Xiaofeng Hou, Jiacheng Liu et al. (+1 more)
Large-scale
vision-
language mixture-of-experts (VL-MoE) models provide strong
multimodal capability, but efficient deployment on
memory-constrained platforms remains difficult. Existing MoE offloading systems are largely designed for text-centric workloads and become much less effective for visual-heavy inputs, where large numbers of visual
tokens induce broader and less predictable expert accesses. We present VisMMoE, a VL-MoE offloading system built on a single systems insight:
pruning redundant visual
tokens can improve offloading not only by reducing computation, but also by reshaping expert demand. We refer to this effect as visual-expert affinity:
token pruning makes expert accesses more concentrated within layers and more stable across layers, producing a smaller and more predictable expert working set. Guided by this insight, VisMMoE combines affinity-aware
token compression, lookahead expert prediction, and cache/pipeline orchestration to improve expert locality and prefetch effectiveness under tight
memory budgets. We implement VisMMoE on multiple frameworks and evaluate it on representative VL-MoE models and
benchmarks. VisMMoE improves end-to-end inference performance by up to 2.68x and 1.61x, respectively, over strong baselines for today's VL-MoE deployments while maintaining competitive accuracy.
Read the full abstract ▾
All authors Cheng Xu, Xiaofeng Hou, Jiacheng Liu, Chao Li
Machine Learning
VARS-FL: Validation-Aligned Client Selection for Non-IID Federated Learning in IoT Systems
Mohamed Lakas and Mohamed Amine Ferrag
Federated learning (FL) systems typically employ stateless client selection, treating each communication round independently and ignoring accumulated evidence of client contribution quality. Under non-IID data, this leads to slow convergence and unstable training, particularly when selection relies on local proxies (e.g., training loss) that are misaligned with the global optimization objective. These challenges are especially pronounced in Internet of Things (IoT) and Industrial IoT (IIoT) environments, where data is highly heterogeneous and distributed across devices observing different traffic patterns. In this paper, we propose VARS-FL (Validation-Aligned Reputation Scoring for
Federated Learning), a client selection framework that quantifies each client's contribution using the reduction in server-side validation loss induced by its update. These per-round signals are aggregated into a Reputation score that combines a sliding-window average of recent contributions with a logarithmically scaled participation term, enabling robust exploration-exploitation selection. VARS-FL requires no changes to local training or aggregation and remains fully compatible with standard FedAvg. We evaluate VARS-FL on a 15-class non-IID IoT intrusion detection task using the Edge-IIoTset
dataset, with 100 clients across multiple seeds, and compare it against FedAvg, Oort, and Power-of-Choice. VARS-FL consistently improves accuracy, F1-Macro, and loss, while accelerating convergence (up to 36% fewer rounds to reach 80% accuracy). These results demonstrate that validation-aligned, history-aware client selection provides a more reliable and efficient training process for
federated learning in heterogeneous IoT environments.
Read the full abstract ▾
Machine Learning · Cross-listed from cs.CL
Beyond Steering Vector: Flow-based Activation Steering for Inference-Time Intervention
Zehao Jin, Ruixuan Deng, Junran Wang et al. (+2 more)
Activation steering has emerged as a promising alternative for controlling
language-model behavior at inference time by modifying intermediate representations while keeping model parameters frozen. However, large-scale evaluations such as AxBench show that existing steering methods are often outperformed by simple in-context prompting and generalize poorly to unseen concepts. We hypothesize that these limitations arise from unvalidated simplifying assumptions shared across prior methods, which typically restrict steering interventions to fixed, single-step, position-invariant transforms. We propose FLAS (Flow-based Activation Steering), which learns a general, concept-conditioned velocity field $v_t(h,t,c)$ that transports unsteered activations to steered ones without relying on these assumptions. On AxBench, FLAS is the first learned method to consistently outperform prompting, reaching held-out harmonic means of $1.015$ on Gemma-2-2B-IT and $1.113$ on Gemma-2-9B-IT without per-concept tuning. Analysis of the learned flow shows curved, multi-step,
token-varying trajectories, which suggests that previous hypotheses on activation space geometry might be incomplete.
Read the full abstract ▾
All authors Zehao Jin, Ruixuan Deng, Junran Wang, Xinjie Shen, Chao Zhang
Machine Learning · Cross-listed from cs.CV
MTL-MAD: Multi-Task Learners are Effective Medical Anomaly Detectors
Bogdan Alexandru Bercean, Florinel Alin Croitoru, Vlad Hondru et al. (+3 more)
Anomaly detection in medical images is a challenging task, since anomalies are not typically available during training. Recent methods leverage a single pretext task coupled with a large-scale pre-trained model to reach state-of-the-art performance. Instead, we propose to learn multiple self-supervised and pseudo-labeling tasks from scratch, using a joint model based on Mixture-of-Experts (MoE). By carefully integrating multiple proxy tasks, the joint model effectively learns a robust representation of normal anatomical structures, so that anomaly scores can be derived based on how well the multi-task learner (MTL) solves each task during inference. We perform comprehensive experiments on BMAD, a recent
benchmark that comprises a broad range of medical image modalities. The empirical results indicate that our multi-task learner is an effective anomaly detector, outperforming all state-of-the-art competitors on BMAD. Moreover, our model produces interpretable anomaly maps, potentially helping physicians in providing more accurate diagnoses.
Read the full abstract ▾
All authors Bogdan Alexandru Bercean, Florinel Alin Croitoru, Vlad Hondru, Ciprian Mihai Ceausescu, Andreea Iuliana Ionescu, Radu Tudor Ionescu
Machine Learning
RepFlow: Representation Enhanced Flow Matching for Causal Effect Estimation
Yifei Xie and Jian Huang
Estimating causal effects from observational data has become increasingly critical in diverse fields including healthcare, economics, and social policy. The fundamental challenge in causal inference arises from the missing counterfactuals and the selection bias. Existing methods are largely limited to point estimates and lack the capacity for distribution modeling. In this work, we propose RepFlow, a novel framework that formulates causal effect estimation as a joint optimization problem integrating representation learning with Conditional Flow Matching (CFM). RepFlow mitigates selection bias by minimizing the entropically regularized Wasserstein distance between treated and control representations. To enhance numerical stability, we further introduce an $L_2$ normalization constraint on latent representations. This balanced representation enables the flow model to accurately capture the distribution of potential outcomes. Extensive experiments across a wide range of
benchmarks demonstrate that RepFlow consistently outperforms existing methods in both point and distributional causal effect estimation.
Read the full abstract ▾
Machine Learning · Cross-listed from cs.CV
DBMSolver: A Training-free Diffusion Bridge Sampler for High-Quality Image-to-Image Translation
Sankarshana Venugopal, Mohammad Mostafavi and Jonghyun Choi
Diffusion-based image-to-image (I2I) translation excels in high-fidelity generation but suffers from slow sampling in state-of-the-art
Diffusion Bridge Models (DBMs), often requiring dozens of function evaluations (NFEs). We introduce DBMSolver, a training-free sampler that exploits the semi-linear structure of DBM's underlying SDE and ODE via exponential integrators, yielding highly-efficient 1st- and 2nd-order solutions. This reduces NFEs by up to 5x while boosting quality (e.g., FID drops 53% on DIODE at 20 NFEs vs. 2nd-order baseline). Experiments on inpainting, stylization, and semantics-to-image tasks across resolutions up to 256x256 show DBMSolver sets new SOTA efficiency-quality tradeoffs, enabling real-world applicability. Our code is publicly available at https://github.com/snumprlab/dbmsolver.
Read the full abstract ▾
Machine Learning · Cross-listed from stat.ML
Tuning Derivatives for Causal Fairness in Machine Learning
Filip Edström, Guilherme W. F. Barros, Tetiana Gorbach et al. (+1 more)
Artificial-intelligence systems are becoming ubiquitous in society, yet their predictions typically inherit biases with respect to protected attributes such as race, gender, or age. Classical
fairness notions, most notably Statistical Parity (SP), demand that predictions be independent of the protected attributes, but are overly restrictive when these attributes influence mediating variables that are considered business necessities. Recent causal formulations relax SP by distinguishing allowed from not-allowed causal paths and by complementing SP with Predictive Parity (PP), requiring the predictor to replicate the legitimate influence of business-necessities. Existing path-based definitions are mainly practical when applied to categorical attributes. This paper introduces a new framework for
fairness in structural causal models that is tailored to continuous protected attributes. We formalize SP and PP through path-specific partial derivatives, establish conditions under which these criteria coincide with prior causal definitions, and characterize when a fair predictor, one that satisfies SP along not-allowed paths while achieving PP along allowed paths, exists. Building on this theory, we propose a fair tuning
algorithm that either constructs such a predictor or, when not possible, allows for a trade-off between SP and PP. We present experiments on simulated and real data to evaluate our proposal, compare it with previously proposed methods, and show that it performs better when PP is considered.
Read the full abstract ▾
All authors Filip Edström, Guilherme W. F. Barros, Tetiana Gorbach, Xavier de Luna
Machine Learning · Cross-listed from stat.ML
CITE: Anytime-Valid Statistical Inference in LLM Self-Consistency
Hirofumi Ota, Naoto Iwase, Yuki Ichihara et al. (+2 more)
Large
language models often improve
reasoning by sampling multiple outputs and aggregating their final answers, but precise and efficient control of error levels remains a challenging task. In particular, deciding when to stop sampling remains difficult when the stopping rule is data-dependent and the set of possible answers is not known in advance. We study anytime-valid certification of a prespecified target answer as the unique mode of the model's response distribution, a guarantee distinct from answer correctness. We propose the Certification by Intersection-union Testing with E-processes (CITE)
algorithm, which provably controls false certification at any prescribed level under arbitrary data-driven stopping, without requiring prior knowledge of the answer category set. We also prove an category-set-size-free stopping-time rate, establish matching minimax lower bounds up to constants in the main regime, and extend the construction to confidence-weighted voting. Simulations and LLM self-consistency experiments show empirical error control and improved certification in diffuse-tail settings.
Read the full abstract ▾
All authors Hirofumi Ota, Naoto Iwase, Yuki Ichihara, Junpei Komiyama, Masaaki Imaizumi
Machine Learning
Retain-Neutral Surrogates for Min-Max Unlearning
Junhao Cai, Dohun Kim, Dowon Kim et al. (+4 more)
Machine unlearning seeks to remove the influence of designated training data while preserving performance on the remaining data. Approximate unlearning can be viewed as a local editing problem; in min-max unlearning, the key local object is the surrogate point at which the retain objective is evaluated. When forget and retain gradients are strongly aligned, an unconstrained forget-maximizing perturbation can move to a surrogate point that increases retain loss. We propose Retain-Orthogonal Surrogate Unlearning (ROSU), which constrains the inner surrogate construction by maximizing first-order forget gain subject to zero first-order retain change under a fixed perturbation budget. This yields a closed-form retain-orthogonal perturbation, a lightweight transported outer update, and amplification along the retain-neutral direction. Our analysis establishes (i) a
curvature-controlled second-order bound on retain damage, (ii) a positive-
alignment regime in which ROSU strictly reduces surrogate retain loss relative to standard min-max perturbations, and (iii) near-equivalence when the two gradients are nearly orthogonal. Across
vision and
language benchmarks (CIFAR-10/100, Tiny-ImageNet, TOFU, WMDP), the empirical pattern follows this geometry: ROSU gives its clearest gains in high-coupling regimes while remaining competitive elsewhere.
Read the full abstract ▾
All authors Junhao Cai, Dohun Kim, Dowon Kim, Sung Il Choi, Chengjun Jin, Juhyun Park, Changhee Joo
Machine Learning
QuadraSHAP: $\epsilon$-Exact Shapley Values for Product Games in Logarithmic Parallel Time
Majid Mohammadi, Grigory Reznikov, Pavel Sinitcyn et al. (+2 more)
We introduce QuadraSHAP, a method for $\epsilon$-exact Shapley computation in product games, cooperative games whose coalition values factorize across players. Given a tolerance $\epsilon\ge0$, the method determines a computational budget before evaluation, guaranteeing an absolute attribution error of at most $\epsilon$ for every feature in exact arithmetic. By extending to weighted sums of product games, the framework supports baseline and empirical interventional attribution across a broad class of models, including log-link regression models, Cox proportional-hazards models, odds-scale classifiers, product-kernel machines, and tree-based models. For a $d$-player product game, we replace the exponentially large coalition sum by a one-dimensional integral of a
polynomial of degree at most $d-1$. Gauss–Legendre quadrature with $m_q=\lceil d/2\rceil$ nodes therefore recovers exact Shapley values when $\epsilon=0$. For positive tolerances, a computable error bound derived from the game factors decays geometrically with the node count and can certify substantially smaller budgets. To support efficient evaluation at scale, we share computations across features and evaluate products in log-space with sign tracking, mitigating intermediate overflow and underflow. Given the quadrature rule, computing all $d$ attributions requires $O(d\,m_q)$ work per product game, and admits $O(\log d)$ parallel time with sufficient processors. On a survival analysis problem with $396{,}065$ features, the exact quadrature configuration computes all feature attributions in approximately five minutes of GPU evaluation per explanation, at a scale where exhaustive coalition enumeration is infeasible. Explicit error control enables a further reduction: setting $\epsilon=10^{-6}$ selects a smaller, theoretically certified quadrature budget and reduces the mean evaluation time to $0.36$ seconds per explanation.
Read the full abstract ▾
All authors Majid Mohammadi, Grigory Reznikov, Pavel Sinitcyn, Krikamol Muandet, Siu Lun Chau
Machine Learning · Cross-listed from cs.AI
XDecomposer: Learning Prior-Free Set Decomposition for Multiphase X-ray Diffraction
Hanyu Gao, Bin Cao, Yunyue Su et al. (+2 more)
Multiphase powder X-ray diffraction (PXRD) analysis remains a fundamental bottleneck in structure identification, as real-world synthesis often produces complex mixtures whose constituent phases (components) cannot be reliably disentangled. While recent advances in representation-based
crystal retrieval and generation suggest the possibility of inferring structures directly from PXRD, existing approaches largely assume single-phase inputs and break down in multiphase settings. Here, we present XDecomposer, a prior-free framework for joint decomposition and identification of multiphase XRD patterns without requiring candidate phase lists, structural templates, or prior knowledge of phase number. We formulate multiphase diffraction analysis as a set prediction problem, where the model infers an unordered set of phase-resolved components, their mixture proportions, and corresponding structural representations within a unified architecture. A phase-query-driven decomposition mechanism, together with diffraction-consistent physical reconstruction, enables accurate source separation while preserving crystallographic fidelity. Extensive experiments on both simulated and experimental
datasets show that XDecomposer substantially improves reconstruction accuracy and phase identification across diverse chemical systems, while maintaining strong generalization to unseen mixtures. These results provide a practical route toward data-driven, source-resolved multiphase XRD analysis and reduce long-standing dependence on prior-guided iteratively phase matching. The code is openly available at https://github.com/Licht0812/XDecomposer
Read the full abstract ▾
All authors Hanyu Gao, Bin Cao, Yunyue Su, Tong-Yi Zhang, Qiang Liu