Machine Learning
Only Project Once: Projection-Adaptive Loss for Exact Constraint Satisfaction
Tim Aebersold, Soheyl Massoudi and Mark Fuge
Precise constraint satisfaction is a prerequisite to deploying learned models in many areas, motivating methods that repair raw neural predictions with a repair procedure. Current methods unroll multiple repair steps in training and softly penalize constraint violations that remain after the unroll. This is compute- and
memory-intensive, lacks robustness when the repair fails to converge, and surrenders most of the constraint satisfaction work to the repair. Our central finding is that, contrary to common practice, a single detached projection step suffices in training. We accomplish this with a Projection-Adaptive Loss (PAL), which uses the constraint residual after this single step to adaptively weigh constraint penalties on the raw prediction. In experiments, PAL is the only method that retains virtually perfect feasibility on extremely nonlinear constraints, and matches or outperforms current methods on synthetic and engineering
benchmarks. Because it only requires a single detached projection step, PAL trains 2.5x faster than the canonical repair-based method (DC3) on its own ACOPF
benchmark. PAL can also be trained when constraints are expensive to evaluate (e.g., via neural surrogates), a setting where current unrolled methods are
memory-intractable.
Read the full abstract ▾
Machine Learning · Cross-listed from cs.FL
From Transformers to Weighted Automata: Towards the Verification of Large Language Models
Smayan Agarwal, Aslah Ahmad Faizi, Shobhit Singh et al. (+1 more)
Large
language models (LLMs) are increasingly deployed in safety-critical settings, yet their black-box nature makes it difficult to provide formal guaranties about their behavior. Existing verification approaches rely primarily on empirical probing and testing, leaving open the question of how to reason rigorously about general-purpose trans- former architectures. In this work, we establish a principled bridge between
transformers and weighted automata, a classical model from formal
language theory. This connection enables us to transfer verification tools from automata the- ory to the analysis of LLMs. Our contributions are twofold: First, we develop a formal correspondence between
transformer architectures and weighted automata over reals, showing how distributional properties of LLMs can be captured within this framework. Second, we introduce an identity testing
algorithm for weighted automata that provides a statis- tical method for distinguishing whether two
stochastic models define the same distribution up to a tolerance threshold. This work provides the first formal bridge between modern neural se- quence models and classical automata theory, clarifying both the poten- tial and the computational challenges for rigorous LLM verification.
Read the full abstract ▾
All authors Smayan Agarwal, Aslah Ahmad Faizi, Shobhit Singh, Aalok Thakkar
Machine Learning · Cross-listed from stat.ML
Gradient-Free Sampling from Generative Models via Stochastic Bounded Extremum Seeking
Alexander Scheinker
We introduce a sampling approach for energy- and score-based generative models that requires no gradient evaluations of the model. Replacing the drift term that would normally contain the score $\nabla_\mathbf{x} \log p_\theta(\bf{x})$ with a high-frequency dithered cosine of the model's value, $\sqrt{\alpha\omega}\,\cos(\omega t + k \log p_\theta(\bf{x}))$, produces, in the high-frequency averaging limit, Langevin Markov chain Monte Carlo for energy-based models and the reverse-time SDE of score-based
diffusion. We prove that trajectories of the dithered Itô SDE converge to those of the target SDE, driven by the same Brownian motion, uniformly on compact time intervals in probability, by an averaging argument that extends bounded extremum seeking (ES) to Itô processes, with an explicit $O(\omega^{-1/2})$ mean-square rate under global bounds. The approach is not confined to smooth targets: it extends to $C^{1,1}$ energies with discontinuous
curvature (without ellipticity requirement) and to Sobolev energies whose Hessians exist only off measure zero sets; for Lipschitz energies with gradient kinks the averaged limit remains well posed; the Krylov-Röckner integrability class is the boundary of provability. The approach provides a hard a priori bound on the per-step update rate and applies to explicitly time-varying targets on finite horizons. Gradient-free pixel-space sampling is not competitive with well-tuned backpropagation-based samplers at practical evaluation budgets; the regime where the approach offers an advantage is latent-space sampling when the model is a black box and the target drifts in time. We demonstrate latent-space tracking for time-varying images on CelebA-HQ ($256{\times}256$) from limited 1D projection measurements and latent-space EBM sampling on CIFAR-10.
Read the full abstract ▾
Machine Learning
Coupling Noisy Pairwise Knowledge to the DAG Posterior for Causal Discovery
Guoliang Xu and James E Corter
External causal reports can improve structure learning from limited observations, but their reliability varies across sources and variable pairs. We introduce HB-NoisyKG, a
Bayesian framework that combines observational data with repeated causal reports from sources such as large
language models. Each report is a noisy observation of a direct pair state implied by one DAG. A feature-conditioned Beta prior pools information about pair reliability, and a shared error
matrix captures systematic mistakes. Alternating inference uses the graph posterior to refine reliability estimates, which determine how reports influence subsequent graph updates. The report likelihood uses only graph pair-state marginals, so the same observation layer supports discrete and continuous likelihoods in graph-only and joint inference. Against an 80-restart no-KG baseline, HB uses at most 80 total restarts and lowers mean SHD from 22.39 to 16.06 on five discrete
benchmarks. On a physical light tunnel with random variable IDs and retained descriptions, HB lowers SHD from 39.00 for no-KG to 27.30. On continuous Sachs, graph-only BGe raises AUROC by 0.121 over no-KG Top-K. In a controlled synthetic study, continued updating also reduces mean reliability estimation error and held-out report log loss compared with one-time estimation.
Read the full abstract ▾
Machine Learning · Cross-listed from physics.ao-ph
ClimateBench v2.0: Probabilistic Climate Model Benchmarking
Duncan Watson-Parris, Willa Tobin, Aytaç Paçal et al. (+32 more)
We present ClimateBench v2, a standardized protocol for evaluating
climate models on diagnostics expected to be informative for their skill in projecting mid-century regional temperature and precipitation changes. The protocol is designed to evaluate any physics-based, data-driven, or hybrid
climate model on equal footing using a common set of observational and out-of-distribution tests. We define three tiers of evaluation. Tier I establishes physical credibility through entry-ticket tests of energy conservation, coupled (co-)variability, and basic forced responses. Tier II scores models against post-2015 observations of surface temperature, precipitation, radiative fluxes, sea ice, and key modes of variability using fair CRPS as the primary probabilistic score, complemented by distributional and ensemble-consistency diagnostics. Tier III tests out-of-distribution generalization through paleoclimate simulations spanning the Last Interglacial, Last Glacial Maximum, and Mid-Holocene, and through perfect-model experiments in which data-driven models must predict the future
climate of existing Earth system models from historical data alone. We reserve all observational data after 2015 for testing, and submissions must include multiple ensemble members to enable probabilistic evaluation. This reservation exploits a new opportunity provided by the decade of observations accumulated since the end of the CMIP6 historical experiment, which constitutes an out-of-sample record of forced
climate change (and internal variability) for the current generation of models, and we quantify, in an idealized setting, the information it carries about mid-century warming. We provide the evaluation code, observational reference
datasets, and perfect-model training data as an open
benchmark to drive measurable progress in
climate projection across all modeling approaches.
Read the full abstract ▾
All authors Duncan Watson-Parris, Willa Tobin, Aytaç Paçal, Manuel Schlund, V. Balaji, Kevin Bowman, Chris Bretherton, Peter M. Caldwell, Will Chapman, William D. Collins, Gregory S. Elsaesser, Pierre Gentine, Helene Hewitt, Stephan Hoyer, Ralph Keeling, Nikolay Koldunov, David M. Lawrence, Christian Lessig, Daniel J. Lunt, J. David Neelin, Mike Pritchard, Sarah Purkey, Gavin Schmidt, Tapio Schneider, Michael Schulz, Tiffany Shaw, Isla R. Simpson, Graeme Stephens, Aneesh C. Subramanian, Joao Teixeira, Jessica Tierney, Andrew I. L. Williams, Laure Zanna, Veronika Eyring, Rose Yu
Machine Learning
TMAML: Temporal Model-Agnostic Meta-Learning for Cold-Start Time Series Forecasting
Wannes Janssens, Matthias Bogaert and Dirk Van den Poel
Cold-start forecasting, the task of forecasting a time series with little to no historical data, is a common challenge. Addressing it requires approaches that learn quickly from few datapoints and leverage information from related series, typically through static
covariates, to generalize well to unseen series. While some global forecasting models can generate cold-start predictions by leveraging information shared across multiple series, they are not optimized for out-of-train-set generalization or adaptation from short histories. In this work, we formulate cold-start forecasting as a few-window learning problem and introduce Temporal Model-Agnostic Meta-Learning (TMAML), which tailors the model-agnostic meta-learning
algorithm, originally developed for few-shot adaptation of neural networks, to deep time series forecasting. TMAML constructs meta-tasks as temporally consistent support-query windows and pairs them with a temporal meta-training and meta-testing procedure, yielding forecasting models that are explicitly optimized for cold-start forecasting. We instantiate TMAML on the Temporal Fusion
Transformer (TFT) and present an initial empirical analysis of forecast accuracy and calibration across three cold-start scenarios: TMAML consistently outperforms or matches a standard ERM-trained TFT, yields better-calibrated forecasts than naive on two of the three scenarios, but does not consistently outperform naive on probabilistic forecast accuracy.
Read the full abstract ▾
Machine Learning · Cross-listed from cs.AI
Label Agreement Does Not Measure Authorization
Amir Sabbaghziarani, Bradley Thomas Baker, Theodore J. LaGrow et al. (+1 more)
Many groups now delegate label ontology and metadata harmonization to agentic LLM pipelines. We built one and audited it. Our aggregate scores looked healthy, but the pipeline kept failing in ways they did not show, so we set out to find what they hid. Label agreement asks whether a proposed label matches a reference. It does not ask whether the
agent was entitled to propose it, whether the output was complete enough to act on, or whether the label moved when the evidence moved. We measured those three separately on COBRE and FBIRN, two schizophrenia and control neuroimaging cohorts from different consortia, and they come apart, from label agreement and from each other. Showing the
agent an upstream proposal barely moves label agreement, 0.857 to 0.870, while agreement on the chosen action doubles, 0.409 to 0.830. Output that parses as JSON still drops a required field on 10% of one model's cases and 33% of the other's. And an
agent that replays its first answer scores perfectly on original cases and zero once we change the evidence that decides them. Downstream, a row-order error that none of these metrics reports erases most of the diagnostic signal. So we measure these properties apart, pair each with a control, and gate commitment on the result, which makes failures visible and easy to route to a person. None of this prevents failure. Our reference labels are rule-derived, so agreement with them means consistency, not correctness. Code is available at https://github.com/amir-sbg/Label-Agreement-Does-Not-Measure-Authorization.
Read the full abstract ▾
All authors Amir Sabbaghziarani, Bradley Thomas Baker, Theodore J. LaGrow, Sergey Plis
Machine Learning · Cross-listed from cs.CR
Quantum Machine Learning Protection of Military Quantum Key Distribution Against Cryptographically Camouflaged Attacks
Muhammad Shaheer Bin Junaid
Quantum key distribution proves its protocol secure and says nothing about the hardware beneath it, so military and government operators fielding it for command-and-control keys monitor the channel for implementation attacks, and that monitoring has a blind spot. An adversary with a kleptographic foothold in the generator of a public per-block value \(x=g^v \pmod p\) can hide attacked blocks in honest noise, gating them on a predicate of its discrete logarithm, making detection a discrete logarithm problem that defeats every efficient classical monitor yet yields to a
quantum kernel recovering \(v\) through Shor's
algorithm. I formalise these cryptographically camouflaged attacks, reduce their hardness to an established learning separation, prove a single-frequency fidelity kernel cannot represent an interval predicate, and test them on Ghillie, a decoy-state BB84 simulator with a positive key rate to 142 km. From 10- to 14-bit groups over two seeds, a classical monitor reads 0.458 to 0.516 on camouflaged attacks while the
quantum kernel reads 1.000, and both catch overt attacks above 0.99. Finite-precision recovery under depolarising noise and a hardened predicate lower the
quantum result to 0.916 through 0.983 with the classical monitor at chance, and a feasibility probe on IBM Heron processors tracks the exact kernel within 0.034. A defender can therefore discard precisely the compromised key material, although the advantage is asymptotic, awaits fault tolerance, and holds only when the feature map matches the adversary's predicate, since a low-frequency map reads 0.545 on a residue pattern and 0.982 once aligned.
Read the full abstract ▾
Machine Learning
PhaseGate: Phase-Aware CPU Retrieval Scheduling for On-Device LLMs on Unified Memory
Seoyoon Yum and Sehoon Kim
On-device assistants run GPU-based LLM inference alongside CPU
retrieval on unified-
memory systems. Under a saturated local-
retrieval workload, four concurrent
retrieval workers raise 95th-percentile (p95) decode latency by 60-61% on two M4 systems, whereas prefill latency rises by only 5.7-6.9%. We study LLM phase as an admission signal for independent CPU
retrieval under controlled LLM workloads. PHASEGATE calibrates separate concurrency limits for prefill and decode, selecting four and one on our base-M4 configuration. Under a backlogged queue, it achieves 2.0 times the aggregate
retrieval throughput of the best tested feasible fixed policy, with both p95 LLM latency metrics within 1.25 times their no-
retrieval baselines in all seven held-out runs. A phase-blind control, TimeGate, uses the same two limits on a calibration-derived schedule without observing LLM phase. It achieves similar
retrieval throughput but violates the output-
token latency limit in every run. M2 and M2 Pro Mac minis reproduce the policy ordering, while output-length sweeps show that the advantage narrows as decode occupies more of each request.
Read the full abstract ▾
Machine Learning
Weight Decay and Neuron Condensation: A Three-Stage Analysis of Two-Layer ReLU Networks
Cheng Xu, Pengxiao Lin, Zhangchen Zhou et al. (+1 more)
Weight decay is widely used as a regularization technique in neural network training, yet its role in
neuron condensation (parameter direction
alignment) remains unclear. Starting from a parameter initialization in the neural tangent kernel regime, we characterize training dynamics under weight decay through three stages: rapid fitting, amplitude compression, and
neuron condensation. Using a two-layer ReLU network, we analyze a residual correlation field that governs both
neuron amplitudes and directions. During rapid fitting, the residual approaches a quasi-static
equilibrium maintained by weight decay while the tangent kernel remains nearly unchanged. In the early stage of amplitude compression, kernel decay amplifies the residual correlation field, whose isolated attracting extrema provide candidates for condensation directions. As
neuron amplitudes stabilize, we bound the drift of attracting extrema and demonstrate contraction of
neuron directions around them, leading to
neuron condensation. This staged analysis provides a dynamical understanding of how weight decay promotes a condensed representation, beyond reducing parameter norms.
Read the full abstract ▾
All authors Cheng Xu, Pengxiao Lin, Zhangchen Zhou, Zhi-Qin John Xu
Machine Learning · Cross-listed from cs.NI
Diffusion-Based Stress Testing of Overload Monitoring for Resilient Emergency Cellular Networks Using Internet CDR Proxies
Bilal Hussain, Xiao Tang, Tan Li et al. (+3 more)
Disasters can overload cellular control-plane signaling within minutes, yet fine-grained
Radio Resource Control (RRC) or Next Generation (NG) Application Protocol (NGAP) telemetry is
privacy-sensitive and costly to collect for analytics. Many emergency monitoring pipelines therefore rely on coarse Call Detail Record (CDR) aggregates. We treat Internet activity in CDR grids as a practical proxy for hidden signaling stress under that constraint. We train a lightweight convolutional neural network (CNN) on stylized overload injections, stress-test it with
diffusion-synthesized surges that preserve normal traffic structure, and adapt the detector by retraining on hard synthetic samples. Under stress-test conditions, the default alert threshold fails even though receiver operating characteristic (ROC) curves stay strong: the detector still assigns overloaded
cells a larger overload probability than normal
cells, but those probabilities fall below the default cutoff 0.5 and are labeled normal, so the F1-maximizing threshold – selected post hoc on the same stress-test grids (oracle $\tau^*$) – shifts by $0.32 \pm 0.03$ (operating-point drift). Across three random seeds, hard-sample adaptation raises thresholded performance (F1) from 0% (no alerts at the default cutoff 0.5 on any seed) to $85.67 \pm 14.37$% and ranking from ROC-AUC $0.886 \pm 0.040$ to $0.99996 \pm 0.00007$.
Diffusion-synthesized surges expose threshold fragility that matched-condition training – training and testing on the same stylized injections – hides, and hard-sample adaptation restores usable alerts at the default cutoff. Together, these steps define a reusable pre-deployment stress test for emergency monitors. Internet-only CDR input further supports lightweight AI-native workflows that combine monitoring, recalibration, and adaptation.
Read the full abstract ▾
All authors Bilal Hussain, Xiao Tang, Tan Li, Muhammad Azhar, Danista Khan, Fawad Ahmad
Machine Learning · Cross-listed from physics.soc-ph
All against the machine: the Solo score for rating skill in variable environments
David Reguera, Xavier R. Hoffmann, Irene Pérez et al. (+5 more)
We propose a distribution-free metric to rate individual skill in “player-versus-environment” settings, where participants face heterogeneous tasks without direct opponents. Such settings are common in digital platforms, games, education, finance, and the
benchmark evaluation of AI
agents. They combine high randomness, tasks of widely varying difficulty, and unknown heterogeneity across individuals. Our metric maps each task outcome to a bounded performance score with zero population mean and variance bounded by $1/3$, regardless of the outcome distribution, so that scores are directly comparable across tasks. Aggregating these scores over tasks yields an interpretable skill score for each individual (the “Solo” rating), and a reshuffling null model tests whether that score exceeds what chance alone would produce. We validate the approach on the progression of $2\times10^5$ players in the mobile games Candy Crush Saga and Bubble Witch 3 Saga. The metric identifies high- and low-skill players with high statistical confidence, their classification persists over hundreds of subsequent levels, and a windowed version of the score tracks changes in performance along progression.
Read the full abstract ▾
All authors David Reguera, Xavier R. Hoffmann, Irene Pérez, Pol Colomer-de-Simón, Miquel Masoliver, Xavier Guardiola, Jan Wedekind, Marián Boguñá
Machine Learning
Proximal Causal Learning under Unmeasured Confounding
Ying Tang and Yi Wang
Estimating treatment effects from observational data typically relies on the No Unmeasured Confounding Assumption (NUCA), which rarely holds in practice. Proximal causal learning (PCL) addresses unmeasured confounding via proxy variables, yet existing methods require the proxy variables to be pre-specified. Thus, we propose PCL-U, a framework that learns proxy variables directly from observed
covariates. PCL-U uses neural encoders to decompose
covariates into treatment-inducing, outcome-inducing, and shared proxies, guided by minimax mutual information objectives, and obtains causal estimates through a practical moment-based risk function. Experiments on
benchmarks show that PCL-U matches or outperforms existing baselines. Besides, there are two types of synthetic
datasets with varying dimensions and confounding strengths that illustrate that our method maintains stable estimation accuracy.
Read the full abstract ▾
Machine Learning
Length Generalization Needs Proper Regularization
Pavlo Vasylenko, Matthias Lindemann, André F. T. Martins et al. (+1 more)
Length generalization is the ability of sequential models to perform well on context lengths unseen during training. In this work, we show that the challenge of achieving length generalization is related not only to architectural choices such as positional encoding and the
attention mechanism but also to the training procedure itself. We study how regularization affects length generalization and find that weight decay can hinder extrapolation. In contrast, dropout improves extrapolation when its placement within the architecture is reconsidered. In that regard, we show that the standard placement of dropout before layer normalization introduces a systematic distributional mismatch, and that applying dropout just before the linear projection resolves this issue. For example, a modified SmolLM3 with sliding window
attention, continually pre-trained with dropout, can extrapolate perfectly to 64$\times$ on Needle-in-a-Haystack and far beyond the pre-training context size on RULER and HELMET. Mamba2 also benefits from dropout, suggesting an architecture-agnostic nature of the problem. We further propose Variance-Preserving Affine Dropout (VPAD), a new dropout strategy that substantially reduces the resulting pre-activation variance mismatch, leading to further extrapolation improvements in
transformers.
Read the full abstract ▾
All authors Pavlo Vasylenko, Matthias Lindemann, André F. T. Martins, Marcos Treviso
Machine Learning · Cross-listed from cs.AI
EvoCast: Reliable Autonomous Research Agents for Iterative Forecasting Architecture Evolution
Kaipeng Xu, Xianli Yan, Yan Wang et al. (+2 more)
Deep time-series forecasting models have rapidly diversified, yet adapting them to a specific task still requires extensive expert effort in model selection, mechanism diagnosis, architecture design, implementation, and evaluation. Existing AutoML methods are constrained by predefined search spaces, while general-purpose LLM research
agents lack reliable control over experimental protocols and model promotion. We introduce EvoCast, a fully autonomous research-
agent system for iterative forecasting architecture
evolution. EvoCast first establishes and diagnoses a task-specific baseline through executed mechanism ablations, then generates evidence-grounded research directions from
dataset characteristics, diagnostic results, prior rounds, and failure records. Its central design, cognition-authority separation, assigns open-ended hypothesis generation and code implementation to LLM
agents, while deterministic program authorities control source-edit boundaries, canonical evaluation, and promotion decisions. Experimental outcomes are accumulated as evidence to guide subsequent rounds. Results show that EvoCast completes complex architecture modifications with higher implementation success and lower
agent-side
token/time cost, and develops task-specific architectures that outperform selected baselines, strong forecasting models, and
agent baselines in three real-world forecasting cases. The code is available at https://github.com/18e0-x/EvoCast.
Read the full abstract ▾
All authors Kaipeng Xu, Xianli Yan, Yan Wang, Xiang Liu, Shan Liu
Machine Learning · Cross-listed from cs.AI
ManifoldCache: Training-Free Diffusion Acceleration via Constraint Manifold Caching
Prashant Pandey, Devineni Sri Venkatraya Chowdary and Brejesh Lall
Diffusion models for structured scientific generation must produce samples satisfying hard
geometric constraints imposed by physics, chemistry, or biology, yet inference in these settings is prohibitively slow, demanding hundreds to thousands of neural-function evaluations per sample. We unify eight state-of-the-art models spanning medical volumetrics, molecular conformations,
protein backbone design,
crystal structure prediction, and multi-view 3D scenes under a single abstraction, Constraint-
Manifold Diffusion Models (CMDMs), in which the target distribution is supported on a
manifold defined by an externally specified constraint map. All existing acceleration families fail on this class: quantization exhausts
memory on high-dimensional volumetric operators;
pruning breaks constraint fidelity; fast ODE solvers allow trajectories to drift off the constraint
manifold; and feature-caching heuristics are blind to constraint geometry, inducing mode confusion in the high-noise regime. We introduce ManifoldCache, the first training-free, data-free accelerator designed from first principles for CMDMs. The key insight is that the conditional score decomposes orthogonally into a normal component, which enforces constraint satisfaction, and a tangential component, which navigates within the
manifold. Exploiting this structure, we prove that the noise-schedule midpoint is a sharp safe-caching boundary: caching before it incurs provably bounded error, while caching after it guarantees a strictly positive fraction of trajectories suffer mode confusion, a gap that persists up to the boundary. We further prove that deeper network blocks admit provably larger certified cache strides within the safe phase, as a consequence of the score decomposition propagating through block Jacobians. The resulting schedule requires no calibration data, along with zero training overhead.
Read the full abstract ▾
Machine Learning
LoRA's Second Descent Extends Beyond Parameter Parity
Yueran Ma
Double descent has sparked considerable interest, with recent work relating it to the data, the model and the learning configuration. Practical fine-tuning commonly involves training a small adapter on top of frozen pretrained weights, as in low-rank adaptation (LoRA). The adapter's rank is the hyperparameter that sets its capacity, yet how this rank relates to double descent has not been well explored. We quantify this relation under label noise on four
vision backbones and a 7B
language model with a module-matched rank sweep (MMRS), which extends past full rank and compares every rank with dense fine-tuning of the same modules, paired by seed. On DeiT-Tiny, risk is lowest at rank one and rises sharply as the adapter becomes able to fit the noisy labels, forming an interpolation cliff. Past the peak, risk falls again, but every tested post-peak rank that still saves parameters remains above dense risk. Rank-one LoRA outperforms dense fine-tuning on three of the four
vision backbones, consistent with strong regularization at small rank. LoRA thus exhibits a second descent, but matches dense risk only after losing its parameter advantage, first on DeiT-Tiny at four times dense's projection weights. Code is available at https://anonymous.4open.science/r/lora-second-descent-C7B5.
Read the full abstract ▾
Machine Learning · Cross-listed from cs.CV
Homogeneous Semantic Alignment and Hierarchical Expert Routing for Radiology Report Generation
Erjian Zhang, Jiayuan Ma, Liejun Wang et al. (+3 more)
Radiology report generation (RRG) aims to convert medical images into diagnostic texts to assist in clinical decision-making and alleviate the workload of physicians. Although existing methods have made extensive progress in cross-modal interaction and the incorporation of external priors, the distribution shift of underlying representations and the undifferentiated rigid coupling of heterogeneous information cause weak visual abnormality cues to be easily diluted by massive text priors and generation inertia during decoding. To overcome this bottleneck, inspired by cognitive science, we propose a novel two-stage Homogeneous Semantic
Alignment and Hierarchical Expert Routing (HSA-HER) framework. First, the model introduces an explicit homogeneous distribution constraint in the underlying latent space to effectively eliminate the cross-modal distribution shift between visual and textual features, thereby extracting purified visual features as semantic anchors that accurately align with diseases. Second, for heterogeneous clinical evidence composed of visual features, local entities, and global
retrievals, we design a hierarchical expert routing mechanism guided by these disease semantic anchors. This mechanism abandons the undifferentiated rigid coupling paradigm. Specifically, it dynamically activates expert networks to perform targeted mining and semantic reconstruction on multi-source evidence, and adaptively allocates fusion weights. Extensive experiments on three mainstream
benchmark datasets demonstrate that HSA-HER achieves state-of-the-art performance, accurately depicting complex imaging details and key diagnostic information.
Read the full abstract ▾
All authors Erjian Zhang, Jiayuan Ma, Liejun Wang, Yikemaiti Sataer, Xiaoming Tao, Zhiqing Guo
Machine Learning · Cross-listed from cs.SE
DreamTest: World-Model Surrogates for Search-Based Testing of Deep Reinforcement Learning Agents
Qinghua Xu, Guancheng Wang, Boxi Yu et al. (+2 more)
Testing deep reinforcement learning (DRL)
agents in cyber-physical systems aims to uncover diverse failures before deployment, but each execution can be expensive. Surrogate-assisted testing reduces this cost by learning to predict which test configurations are likely to fail. Prior surrogates treat the system as a black box and predict pass or fail outcomes directly; we instead model how a test unfolds and estimate failure from an imagined episode. We introduce DreamTest, a world-model surrogate for testing DRL
agents. DreamTest adapts a recurrent state-space model to learn
agent behaviour and environment dynamics from the
agent's training log. Given a candidate configuration, imagined rollouts produce a failure score that guides search without executing every candidate in a simulator or real system. We evaluate DreamTest for failure prediction, test generation, and failure diversity on Parking, Humanoid, and DonkeyCar. Mean area under the precision-recall curve (AUPRC) exceeds the strongest baseline by 97%, 12%, and 39%, respectively, and gains on five out-of-distribution test sets reach 145%, 29%, and 44%. Under the same simulator-validation budget, the best "DreamTest + search" combinations find 29%, 22%, and 79% more novel failures on average. Across clusterings with k = 2-40, failures generated with DreamTest cover the most behavioural clusters for almost all k, indicating that DreamTest consistently discovers behaviourally diverse failures.
Read the full abstract ▾
All authors Qinghua Xu, Guancheng Wang, Boxi Yu, Liting Lin, Lionel Briand
Machine Learning
Understanding Clustering in Slot Attention via Particle Dynamics
Vasudev Joy, Rajat Rasal, Avinash Kori et al. (+2 more)
Studying
attention through the lens of interacting particle dynamics has shown how
token clustering can emerge from the underlying dynamics. We extend this perspective to slot
attention, a method for object-centric image segmentation and representation learning in which learned components obscure how much of the clustering behaviour is intrinsic to the
attention dynamics. We therefore introduce simplified slot
attention (SSA), a parameter-free variant whose dynamics are connected to soft $k$-means clustering and which provides a straightforward mechanistic explanation for the emergence of object-centric representations. On the Pascal VOC
dataset, SSA achieves performance comparable to that of slot
attention, demonstrating that competitive object-centric segmentation can be achieved without learned neural-network components.
Read the full abstract ▾
All authors Vasudev Joy, Rajat Rasal, Avinash Kori, Anthea Monod, Ben Glocker
Machine Learning
BARQ: Balanced Codebook Refinement for Low-Bit LLM Quantization
Chenhang Cui, Xu Xie, Linrui Xu et al. (+4 more)
As large
language models (LLMs) grow in parameter count, model storage and parameter
memory traffic have become major bottlenecks to efficient deployment. Codebook-based weight quantization reduces these costs, but imbalanced nearest-codeword assignments during fitting can leave some codewords insufficiently updated, limiting effective codebook utilization. To address this limitation, we propose Balanced Assignment Refinement for Quantization (BARQ), which improves quantization quality through balanced fitting of existing codebooks. Specifically, we first compute joint soft assignments between weight blocks and codewords through entropically regularized optimal transport with uniform marginals and
curvature-weighted reconstruction costs, ensuring equal positive fitting mass for every codeword in the exact solution. We then refine the codewords through an assignment-weighted barycentric update, which we prove minimizes the fitting objective for fixed assignments. For finite Sinkhorn iterations, the implemented update retains this optimality provided all codeword masses exceed the denominator floor. Finally, we discard the soft assignments and use the refined codebook for standard hard nearest-codeword encoding, with our analysis establishing sufficient conditions for reducing hard-quantization distortion and evaluation loss. Across multiple LLMs, BARQ achieves lower perplexity and higher mean zero-shot accuracy than the evaluated baselines at comparable bit budgets. The code is available at https://github.com/chenhangcuisg-code/BARQ.
Read the full abstract ▾
All authors Chenhang Cui, Xu Xie, Linrui Xu, Xiaohao Liu, Xingyu Zhu, Fei Shen, Tat-Seng Chua
Machine Learning · Cross-listed from cs.CL
DV-Lens: Revealing the Functional Organization of Language Model Parameters
Chenhang Cui, Jian Yu, Shuyi Miao et al. (+5 more)
Understanding parameter functions helps elucidate the internal mechanisms of large
language models (LLMs). However, how to connect parameters from different modules to verifiable output effects and further characterize the relationship between their functional organization and model capability remains to be explored. To this end, we introduce the downstream vocabulary lens (DV-Lens), a parameter-level interpretability framework that links native parameter directions to their downstream vocabulary responses. Specifically, we first estimate module-specific downstream Jacobians over a reference
prompt set for
attention query, key, value, and output (Q/K/V/O) projections and feed-forward networks (FFNs). Second, we use these mappings to project native parameter columns into the final vocabulary space, obtaining signed readouts that characterize their average local output responses. Third, we group parameter columns by their vocabulary readouts and introduce downstream vocabulary complexity (DV-Complexity), which quantifies within-group structural variation using normalized reconstruction residuals of the original weights. At the parameter level, randomized controls and finite-difference tests show that DV-Lens readouts capture non-random vocabulary structure and predict local logit changes with 98.0% coordinate-orientation agreement across 720 cases from nine models. These readouts further guide parameter ablation, steering, and swapping across 21 models, shifting target-
token probabilities in the predicted directions under controlled conditions. At the model level, the joint-parameter score of DV-Complexity achieves a Spearman correlation of 0.904 with
benchmark-based capability rankings across 48
language models. Together, these results provide intervention-based evidence for DV-Lens interpretations and reveal an association between DV-Complexity and model capability.
Read the full abstract ▾
All authors Chenhang Cui, Jian Yu, Shuyi Miao, Xiaohao Liu, Rui Huang, Fei Shen, An Zhang, Tat-Seng Chua
Machine Learning
Target-free Latent Safety Alignment
Luoyu Chen, Weiqi Wang, Chenhan Zhang et al. (+3 more)
Large
language models (LLMs) remain highly vulnerable to jailbreak attacks that induce harmful behaviors and circumvent safety
alignment. To defend against such attacks,
adversarial training paradigms have been proposed to first simulate failure modes and then train the model to correct them, yielding promising improvements in safety
alignment. However, these methods typically construct
adversarial samples either by encouraging fixed harmful target completions or by performing targeted activation ablation derived from fixed benign–harmful data pairs. As a result, the generated
adversarial samples tend to induce homogeneous harmful behaviors that poorly reflect the diversity of behaviors elicited by real-world jailbreak attacks. This behavior-level narrowness fundamentally limits their robustness. To address this issue, we propose a target-free
adversarial training framework that generates
adversarial samples in an unsupervised manner. By amplifying and diversifying behavior-level shifts in the model's latent space, our approach produces semantically diverse
adversarial samples that induce a wide range of harmful behaviors. This expanded behavioral coverage exposes more diverse failure modes and thereby improves safety
alignment. To quantify this effect, we use semantic
entropy as an output-level measure of
adversarial behavioral diversity. Empirically, our method elicits diverse harmful behaviors in the target model, substantially mitigating behavioral narrowness and improving robustness to jailbreak attacks.
Read the full abstract ▾
All authors Luoyu Chen, Weiqi Wang, Chenhan Zhang, Zhiyi Tian, Yuxian Huang, Shui Yu
Machine Learning
One-Step Generation via Riemannian Wasserstein Gradient Flows
David Li, Chanhyuk Lee, Jaehoon Yoo et al. (+7 more)
Recently, Drifting Models and Wasserstein Gradient Flows have attracted substantial
attention because they move iterative distributional refinement to training and amortize it into a generator, enabling fast inference. However, existing formulations have been developed largely for continuous Euclidean domains, such as image spaces, where particles admit unconstrained additive updates. On constrained spaces, these updates can leave the valid domain or ignore its geometry, making them unsuitable targets for training. Recent work has adapted updates to these spaces, but has focused on particular fields or offered limited empirical comparison. We derive and compare several geometry-aware fields within a common training framework for one-step generators. We test the method on data with different structures and obtain competitive one-step results in each setting. The best-performing field varies by task, showing why the choice of objective matters in practice.
Read the full abstract ▾
All authors David Li, Chanhyuk Lee, Jaehoon Yoo, Nikita Gushchin, Floor Eijkelboom, Eric Moulines, Maxim Panov, Alexander Korotin, Seunghoon Hong, Jinwoo Kim
Machine Learning · Cross-listed from cs.AI
Causally Fair Generation with Large Language Models
Patrik Okanovic, Torsten Hoefler and Drago Plecko
Large
language models (LLMs) are increasingly used to generate, complete, and transform information in settings where their outputs can shape consequential decisions, raising concerns about their impact on demographic disparities. In this context, causal inference provides a principled basis for assessing
fairness, because it attributes observed disparities to the mechanisms that generated them, which a purely statistical approach cannot do even with infinite data. In LLM generation, a query may request several causally related variables, each of which is both an outcome of interest and a possible cause of other outputs, and the information supplied in the
prompt need not follow a topological or a temporal order. This calls for methods that can analyze and selectively remove disparities from such a flexible generation process. In this paper we introduce Causally Fair Generation with LLMs (CFG, for short). CFG extracts relevant concepts, grounds generation in a reference population and causal diagram, and removes user-selected causal effects. CFG also allows pathways deemed justifiable for the task's utility to be retained, which is known in legal literature as business necessity. Further, we provide formal guarantees for our method when eliminating all discriminatory causal effects in the adapted population model, under appropriate causal assumptions. We evaluate CFG with four LLMs in three real-world settings based on population data and on a synthetic
dataset with a known causal ground truth.
Read the full abstract ▾
Machine Learning
LocusRL: Diagnosing LLM Reward and Policy Interventions in Competitive Games
Chengyu Luan, Bo Xin, Songyan Guo et al. (+4 more)
Large
language models can intervene in reinforcement learning through both reward design and action selection, yet aggregate performance offers an incomplete account of what these interventions actually do. Similar returns can conceal different learning mechanisms, while plausible rewards can induce undesirable behavior. We introduce LocusRL, a diagnostic framework that connects controlled reward-policy comparisons with audits of reward judgments, signal delivery, optimization objectives, and executed actions. The framework traces performance differences to testable explanations and checks targeted corrections through executable rules and counterfactual replay. Across two evaluation batches covering ten Connect Four training seeds, we uncover seed-dependent reversals in intervention effects and show how tracing actual updates changes their interpretation: historical Qwen training operates through reward-weighted teacher-action likelihood. A separate matched three-seed reward-direction experiment distinguishes sensitivity to a learning signal from its usefulness. With terminal rewards held fixed, a sign-reversed dense oracle yields a 2.8% aggregate win rate, compared with 57.2% for terminal-only training and 46.7% for the positive dense oracle. Thus, a reward can strongly influence learning without improving performance. At the decision level, counterfactual replay verifies a winning correction to a diagnosed action error. Complementary experiments in Leduc and reward-validation studies in Goofspiel extend the analysis to imperfect-information settings, revealing how reference-label definitions and validation-data exposure affect intervention assessment. Together, these findings show why evaluating LLM interventions requires tracing how their outputs become learning signals and actions. LocusRL turns aggregate outcomes into actionable diagnoses and verifiable corrections.
Read the full abstract ▾
All authors Chengyu Luan, Bo Xin, Songyan Guo, Yuxiang Zuo, Ahmed Yazdan, Jiahang Li, Yicheng Liu
Machine Learning · Cross-listed from cs.AI
What Does a Harness Buy? Tokens, Mostly
Yangze Liu and Zhongyi Han
A coding
agent is a
language model wrapped in a harness: the system
prompt, the tool set, and the context management that turn a chat model into something that can work inside a repository. Production harnesses ship releases daily, vendors advertise pass-rate gains from harness changes, and leaderboards mix harnesses freely. What is rarely measured is how much the harness itself moves the score when the model is held fixed. We run five models through three production harnesses, Claude Code, mini-SWE-
agent, and OpenCode, on SWE-bench Verified, and rerun the same configurations to calibrate how much a score moves when nothing changes but the run. On 447 tasks and the two models we ran there, Claude Code and mini-SWE-
agent, the heaviest and the lightest harness, are equivalent within five points. On a 45-task hard subset and five models, swapping the harness flips as many tasks as rerunning the same harness, 13% in both cases, and the tasks a harness wins in one run are not the tasks it wins in the next. The one harness effect that clears the noise is a loss, not a gain: OpenCode trails by up to 9 points on the large pool, and on one model half of that gap sits in runs its output cap cut short. What the harness does decide is the bill. With the same model, the same tasks, and one price list, cost per task differs by up to 3x across harnesses. The gap is set at the first call, by the preamble of system
prompt and tool schemas each harness sends with every step, and scaled by the number of steps; per-step growth and per-call tool output differ far less. The provider's price for cached input scales the bill and does not reorder it. The rerun data also give the resolution a harness comparison needs: at the discordance we observe, 45 tasks catch a 13-point gap only half the time and no gap with 80% power, and 447 tasks resolve 5 points, still coarser than the gains many harness changes claim.
Read the full abstract ▾
Machine Learning
COSMOS: Soft Mechanism Mixtures with Verifiable Routing for Long-Horizon PDE Forecasting
Anupam Rawat, Manikandan Padmanaban and Jagabondhu Hazra
Neural operators offer an efficient alternative to classical PDE solvers, but most learn a monolithic map per equation family and discretization. Real systems are compositional: transport,
diffusion, wave, and reaction processes can act simultaneously. Existing mixture-of-experts operators typically use sparse top-$K$ routing, although concurrent physics is naturally a blend rather than a discrete choice. We propose COSMOS (Cooperative Operator Specialists with Mechanism-level Operator Soft-routing), a soft mechanism-mixture neural operator. Four process-biased specialists remain active at every step and are continuously mixed by a learned gate, with their features fused by a small network. Specialists share a coarse latent grid, while a zero-initialized full-resolution residual restores detail lost through the bottleneck. We also introduce an operator-splitting compositional
benchmark with known mixture weights $w^\star$ per trajectory. Against family-tuned FNO under 20-step rollouts, an initial 3-seed evaluation suggested gains on
diffusion–reaction, parity on Navier–Stokes, and weaker shallow-water performance. An 11-seed audit showed that the
diffusion–reaction gain was unstable, motivating caution in small-seed rollout comparisons and precluding a reliable accuracy-win claim on these families. Ablations show that uniform routing or removing the specialist mixture substantially degrades stable-regime rollouts. On the labeled
benchmark, dense soft routing yields $2.2\times$ lower error than hard top-1 routing at identical fusion; using generator weights $w^\star$ at inference further lowers rollout error to $0.034$, diagnosing limitations of the learned gate. However, routing labels do not align with the specialists' intended mechanisms: COSMOS supports compositional accuracy, not mechanism identity.
Read the full abstract ▾
Machine Learning
UnAct: Gradient-Free Unlearning via Targeted Activation Intervention
Saeed Abdul Muizz, Aayat Rafiq, Iqra Altaf Gillani et al. (+1 more)
Machine unlearning seeks to remove the influence of designated training data from a trained model without retraining from scratch. Retrain-free methods such as Selective Synaptic Dampening (SSD) and its label-free variant LFSSD avoid full retraining but still require backpropagation and parameter importance computed over the entire
dataset. We ask: what happens when a deletion request arrives with only a few images of the class to be forgotten? To answer this question, we introduce UnAct, a gradient-free class-unlearning method that needs only forward passes over the forget images. UnAct scores late-layer units by their responses, attenuates the most responsive connections, and repeats this for up to 20 rounds using no gradients, no labels, and no retained data. On ResNet-18 trained with CIFAR-10, CIFAR-20, and CIFAR-100, UnAct is competitive with SSD and LFSSD when forgetting entire classes and, unlike them, never collapses the network when forget data is scarce. On ResNet-18, across all tested sizes, UnAct's retain accuracy stays within 2.5 points of retraining, while SSD and LFSSD, at their full-class operating points, lose up to 86 points on some classes. With five forget images on CIFAR-10, UnAct's distance to retraining is 0.21 points, against 67 for LFSSD and 90 for SSD, and re-selecting SSD's threshold at each size with an oracle does not close the gap. In preliminary transfer to ViT-B/16, UnAct's distance to retraining is 11.5 against 33.7 for SSD, and a request is 19x faster than SSD when SSD computes its importance at request time. The code is available at https://github.com/abdulmuizz0903/UnAct
Read the full abstract ▾
All authors Saeed Abdul Muizz, Aayat Rafiq, Iqra Altaf Gillani, Janibul Bashir
Machine Learning
On the Trade-off Between Information Loss and Generalization in Sparse Attention
Zhongqi Fan and Zheng Tan
To mitigate the quadratic complexity bottleneck of the
Transformer, sparse
attention has emerged as a pivotal technology. Despite the extensive empirical success of sparse
Transformers, the theoretical understanding of sparse
attention remains fragmented. In particular, two fundamental questions remain unclear: (1) How does sparsification affect the information fidelity of
attention mechanisms? (2) How does this information loss interact with the generalization behavior of the model? To bridge this gap, this paper proposes a systematic analysis of the Jensen-Shannon (JS) divergence and of the generalization gap of sparse
attention mechanisms. Specifically, we first characterize the approximation error via the JS divergence. Through an order-statistics-based concentration analysis of the truncation mass alpha — where the
attention scores are assumed to be independent and identically distributed sub-Gaussian random variables with parameter sigma — the JS divergence between the full
attention distribution and the sparse
attention distribution is shown to admit the closed form log 2 + ((1 - alpha)/2) log(1 - alpha) - ((2 - alpha)/2) log(2 - alpha). Subsequently, we derive a generalization bound through Rademacher complexity, quantified by O(gamma * sqrt(M/n) * (sqrt(log(3eL/M)) + sqrt(pi)/2)). Furthermore, building on a mutual-information-based generalization bound together with an
entropy and covering-number analysis of the sparse hypothesis class, we obtain the sparsity-dependent generalization bound O(sqrt((M/(2n)) * (log(eL/M) + log(1 + 2/epsilon)))). Our analysis shows that sparsity reduces the complexity of the considered hypothesis class while introducing approximation error that can be quantified by the JS divergence. These findings provide a theoretical characterization of the trade-off between information fidelity and generalization in sparse
Transformer architectures.
Read the full abstract ▾