Machine Learning
SWE Atlas: Benchmarking Coding Agents Beyond Issue Resolution
Mohit Raghavendra, Soham Dan, Miguel Romero Calvo et al. (+12 more)
We introduce SWE Atlas, a
benchmark suite for coding
agents spanning three professional software engineering workflows: Codebase Q&A (124 tasks), Test Writing (90 tasks), and Refactoring (70 tasks). SWE Atlas differs from prior SWE
benchmarks in three key ways: it targets underrepresented but practically important task categories, uses comprehensive category-specific evaluation protocols, and adopts under-specified, agentic task formulations that better reflect real-world usage. Its evaluation framework combines programmatic checks with rubric-based assessment. This goes beyond functional correctness, evaluating software engineering quality, including test and refactor completeness, maintainability, reusable abstractions, and codebase hygiene. We evaluate a range of frontier and open-weight models on SWE Atlas and find that GPT-5.4 and Opus 4.7 achieve the strongest overall performance, while even the best open-weight models score poorly. Our analysis suggests that top models rely on extensive codebase exploration and runtime-driven
reasoning. However, even top models consistently struggle with subtle edge cases, complex runtime analysis, and adherence to software engineering best practices. Overall, SWE Atlas provides a complementary evaluation suite for measuring both correctness and engineering quality in coding
agents.
Read the full abstract ▾
All authors Mohit Raghavendra, Soham Dan, Miguel Romero Calvo, Yannis Yiming He, Johannes Baptist Mols, Gautam Anand, Cole McCollum, Edgar Arakelyan, Vijay Bharadwaj, Andrew Park, Jeff Da, MohammadHossein Rezaei, Bing Liu, Brad Kenstler, Yunzhong He
Machine Learning
Convergence Analysis of Newton's Method for Neural Networks in the Overparameterized Limit
Konstantin Riedl, Konstantinos Spiliopoulos and Justin Sirignano
A convergence analysis is developed for the regularized Newton method for training neural networks (NNs) in the overparameterized limit. As the number of hidden units tends to infinity, the NN training dynamics converge in probability to the solution of a deterministic limit equation involving a "Newton neural tangent kernel" (NNTK). Explicit rates characterizing this convergence are provided and, in the infinite-width limit, we prove that the NN converges exponentially fast to the target data (i.e., a global minimizer with zero loss). We show that this convergence is uniform across the frequency spectrum, addressing the spectral bias inherent in gradient descent. The
eigenvalues of the NTK for gradient descent accumulate at zero, leading to slow convergence for target data with high-frequency components. In contrast, the NNTK has uniformly lower bounded
eigenvalues if the regularization parameter is selected appropriately, allowing Newton's method to converge more quickly for data with high-frequency components. Mathematical challenges that need to be addressed include the implicit parameter update of the Newton method with a potentially indefinite Hessian
matrix and the fact that the dimension of this linear system of equations tends to infinity as the NN width grows. This substantially complicates deriving the training dynamics in the overparameterized limit as well as proving the convergence of the finite-width dynamics thereto. Our analysis identifies a scaling formula for selecting the regularization parameter, which we show can vanish at a suitable rate as the NN width becomes larger. In addition, we prove that, for sufficiently large numbers of hidden units, the regularized Hessian remains positive definite during training and the Newton updates for individual NN parameters converge to zero, demonstrating that the model behaves as a linearization around the initialization.
Read the full abstract ▾
Machine Learning
What Time Is It? How Data Geometry Makes Time Conditioning Optional for Flow Matching
Alec Helbling, Sebastian Gutierrez Hernandez, Benjamin Hoover et al. (+2 more)
Recent work has shown that models flow matching models can be trained without explicit time conditioning, challenging the standard view that the interpolation time is needed to disambiguate velocity targets. But why should a time-blind model work at all? Decomposing the time-blind flow matching loss, we identify two sources of irreducible error: a coupling variance, which arises from ambiguous velocity targets induced by how noise and data points are paired, and the time-blindness gap, which is the additional error caused by ignoring time. This gap shows that time-blind training is strictly harder than conventional training, reinforcing the puzzle that time-blind models work so well in practice. We resolve this tension by showing that the geometry of high-dimensional data makes time identifiable directly from noisy observations. When data concentrates near a $k$-dimensional subspace, time can be recovered from the statistical structure of noisy interpolants in directions orthogonal to the data; under a spiked-covariance model, this yields a closed-form
estimator that recovers $t$ from a single observation $z$ at rate $O(1/\sqrt{d-k})$ for ambient dimension $d$. As a consequence, we prove that the time-blindness gap is asymptotically negligible relative to the coupling variance. We empirically demonstrate our identifiability result on real-world data and show that changing the coupling has a much larger effect on loss and sample quality than removing time conditioning across CIFAR-10, CelebA-HQ, and FFHQ. These results explain why time-blind flow matching works and show that the main practical lever is the choice of coupling, not explicit time conditioning.
Read the full abstract ▾
All authors Alec Helbling, Sebastian Gutierrez Hernandez, Benjamin Hoover, Duen Horng Chau, Parikshit Ram
Machine Learning
Private Vertical Federated Inference for Time-Series
Lucas Fenaux, Larris Xie, Aditya Bang et al. (+3 more)
Institutions may benefit from collaborative inference on time-series data. In settings where
privacy is necessary, multi-party computation (MPC) is a straightforward approach to providing strong guarantees, yet it remains prohibitively expensive and scales poorly with modern
transformer architectures. Vertical
Federated Learning (VFL) offers efficiency but suffers from
privacy leakage at the
embedding level, and securing the entire VFL model head via MPC remains prohibitively slow and communication-heavy for larger models. To enable practical, secure inference at scale, we propose "Public/Private Hybrid Head-VFL" (PPHH-VFL). This hybrid architecture splits the model head into an efficient plaintext public head and a secure, lightweight MPC private head. By applying
adversarial training to the public
embeddings, we mitigate
privacy leakage; concurrently, the small private head securely preserves the flow of sensitive information needed for high downstream utility. Empirical evaluations on models ranging up to 86 million parameters demonstrate that PPHH-VFL accelerates inference by up to six orders of magnitude compared to end-to-end MPC. Compared to a standard VFL+MPC baseline, our approach scales significantly better, achieving a speedup of up to 44.4x in WAN and a 91.2x reduction in communication costs (dropping from 1.7 GB to 19 MB per batch), while simultaneously improving downstream classification accuracy by 2.50% and regression RMSE by 40.7%.
Read the full abstract ▾
All authors Lucas Fenaux, Larris Xie, Aditya Bang, Alex Zhang, Kevin Wilson, Florian Kerschbaum
Machine Learning
CDS4RAG: Cyclic Dual-Sequential Hyperparameter Optimization for RAG
Pengzhou Chen and Tao Chen
Retrieval-Augmented Generation (RAG) is sensitive to the vast hyperparameters of the retriever and generator, yet optimizing them using given queries is a challenging task due to the complex interactions and expensive evaluation costs. Existing
algorithms are ineffective and slow in convergence, since they often treat RAG as a monolithic black box or only optimize partial hyperparameters. In this paper, we propose CDS4RAG, a framework that optimizes the full RAG hyperparameters using given queries via a new cyclic dual-sequential formulation. CDS4RAG is special in the sense that it distinguishes the hyperparameters of the retriever and generator, cyclically optimizing them in turn. Such a paradigm allows us to design fine-grained within-cycle budget provision and expedite the optimization via cross-cycle seeding when optimizing the generator. CDS4RAG is also an
algorithm-agnostic framework that can be paired with diverse general
algorithms. Through experiments on four common
benchmarks and two backbone LLMs, we reveal that CDS4RAG considerably boosts the vanilla
algorithms in 21/24 cases while significantly outperforming state-of-the-art
algorithms in all cases with up to 1.54x improvements of generation quality and better speedup.
Read the full abstract ▾
Machine Learning
P-Flow: Proxy-gradient Flows for Linear Inverse Problems
Zehua Jiang, Fenghao Zhu, Xinquan Wang et al. (+2 more)
Generative models based on flow matching have emerged as a powerful paradigm for inverse problems, offering straighter trajectories and faster sampling compared to
diffusion models. However, existing approaches often necessitate differentiating through unrolled paths, leading to numerical instability and prohibitive computational overhead. To address this, we propose P-Flow, a framework that stabilizes the reconstruction process by leveraging a proxy gradient to update the source point. This approach effectively circumvents the numerical instability and
memory overhead of long-chain differentiation. To ensure consistency with the prior distribution, we employ a Gaussian spherical projection motivated by the concentration of measure phenomenon in high-dimensional spaces. We further provide a theoretical analysis for P-Flow based on
Bayesian theory and Lipschitz continuity. Experiments across diverse restoration tasks demonstrate that P-Flow delivers competitive performance, especially under extreme degradations such as severely ill-posed conditions and high measurement noise.
Read the full abstract ▾
All authors Zehua Jiang, Fenghao Zhu, Xinquan Wang, Chongwen Huang, Zhaoyang Zhang
Machine Learning
Interactive Critique-Revision Training for Reliable Structured LLM Generation
Fei Xu Yu, Zuyuan Zhang, Mahdi Imani et al. (+2 more)
In structured decision-making workflows such as form filling, compliance checking, and maintenance reporting, LLM outputs must be locally correct, globally consistent, and auditable against task-specific rules. Existing refinement methods often rely on heuristic debate, self-play, or LLM-generated supervision, creating a second-order assurance problem. We propose DPA-GRPO (Dual Paired-Action Group-Relative Policy Optimization), a paired-action training method for a two-player generator–verifier game with structured verifier interventions. The generator proposes outputs and may revise them when challenged; the verifier either remains silent or raises a safety assurance case (SAC) containing a claim, argument, and evidence. These SAC/no-SAC and KEEP/REVISE decisions induce paired counterfactual action groups, which DPA-GRPO uses for role-specific KL-regularized GRPO updates. We analyze the unregularized game and show that positive probability on strictly lower-reward intervention or revision actions creates a profitable unilateral deviation. Under standard
stochastic-approximation assumptions, DPA-GRPO tracks the corresponding game ODE, whose isolated asymptotically stable limit points are stationary and candidate local
equilibria under role-wise local optimality. Experiments on TaxCalcBench TY24 show that DPA-GRPO improves structured decision accuracy over zero-shot generation and generator-only RL baselines across Qwen3-4B and Qwen3-8B. Training increases correct silent acceptance, reduces missed errors, and improves calibrated revision behavior, indicating gains for both generator and verifier.
Read the full abstract ▾
All authors Fei Xu Yu, Zuyuan Zhang, Mahdi Imani, Nathaniel D. Bastian, Tian Lan
Machine Learning
LLM Advertisement based on Neuron Auctions
Peiran Yun, Wenxin Xu, Jiayuan Liu et al. (+4 more)
As Large
Language Models (LLMs) transition into conversational
agents, generative advertising emerges as a crucial monetization strategy. However,
embedding advertisements within unstructured LLM outputs introduces a critical trilemma: balancing advertiser payoffs, platform revenue, and user experience. Existing methods, such as
prompt injection or rigid position slots, disrupt semantic coherence and lack a parametric framework for independent control, rendering rigorous mechanism design intractable. To bridge this gap, we introduce
Neuron Auctions, a novel paradigm that shifts the auction object from the surface text space to the LLM's internal representations. Leveraging mechanistic interpretability, we identify brand-specific feed-forward network (FFN)
neurons and demonstrate that competing brands activate within approximately orthogonal subspaces. This near-perfect independence allows us to define continuous, disentangled intervention budgets (specifically,
neuron counts and amplification factors) as auctionable commodities. Building on this computational carrier, we design a continuous menu-based auction mechanism that naturally guarantees strategy-proofness and optimizes revenue for the platform. By explicitly incorporating a user utility penalty into the platform's optimization objective, our framework dynamically prices out overly aggressive interventions. Extensive experiments demonstrate that
Neuron Auctions effectively preserve natural discourse quality while achieving an optimal
alignment between commercial incentives and user satisfaction.
Read the full abstract ▾
All authors Peiran Yun, Wenxin Xu, Jiayuan Liu, Yihang Zhang, Liang Zeng, Lingkai Kong, Tonghan Wang
Machine Learning · Cross-listed from eess.IV
Rajdeep Singh Hundal, Yan Xiao, Jin Song Dong et al. (+1 more)
Many
vision datasets now provide
segmentation masks in addition to annotated images to support a wide range of tasks. In this work, we propose Class Activation Map
Attention Learning (CAMAL), an efficient and scalable method that utilizes
segmentation masks to improve
attention alignment and faithfulness in
vision models. Specifically,
attention alignment refers to the degree to which a model's
attention aligns with ground-truth discriminative regions, while
attention faithfulness refers to the degree to which a model's
attention influences its decision. Improving both
attention alignment and faithfulness is essential for ensuring that model
attention is both spatially accurate and causally meaningful. To improve
attention alignment and faithfulness in
vision models, CAMAL first extracts the model's
attention for each image during training and then compares the
attention to ground-truth discriminative regions obtained from the corresponding
segmentation masks. CAMAL then acts as an auxiliary regularizer, encouraging
attention that aligns with ground-truth discriminative regions, while suppressing
attention elsewhere. We evaluated CAMAL across two learning paradigms – Deep Learning (DL) and Deep Reinforcement Learning (DRL) – and observed consistent, significant improvements in both
attention alignment and faithfulness. In particular, CAMAL yields statistically significant gains in
attention alignment across all settings, and improves
attention faithfulness by over 35% compared to recent work. Moreover, we show that improved
attention alignment and faithfulness enhance explainability, while yielding improved or comparable generalization performance without increasing inference cost. These findings demonstrate that the spatial information contained within
segmentation masks can be effectively leveraged to guide model
attention across learning tasks.
Read the full abstract ▾
All authors Rajdeep Singh Hundal, Yan Xiao, Jin Song Dong, Manuel Rigger
Machine Learning · Cross-listed from eess.IV
FQPDR: Federated Quantum Neural Network for Privacy-preserving Early Detection of Diabetic Retinopathy
Debashis De, Mahua Nandy Pal and Dipankar Hazra
Diabetic Retinopathy (DR) is a common complication of diabetes that can lead to blindness of people. Detecting DR at the earliest stage is essential to prevent irreversible eye damage. Microaneurysm dots are the first signs of DR. As the dots are tiny and of low contrast, detecting mild DR is a very challenging task.
Federated learning (FL) preserves data
privacy, which is a major concern for medical image processing. FL is a collaborative learning method, which shares only the model parameters with a server, without sharing the patient data to a central server. Inspired by classical FL, we propose a
federated learning-based
quantum neural network (
federated QNN) for this task. We implemented the models with limited samples and few learnable parameters from the E-ophtha and Retina MNIST
datasets. The crossevaluation efficiency of the proposed
federated quantum neural network system for
privacy-preserving early detection of diabetic retinopathy (FQPDR) in Kaggle
dataset images indicates the robustness of the light weight learning models. FQPDR performances are inspiring while considering existing non-FL and FL methods.
Read the full abstract ▾
Machine Learning
The Reciprocity Gradient
Yue Lin, Pascal Poupart, Shuhui Zhu et al. (+5 more)
Communication is fundamental to sustaining reciprocity and cooperation in strategic interactions. We identify and formulate the influence attribution problem as the central optimization difficulty inherent in such dynamics for a learning
agent: any action or signal the
agent emits reshapes the reputations of many third parties along combinatorially branching paths before feeding back into its own future rewards, forcing the
agent to account for all of these indirect channels at once when choosing every action. To address this, we introduce the reciprocity gradient, which explicitly backpropagates reward gradients through private
estimators of opponents' policies trained from public observations. The gradient flows through the reputation chain itself analytically, rather than being estimated from sampled returns. It jointly optimizes actions and evaluative signals without intrinsic rewards or reward shaping. Empirically, the method recovers near-optimal context-sensitive policies, while sample-based baselines collapse into constant-output policies.
Read the full abstract ▾
All authors Yue Lin, Pascal Poupart, Shuhui Zhu, Dan Qiao, Wenhao Li, Yuan Liu, Hongyuan Zha, Baoxiang Wang
Machine Learning
SDG-MoE: Signed Debate Graph Mixture-of-Experts
Stepan Kulibaba, Kirill Labzin, Artem Dzhalilov et al. (+4 more)
Sparse MoE models achieve a good balance between capacity and compute by routing each
token to a small subset of experts. However, in most MoE architectures, once a
token is routed, the selected experts process it independently and their outputs are combined via a weighted sum. This leaves open whether enabling communication among them could improve performance. While prior work has raised this question, direct interaction among the active routed experts remains underexplored. In this paper, we propose SDG-MoE (Signed Debate Graph Mixture-of-Experts), a novel architecture that adds a lightweight, iterative deliberation step before final aggregation. SDG-MoE introduces three components: (i) two learned interaction
matrices over the active experts, a support graph $A^+$ and a critique graph $A^-$, capturing reinforcing and corrective influences; (ii) a signed message-passing step that updates expert representations before aggregation; and (iii) a disagreement-gated Friedkin-Johnsen-style anchoring that controls deliberation strength while preventing expert drift. Together, these enable a structured deliberation process where interaction strength scales with disagreement and specialization is preserved. We also provide a theoretical analysis establishing stability conditions on expert states and showing that deliberation adds only low-order overhead over the active set. In controlled three-seed pretraining experiments, SDG-MoE improves validation perplexity over both an unsigned graph communication baseline and vanilla MoE, outperforming the strongest baseline by 19.8%, and gives the best external perplexity on WikiText-103, C4, and Paloma among the compared systems.
Read the full abstract ▾
All authors Stepan Kulibaba, Kirill Labzin, Artem Dzhalilov, Roman Pakhomov, Oleg Svidchenko, Alexander Gasnikov, Aleksei Shpilman
Machine Learning
LLM Wardens: Mitigating Adversarial Persuasion with Third-Party Conversational Oversight
Lennart Wachowiak, Scott D. Blain, David Williams-King et al. (+1 more)
LLMs are increasingly capable of persuasion, which raises the question of how to protect users against manipulation. In a preregistered user study (N=120) across four decision-making scenarios, we find that an
adversarial LLM with a hidden goal succeeds in steering users' decisions 65.4% of the time. We then introduce a "warden" model: a secondary LLM that monitors the human-AI interaction trace in real time and issues non-binding, private advisories to the user when it detects manipulation. Adding a warden more than halves the adversary's success rate to 30.4%, with a much smaller (8.6 percentage points) reduction for genuine interactions. To probe the mechanism behind these results, we release COAX-Bench, a simulation
benchmark spanning 14 decision-making scenarios, including hiring, voting, and file access. Across 16,212 simulated multi-
agent interactions, capable
adversarial LLMs achieve their hidden goals in 34.7% of cases, which warden models reduce to 12.3%. Notably, even warden models substantially weaker than the adversary they oversee provide meaningful protection, suggesting a path for scalable oversight of more capable models.
Read the full abstract ▾
All authors Lennart Wachowiak, Scott D. Blain, David Williams-King, Samuele Marro
Machine Learning
When Attention Beats Fourier: Multi-Scale Transformers for PDE Solving on Irregular Domains
Brandon Yee, Pairie Koh, Jack Rodriguez et al. (+1 more)
We study the problem of architecture selection for deep learning models trained to solve partial differential equations (PDEs), asking when
transformer-based architectures with learned
attention outperform
Fourier-domain neural operators. We introduce the Multi-Scale
Attention Transformer (\msat{}), a deep learning architecture that encodes spatiotemporal solution histories as
token sequences and trains end-to-end via a composite supervised objective with optional physics-informed regularization terms. We conduct a comprehensive empirical evaluation against nine baselines – including physics-informed neural networks (PINNs), neural operators (FNO, DeepONet, GNOT), and state-space models (Mamba-NO) – across five
benchmark problems from the PINNacle suite, using identical train/test splits and reference data for all methods. \msat{} achieves state-of-the-art generalization on complex geometry problems ($L^2_\mathrm{rel} = 0.0101$ on Heat2D-CG, a $3.7\times$ improvement over FNO) at $34\,\mathrm{s}$ total inference vs.\ $120{,}812\,\mathrm{s}$ for Mamba-NO. Ablation studies over the physics regularization component reveal a precise inductive bias tradeoff: physics priors reduce test error on
diffusion-dominated problems but degrade generalization on chaotic and recirculating-flow regimes, directly characterizing the prior misspecification boundary. Approximation error bounds as a function of domain boundary complexity $\kappa$ provide a theoretical basis for these empirical findings and a principled rule for architecture selection.
Read the full abstract ▾
All authors Brandon Yee, Pairie Koh, Jack Rodriguez, Mihir Tekal
Machine Learning
RDKV: Rate-Distortion Bit Allocation for Joint Eviction and Quantization of the KV Cache
Junkai Zhang, Hang Guo, Luca Benini et al. (+1 more)
Large
language models (LLMs) have shown strong performance across diverse tasks, but their inference with long input contexts is bottlenecked by
memory size and bandwidth. The Key-Value (KV) cache size grows linearly with sequence length and needs to be re-read from off-chip high-bandwidth
memory (HBM) to on-chip
memory at every decoding step, resulting in
memory-bound inference. Existing methods reduce the cache by either eviction or quantization, but typically treat the two in isolation. In this paper, we cast KV cache compression as a rate-distortion problem, under which eviction and quantization are two end-points of the same bit allocation scheme. This exposes the need to optimize them jointly, motivating our method, RDKV (Rate-Distortion KV cache compression). RDKV derives the weight of each
token or channel from the distortion that compression induces on the
attention computation. Based on these weights, it assigns each
token or channel a bit-width ranging from full precision down to zero bits guided by reverse water-filling, applied once after the prefilling stage. Experiments on LongBench, RULER, and InfiniteBench show that RDKV outperforms the best evaluated baseline by 9.1% on average. On LongBench it recovers 97.81% of full-cache accuracy with only 2.48% cache retention. Compared with full-cache FlashAttention-2 decoding, it achieves 4.5x decode speedup and 1.9x peak
memory reduction with 128K context length, while maintaining comparable performance.
Read the full abstract ▾
All authors Junkai Zhang, Hang Guo, Luca Benini, Yawei Li
Machine Learning
Reflective Prompted Policy Optimization: Trajectory-Grounded Revision and Salience Bias
Rahaf Abu Hara, Vaibbhav Murarri and Claudio Zito
Existing LLM-based policy optimizers see only scalar rewards: that a policy scored 0.45, but not whether the
agent got stuck in a loop, fell into a hole on the third step, or performed well on 19 out of 20 rollouts and failed catastrophically on one. We propose Reflective Prompted Policy Optimization (R2PO), a two-stage LLM framework for policy search over compact policy classes that augments scalar reward feedback with trajectory-level behavioral evidence. A Search-LLM proposes candidate policy parameters; the environment executes them; a Critic-LLM inspects the resulting rollouts and proposes targeted revisions grounded in observed states, actions, and rewards. Across ten environments, ablations show R2PO's gains require separating global search from behavior-grounded revision and using selection to filter high-variance edits. We further identify a dominant failure mode, salience bias: when presented with multiple rollouts, the Critic-LLM fixates on improving a single failure even when most trajectories succeed. In a three-trajectory variant where the Critic-LLM sees the best, worst, and median rollout, this behavior explains 76.6% of regressions on CartPole. R2PO mitigates this by
reasoning over aggregate rollout statistics, median-trajectory selection, and a revision rule. Using a 20B open-weight model, R2PO achieves the highest mean best reward across all ten environments, reaches near-optimal performance substantially earlier (e.g., near-maximum CartPole reward within ~500 episodes), and trains far more stably than both deep RL and prior LLM-based methods. These results show that treating trajectories as first-class in-context evidence, rather than artifacts reduced to scalar returns, changes how even comparatively small LLMs search over policy spaces, enabling them to learn faster, diagnose more precisely, and reliably improve external controllers.
Read the full abstract ▾
Machine Learning
FlashSVD v1.5: Making Low-Rank Transformers Inference Actually Fast
Wenhao Wu, Zishan Shao, Kangning Cui et al. (+5 more)
SVD-based Low-rank compression reduces
transformer parameters and nominal FLOPs, but these savings often translate poorly into real LLM serving speedups. We show that this gap is largely a runtime problem: factorized checkpoints fragment execution paths, and the resulting overhead differs substantially between prefill and autoregressive decode. We present FlashSVD v1.5, a unified inference runtime for serving SVD-compressed
transformers. FlashSVD v1.5 maps diverse public SVD compression families to a common factorized representation and combines phase-specific kernels with dense-KV decode, packed MLP execution, and per-layer CUDA-graph replay to reorganize the low-rank serving path into a thin runtime. Across representative decoder-serving settings, FlashSVD v1.5 achieves up to 2.55x decode and 2.39x end-to-end speedup, and it attains 1.48x average decode and 1.44x average end-to-end speedup across multiple popular SVD compression families. These results suggest that practical low-rank acceleration requires runtime co-design, not compression
algorithms alone. Our code is available at: https://github.com/Zishan-Shao/FlashSVD.
Read the full abstract ▾
All authors Wenhao Wu, Zishan Shao, Kangning Cui, Jinhee Kim, Yixiao Wang, Hancheng Ye, Danyang Zhuo, Yiran Chen
Machine Learning · Cross-listed from cs.CR
Seed Hijacking of LLM Sampling and Quantum Random Number Defense
Ziyang You, Xiaoke Yang, Zhanling Fan et al. (+3 more)
Large
language models (LLMs) rely on deterministic pseudorandom number generators (PRNGs) for autoregressive sampling, creating a critical supply-chain attack surface overlooked by existing defenses. We present SeedHijack, a
backdoor attack that manipulates PRNG outputs to force attacker-specified
token selection without altering model logits. In a 540-trial
benchmark on GPT-2 (124M), the attack achieves 99.6% exact
token injection across 9 sampling configurations; it reaches 100% success on four aligned models (1.5B-7B, RLHF/SFT/
reasoning distillation) and bypasses all
alignment methods tested in this work. We further propose a defense based on a hardware
quantum random number generator (QRNG), which neutralizes the attack in our evaluated threat model with negligible median overhead (+0.6% latency, +7.7 MB
memory). Our work identifies a critical sampling-layer vulnerability and provides a practical, deployable QRNG-based defense.
Read the full abstract ▾
All authors Ziyang You, Xiaoke Yang, Zhanling Fan, Feng Guo, Xiaogen Zhou, Xuxing Lu
Machine Learning
Revitalizing the Beginning: Avoiding Storage Dependency for Model Merging in Continual Learning
Xi Wang and Cheng Deng
Model merging provides a compelling paradigm for integrating specialized expertise into a unified multi-task model, a goal that aligns naturally with the sequential knowledge acquisition in continual learning (CL). However, the requirement for preserving diverse forms of previous knowledge conflicts with the storage limitations inherent to CL. In this paper, we systematically analyze existing model merging methods under the constraints of CL. We find that current methods prioritize global
alignment, which often leads to the accumulation and amplification of task-specific errors within the continuous data stream; and the vanishing gradients at the onset of subsequent tasks frequently cause optimization to stagnate. These leave the merged model in a suboptimal state at the beginning of the next training phase. To address these challenges, we propose Trajectory Regularized Merging (TRM), a framework that reformulates the merging phase as an optimization process within an augmented trajectory subspace. Our framework integrates three synergistic objectives including task
alignment, prediction consistency, and gradient responsiveness to concurrently preserve merged model's historical stability and re-activate optimization dynamics. Extensive experimental results demonstrate that our method achieves state-of-the-art performance across multiple
benchmarks.
Read the full abstract ▾
Machine Learning
Practical Wi-Fi-based Motion Recognition Under Variable Traffic Patterns
Guolin Yin, Junqing Zhang, Guanxiong Shen et al. (+1 more)
Wi-Fi sensing detects human motions and activities by analysing the channel state information (CSI) derived from Wi-Fi transmissions. However, the impact of variable transmission traffic, which dictates the effective sampling rate and interval, is often overlooked. Existing Wi-Fi sensing systems are trained with fixed input size and sampling rate, which suffer from poor sampling rate generalisation. This paper proposes a novel Wi-Fi sensing approach for motion recognition applications, e.g., gesture and activity recognition, under variable traffic patterns. A sampling rate versatile neural network (SRV-NN) based on the
transformer is proposed to efficiently handle variable input-sized sensing signals. A dynamic sampling rate augmentation is employed for variable sampling rates and intervals. To validate our approach, we have carried out extensive experimental evaluation, using two self-collected
datasets, namely SRV activity and SRV gesture, as well as two publicly available
datasets. Our method demonstrated exceptional performance and stability under variable sampling rates, with substantial improvements in average accuracy compared to baseline models without augmentation. The proposed approach significantly enhances stability by greatly reducing accuracy variance across different sampling rates.
Read the full abstract ▾
All authors Guolin Yin, Junqing Zhang, Guanxiong Shen, Simon L. Cotton
Machine Learning · Cross-listed from eess.IV
Non-intrusive Body Composition Assessment from Full-body mmWave Scans
Miriam Senne, Benjamin D. Killeen, Tony Danjun Wang et al. (+1 more)
Body composition assessment (BCA) provides detailed information about the distribution of different tissue types in the body, enabling more precise characterization of individuals than BMI or weight alone. Consistent and frequent BCA would be valuable for personalized medicine, but the gold standard methods for BCA, such as CT and MRI, are only practical for opportunistic monitoring of patients with clinical indications for imaging and are not suitable for routine use in the general population. Here, we consider an imaging modality which is not currently used in medical applications: millimeter wave (mmWave) radar. Commonly used in security settings, mmWave scans enable fast, non-intrusive, and
privacy-preserving reconstruction of full body shape without the need to remove clothing. To demonstrate the feasibility of fast and convenient BCA from mmWave scans, we present a method for BCA value regression using a multi-task learning strategy that leverages synthetic mmWave-like point clouds derived from clinical imaging and parametric human models. We evaluate the model on a pilot cohort of real mmWave scans with bioimpedance-derived body fat measurements, supporting the feasibility of estimating VAT and body fat percentage (BFP) from mmWave data acquired through clothing in a standing posture. We find that the model can predict VAT and BFP with a mean absolute error of 1.0 L and 3.2%, respectively, demonstrating the potential of mmWave scanning for routine BCA in a wide range of settings.
Read the full abstract ▾
All authors Miriam Senne, Benjamin D. Killeen, Tony Danjun Wang, Nassir Navab
Machine Learning
LLMSYS-HPOBench: Hyperparameter Optimization Benchmark Suite for Real-World LLM Systems
Siyu Wu, Yulong Ye, Zezhen Xiang et al. (+3 more)
Large
Language Model (LLM) systems have been the frontier of AI in many application domains, leading to new challenges and opportunities for hyperparameter optimization (HPO) for the AutoML community. However, this type of system exhibits an unprecedented compound space of hyperparameter configuration from both the AI and non-AI components; rich and nonlinear implications from the fidelity factors; and diverse costs of measuring hyperparameter configurations, none of which have been fully captured in existing
benchmarks. This paper presents the first (live)
benchmark suite and
datasets for HPO of real-world LLM systems, dubbed LLMSYS-HPOBench, covering data related to the inference objective values of hyperparameter configurations profiled from running the LLM systems. Currently, LLMSYS-HPOBench contains 364,450 hyperparameter configurations with a dimensionality of 12-23, 3-5 dimensions of fidelity factor leading to 932 settings, 3-9 inference objective metrics, and 2-10 cost metrics, together with generated logs from measuring the LLM systems. What we seek to advocate is not only a revalidation of the existing HPO
algorithms over the frontier LLM systems, but also to provide an evolving platform for the AutoML community to explore new directions of research in this regard. The
benchmark suite has been made available at: https://github.com/ideas-labo/llmsys-hpobench
Read the full abstract ▾
All authors Siyu Wu, Yulong Ye, Zezhen Xiang, Pengzhou Chen, Gangda Xiong, Tao Chen
Machine Learning
GNN for Structural Displacement Prediction
Hung-Fu Chang, Tzu-Kang Lin and Yung-Li Cheng
Accurate prediction of structural displacements under external loading is fundamental to structural health monitoring and seismic safety assessment. Although the finite element method (FEM) remains the prevailing approach because of its high accuracy, its considerable computational cost restricts its suitability for real-time monitoring applications. To address this limitation, this study proposes a data-driven framework based on Graph Neural Networks (GNNs), in which structural systems are represented as graphs with joints modeled as nodes and structural members as edges. By incorporating both
geometric and mechanical properties into the graph representation, the proposed model learns the relationship between applied loads and structural responses directly from simulated data. A synthetic
dataset was generated from a two-story frame structure using ANSYS, and both a conventional Neural Network (NN) and a GNN were trained for comparison. The results show that the proposed GNN framework predicts displacements and rotations with high accuracy and outperforms the NN model, demonstrating its potential as a fast and efficient alternative to traditional FEM-based analysis.
Read the full abstract ▾
Machine Learning
SGC-RML: A reliable and interpretable longitudinal assessment for PD in real-world DNS
Wenbin Wei, Ruixiang Gao, Suyuan Yao et al. (+3 more)
Real-world digital Parkinson's disease assessment faces challenges such as heterogeneous modalities, cross-device bias, and incomplete labeling. Existing methods often focus on average predictive performance, lacking the reliability mechanisms needed for retrospective reliability-aware assessment - namely, determining when the model is reliable, when to reject an assessment, when to retest, and from which symptom dimensions the predictions are based. This paper proposes SGC-RML, which maps
speech, gait, wearable motion, mobility tasks, and clinical variables to a shared 8-dimensional symptom node space (7 clinical symptom nodes and 1 reliability_state auxiliary node), unifying motor and non-motor representations through a symptom atlas. By jointly introducing uncertainty estimation, conformal calibration, and selective decision routing, the model can not only predict symptoms and severity but also reject assessments or suggest retests when evidence is insufficient. We validate this framework on five real-world PD
datasets, covering classification, regression, event detection, and longitudinal severity prediction. Experiments show that SGC-RML achieves an MAE of 4.579 / R^2 of 0.772 on PPMI, an AUC of 0.953 on mPower, and an AUC of 0.825 on PADS. Under leak-free temporal anchoring, as few as 5 subject-specific anchors transform UCI from an essentially non-predictive subject-independent setting (motor MAE 8.38, CCC 0.02) into a calibrated longitudinal assessment (motor MAE 3.24, CCC 0.756) with split-conformal coverage held at the 0.80 target. Under the Daphnet LOSO protocol, it achieves an F1 of 0.803 / AUC of 0.872. These results demonstrate that SGC-RML provides a unified paradigm for accurate, calibrated, auditable, and symptom-interpretable retrospective longitudinal assessment of PD under incomplete
multimodal conditions.
Read the full abstract ▾
All authors Wenbin Wei, Ruixiang Gao, Suyuan Yao, Xuanzhen Zhao, Cheng Huang, Hen-Wei Huang
Machine Learning
Priming: Hybrid State Space Models From Pre-trained Transformers
Aditya Chattopadhyay, Elvis Nunez, Prannay Kaul et al. (+6 more)
Hybrid State-Space models combine
Attention with recurrent State-Space Model (SSM) layers, balancing eidetic
memory from
Attention with compressed fading
memory from SSMs. This yields smaller Key-Value caches and faster decoding than
Transformers, along with a richer architectural design space. Exploring that design space at scale has so far required training from scratch, a barrier that has kept most large-model Hybrid research within a narrow range of architectures. We introduce Priming, a method that turns Hybrid architecture design from a pre-training problem into a knowledge transfer one. Priming initializes a Hybrid model from a pre-trained
Transformer and, through short
alignment and post-training phases, recovers downstream quality using less than 0.5% of the source model's pre-training
token budget. Priming is agnostic to the source
Transformer family (e.g., Qwen, Llama, Mistral), model class (dense or Mixture-of-Experts), and model scale. Priming enables us to run the first controlled comparison of SSM layer types at scale under identical conditions. We evaluate, Gated KalmaNet (GKA), Gated DeltaNet (GDN), and Mamba-2, and show that their expressiveness hierarchy, GKA>GDN>Mamba-2, directly predicts downstream performance on long-context
reasoning tasks. We scale Priming to 8B/32B
reasoning models with native 128K contexts. Our Hybrid GKA 32B improves over its source Qwen3-32B by +3.8 average
reasoning points, while staying within 1% of a
Transformer post-trained on the same data and enabling up to 2.3x higher decode throughput. To foster research on Hybrid architectures, we release a model zoo of primed Hybrid models for long-context
reasoning and instruction following, together with the Priming training and inference code (Sequence Parallelism
algorithms for long-context training, optimized GKA kernels, and vLLM serving plugin), all under Apache~2.0 License.
Read the full abstract ▾
All authors Aditya Chattopadhyay, Elvis Nunez, Prannay Kaul, Benjamin Bowman, Evan Becker, Luca Zancato, David Thomas, Wei Xia, Stefano Soatto
Machine Learning
mHC-SSM: Manifold-Constrained Hyper-Connections for State Space Language Models with Stream-Specialized Adapters
Abdulvahap Mutlu, Şengül Doğan and Türker Tuncer
Manifold-Constrained Hyper-Connections (mHC) introduce a stability-motivated variant of multi stream residual mixing by constraining residual stream mixing
matrices to the
manifold of doubly
stochastic matrices via Sinkhorn-Knopp projection. In his work, we study whether mHC-style constrained multi-stream residual
topology transfers effectively to state space model (SSM)
language modeling. We implement a static mHC mechanism around an SSM block by expanding the residual stream into multiple parallel streams, aggregating streams into a single SSM input through simplex-constrained pre-mixing, scattering the SSM output back to streams through simplex-constrained post-mixing, and applying Sinkhorn-projected residual stream mixing at each layer. We further introduce stream-specialized adapters that add lightweight stream-specific capacity through a shared bottleneck with per-stream scaling, applied both before stream aggregation and after the SSM output prior to scattering. We evaluate baseline single-stream SSM, static mHC SSM, and mHC SSM with adapters on WikiText-2 using identical training settings and report checkpoint-based validation loss, perplexity, throughput, and peak GPU
memory. Under the reported fair checkpoint evaluation, static mHC improves validation loss from 6.3507 to 6.2448 and reduces perplexity from 572.91 to 515.35, while mHC with adapters further improves validation loss to 6.1353 and perplexity to 461.88. These gains are accompanied by modest throughput reductions from 1025.52 to 964.81 and 938.90
tokens per second, and increased peak
memory from 2365 MB to 2568 MB and 3092 MB. The results suggest that mHC-inspired constrained multi-stream residual mixing can yield measurable quality improvements in SSM
language models and that stream-specialized adapter capacity can further enhance performance with predictable efficiency tradeoffs.
Read the full abstract ▾
Machine Learning
What Cohort INRs Encode and Where to Freeze Them
Vasiliki Sideri-Lampretsa, Sophie Starck, Robbie Holland et al. (+2 more)
Reusing the early layers of cohort-trained INRs as initialization for new signals has been shown to accelerate and improve signal fitting, yet it remains unclear which layers of the shared encoder learn transferable representations and what those representations encode. We address both questions for two standard backbones, SIREN and
Fourier-feature MLPs (FFMLP). First, sweeping the freeze depth across the shared encoder at test time, we find that the optimum coincides with the layer of highest weight stable rank. Moreover, freezing at this depth matches or improves on the standard fine-tuning recipe across all our experiments. Second, identifying which layer transfers does not characterize what that layer encodes. To address this we adopt sparse autoencoders (SAEs), the dominant tool in mechanistic interpretability, and present the first SAE decomposition of INR activations into sparse dictionary atoms. Interestingly, SIREN and FFMLP achieve comparable cohort-fitting quality, but learn qualitatively different dictionaries. Cohort SIREN's atoms are localized, tiling the coordinate plane such that each atom fires in a confined region independent of cohort content. Cohort FFMLP's atoms are image-spanning, tracing the contours of memorized cohort signals. Single-atom ablations confirm causal use of these dictionaries: a single FFMLP atom out of 4096 can drop PSNR by up to 10.6 dB across the image, while SIREN ablations remain confined to where the atom fires. Together, these results give the first mechanistic account of what transfers in cohort-trained INRs and turn their activations into inspectable dictionary atoms. These tools open a path towards characterizing what INRs encode and towards architectures designed for generalization rather than memorization.
Read the full abstract ▾
All authors Vasiliki Sideri-Lampretsa, Sophie Starck, Robbie Holland, Julian McGinnis, Daniel Rueckert
Machine Learning
A Qualitative Test-Risk Mechanism for Scaling Behavior in Normalized Residual Networks
Daning Cheng, Zeyu Liu, Jun Sun et al. (+4 more)
The scaling behavior, in which test performance often improves as model size and data increase, is a central empirical phenomenon in modern deep learning, yet its theoretical basis remains incomplete. In this paper, we study depth expansion in normalized residual networks: starting from a trained model in an old hypothesis class, we insert a new residual block at an intermediate layer and ask when such an expansion can yield a provable improvement in test risk. We develop a unified framework that decomposes this question into representational gain, optimization gain, and generalization transfer. First, under a first-order descent condition near zero initialization, we prove that the expanded hypothesis class contains an auxiliary jumpboard model with strictly smaller population risk than the original model. Second, under norm control tailored to post-normalized residual architectures, we establish a norm-based Rademacher complexity bound for the expanded model class. These ingredients lead to two complementary test-risk guarantees: one route passes through population risk and is tighter when a positive population margin is available, while the other works directly at the train/test level, avoids Hoeffding transfer, and is more robust in degenerate regimes. Together, these results provide a
theorem-driven mechanism under which residual depth expansion can improve test performance in normalized residual networks. More broadly, they suggest that scaling is inherently joint: depth creates new improving directions, width enhances the finite-sample observability of weak signals, and data determines whether the statistical cost of expansion can be controlled.
Read the full abstract ▾
All authors Daning Cheng, Zeyu Liu, Jun Sun, Fen Xia, Boyang Zhang, Dongping Liu, Yunquan Zhang
Machine Learning
In-Context Fixation: When Demonstrated Labels Override Semantics in Few-Shot Classification
Ming Liu
While random demonstration labels barely hurt in-context learning (Min et al., 2022), we show that homogeneous labels–even semantically valid ones–collapse accuracy to <=12% across six models (Pythia, Llama, Qwen; 0.8B–8B) and four tasks. The trigger is label-slot content: the model treats
tokens occupying the label position as an exhaustive answer vocabulary, with homogeneity as the maximally collapsed case. A novel set-level fixation finding confirms this: when demonstrations carry varied nonsense
tokens from {foo,bar,vex,nit,orb}, the model places 42–67% of probability on the demonstrated set while P(dog) remains below 0.2%. This is inconsistent with latent-concept
Bayesian accounts (Xie et al., 2022) and reveals that ICL output is constrained vocabulary
retrieval–the model binds its output to the demonstrated
token inventory regardless of semantic plausibility. The effect generalizes to 4-way classification (0% accuracy across three models, 1B–8B) and multi-
token verbalizers ("very positive"), where we decompose fixation into format-level (template adoption) and content-level (polarity override) components that are experimentally dissociable. Mechanistically, per-item paired activation patching on Pythia-1B recovers 98.4% of the gap (95% CI [84%, 112%]), localizing fixation to a layer-7-centered
circuit (rank 2/560, 99.8th percentile; 4-fold CV mean 103%). Cross-architecture logit lens on Llama-3.2-1B replicates the encode-then-override trajectory with causal confirmation (top-5 layers: 89% recovery).
Read the full abstract ▾
Machine Learning
Hierarchical Mixture-of-Experts with Two-Stage Optimization
Gleb Molodtsov, Alexander Miasnikov and Aleksandr Beznosikov
Sparse Mixture-of-Experts (MoE) models scale capacity by routing each
token to a small subset of experts. However, their routers exhibit a fundamental trade-off: strong load balancing can suppress expert specialization, while aggressive diversity often causes routing collapse. We propose Hi-MoE, a grouped MoE framework that decomposes routing control into two coupled levels: (i) inter-group balancing that enforces fair traffic across expert groups, and (ii) intra-group specialization that promotes complementary expert behaviors while preventing within-group collapse. Our analysis provides a principled explanation of how our hierarchical objectives reshape the router, thereby promoting stable specialization and mitigating collapse. We observe consistent improvements over recent sparse-routing and grouped-MoE baselines across NLP and
vision benchmarks, and confirm robustness via scaling studies (model size, expert count) and targeted ablations. In large-scale pre-training on 58B
tokens, Hi-MoE-7B achieves a 5.6% perplexity reduction and a 40% improvement in expert balance over OLMoE-7B across diverse evaluation domains.
Read the full abstract ▾