Machine Learning · Cross-listed from math.NA
A Systematic Analysis of Automatic Differentiation versus Discretization-based Constraints for Physics-Informed PDE Solvers
Xing Guo, Hongwei Tang, Zewei Meng et al. (+3 more)
Physics-informed neural networks (PINNs) represent a growing frontier in using artificial intelligence to solve partial differential equations (PDEs). Automatic differentiation (AD) plays a central role in this paradigm, which is mesh-free and replaces traditional iterative solvers with gradient-based optimization in continuous space. However, the inherent limitations of AD, particularly in handling higher-order derivatives and discontinuous solutions, pose significant challenges for complex problems. This has motivated a growing number of researchers to explore discretization-based constraints as an alternative path. Yet, the respective applicability of these two paradigms remains largely unexplored. In this work, we conduct systematic experiments across a wide spectrum of problems, from simple linear Poisson to high-Mach hypersonic flows with strong discontinuities. Through a rigorous decomposition of approximation, optimization, and truncation errors, we systematically elucidate the fundamental trade-offs and error-governing mechanisms of both paradigms, as well as two representative network architectures: multi-layer perceptron (MLP) and graph neural network (GNN). Our results reveal a consistent trend: as nonlinearity strengthens, the accuracy advantage of discretization-based constraints becomes increasingly pronounced, with smaller optimization errors compensating for the truncation errors. Moreover, the more complex the nonlinearity and boundary conditions, the greater the advantage of GNN over MLP. These insights offer a robust practical guideline for configuring neural PDE solvers in demanding engineering applications. Our source data and code are available at https://github.com/guoxing0809/neuropde_analysis.
Read the full abstract ▾
All authors Xing Guo, Hongwei Tang, Zewei Meng, Yidong Zhang, Shaoqiu Xiao, Feng Liu
Machine Learning
Revisiting Thinning Methods for Kernel Learning Problems
Blanca Cano-Camarero, Yago R. Aguado-Carrillo-de-Albornoz, Ángela Fernández-Pascual et al. (+1 more)
Kernel methods are widely used because of their strong theoretical guarantees and empirical performance. However, their high computational cost limits their applicability to large-scale
datasets. To address this shortcoming, several approaches use Maximum Mean Discrepancy to construct representative subsets that preserve the properties of the full
dataset in a Reproducing Kernel
Hilbert Space. We introduce Backward Kernel Herding, an
algorithm that addresses this problem by iteratively removing points from the
dataset, achieving results comparable to current state-of-the-art approaches while accelerating the subsampling process in realistic scenarios where the reduced size is less than half of the
dataset. Moreover, we overcome a limitation of Kernel Thinning by proposing an extension that enables the construction of subsets of arbitrary size rather that restricting to successive halvings. Finally, we conduct an extensive experimental comparison focusing on the most relevant kernel learning procedures: Gaussian Processes and Kernel Support Vector Machines. The results show that Backward Kernel Herding consistently achieves competitive performance with the most favorable training-time efficiency, while the proposed Flexible Kernel Thinning frequently achieves the best predictive performance. These gains become especially pronounced for moderate compression ratios, highlighting the benefits of incorporating supervised information into the thinning process. In terms of
memory consumption, Flexible Kernel Thinning is also competitive, whereas Backward Kernel Herding remains an alternative when computational efficiency is the primary objective. Overall, no single method dominates across all scenarios, underscoring the importance of selecting the reduction strategy according to the desired trade-off between predictive performance, training cost, and
memory requirements.
Read the full abstract ▾
All authors Blanca Cano-Camarero, Yago R. Aguado-Carrillo-de-Albornoz, Ángela Fernández-Pascual, José R. Dorronsoro
Machine Learning · Cross-listed from eess.SP
Masking Radar Cognition under Adversarial Surveillance: A Distributional Privacy Framework
Sreedevi K, Nandhini K, Anup Aprem et al. (+1 more)
In this article, we propose an online electronic counter-countermeasure (ECCM) framework designed to conceal the strategic decision-making processes of a cognitive radar (CR) operating under
adversarial surveillance. We model the CR under two distinct decision paradigms: a static constrained utility-maximizing behavior and a dynamic expected utility-maximizing behavior. The radar's utility function is modeled via a von Mises–Fisher (vMF) distribution, with the distributional parameter constituting the private information to be protected from
adversarial inference. We adopt a distribution
privacy framework to conceal this private information and provide formal distribution
privacy guarantees for cognition masking. In this work, we develop cognition-hiding
algorithms for both static constrained utility maximization (WDPCH-SU), and dynamic expected utility maximization (WDPCH-DU). Through rigorous mathematical analysis, we show that both WDPCH-SU and WDPCH-DU satisfy $\epsilon$-distribution
privacy ($\epsilon$-DistP) against inference-based
adversarial attacks and present the
privacy–performance trade-off bounds, quantifying utility loss (in static setting) and expected utility deviation (in dynamic setting) as functions of $\epsilon$. Numerical results show that WDPCH-SU gives about 15% improvement in utility loss at maximum
privacy compared to the existing methodology while WDPCH-DU achieves a greater reduction in
adversarial Fisher information without requiring explicit Fisher information constraints, at a moderate, analytically bounded utility deviation. These results are highly promising in many 6G communication scenarios such as network slicing for automated driving and swarm
UAV coordination, where it is essential to keep the resource allocation policy robust against
privacy attacks.
Read the full abstract ▾
All authors Sreedevi K, Nandhini K, Anup Aprem, Deepthi P P
Machine Learning · Cross-listed from cs.CV
RelightFormer: Feed-forward Generative Transformer for Multiview Object Relighting
Hejun Wang, Jinxi Li, Junwei Jiang et al. (+4 more)
Image relighting is traditionally tackled via complex inverse rendering pipelines, which suffer from ill-posed optimization, or single-image generative models that ignore crucial multi-view cues necessary for understanding 3D geometry and material interactions. To address these limitations, we introduce a feed-forward generative
Transformer for direct single- and multi-view image relighting that entirely bypasses explicit intrinsic property estimation. Adapted from a
video foundation model, our architecture features a latent illumination module that dynamically injects target environment maps into spatial features via cross-
attention. Furthermore, we employ permutation-invariant positional encodings to symmetrically process unordered multi-view inputs without sequential bias. To train this robust data-driven model, we construct the massive Laval Objaverse
Dataset (LOD), comprising 90K objects and 39K unique illuminations. Extensive experiments demonstrate state-of-the-art visual quality, photorealistic relighting quality, and strong zero-shot generalization across single-view, multi-view, and novel-view relighting tasks.
Read the full abstract ▾
All authors Hejun Wang, Jinxi Li, Junwei Jiang, Shiwei Mao, Hu Cheng, Shouwang Huang, Bo Yang
Machine Learning · Cross-listed from cs.CV
Multi-label versus multi-class classification of blood cells and their aggregates in microfluidic channels
Igor Zingman, Shada Abuhattum, Sara Kaliman et al. (+8 more)
Deformability cytometry (DC) is a type of imaging flow cytometry, which uses a camera-equipped device to measure cellular stiffness in addition to other cellular properties at high throughput. Cellular properties such as area and elongation can identify
cell types, but this requires prior knowledge of distinguishing properties and cannot be applied to clinically important
cell aggregates. Using DC data, we evaluated conventional multi-class (MC) classification and introduced a multi-label (ML) approach for identifying blood
cells and their aggregates. In particular, an ML classifier can simultaneously assign multiple
cell-type labels to a single imaged event. We show that, unlike MC classification, ML classification can identify
cell aggregates not represented in the training data. It also avoids the need for exhaustive, strictly defined aggregate labels, thereby simplifying and speeding up annotation. Since automated blood analyzers do not reliably analyze
cell aggregates, our approach may help address this clinical gap.
Read the full abstract ▾
All authors Igor Zingman, Shada Abuhattum, Sara Kaliman, Maximilian Schlögel, Paul Müller, Markéta Kubánková, Nadine Ströhlein, Manuela Hauke, Lena Schnörer, Martin Kräter, Jochen Guck
Machine Learning · Cross-listed from cs.AI
RAFM-SER++: A Lightweight Multimodal Emotion Recognition Framework for Real-Time Behavioral Monitoring in Surveillance Systems
Ngo Truong Dinh, Tung-Lam Bui, Chi-Trung Duong et al. (+3 more)
Recent
multimodal Speech Emotion Recognition (SER) systems achieve high accuracy through interaction-heavy cross-modal
transformers, but their computational cost limits deployment in latency-sensitive and resource-constrained surveillance systems. To address this challenge, we propose RAFM_SER++, a lightweight
multimodal SER framework featuring an asymmetric Residual
Attention Fusion Mechanism (RAFM). Rather than relying on computationally expensive bidirectional interactions, RAFM injects affective
speech cues into semantic text representations through a one-directional residual
attention pathway. Combined with a BYOL-inspired cross-modal
alignment objective and
attention-guided pooling, the proposed framework improves
multimodal representation learning while maintaining low computational overhead. Experiments on the IEMOCAP and ESD
benchmarks demonstrate that RAFM_SER++ consistently outperforms the HuBERT-Base baseline and achieves a superior accuracy-efficiency trade-off compared with the state-of-the-art MemoCMT. Specifically, RAFM_SER++ reduces trainable parameters by more than 60%, achieves faster inference (79.60 it/s), and attains BACC scores of 81.10% on IEMOCAP and 95.39% on ESD. These results indicate that lightweight asymmetric
multimodal fusion is an effective alternative to interaction-heavy cross-modal
transformers for real-time surveillance applications.
Read the full abstract ▾
All authors Ngo Truong Dinh, Tung-Lam Bui, Chi-Trung Duong, Vien Nguyen Thi, Viet-Anh Nguyen, Phuc-Lu Le
Machine Learning
Impact of canny edge detection preprocessing on performance of machine learning models for Parkinson's disease classification
Sameer Bhat and Piotr Szczuko
This study investigates the classification of individuals as healthy or at risk of Parkinson's disease using machine learning (ML) models, focusing on the impact of
dataset size and preprocessing techniques on model performance. Four
datasets are created from an original
dataset: DS_0, (normal
dataset), DS_1 (DS_O subjected to Canny edge detection and Hessian filtering), DS_2 (augmented DS_0), and DS_3 (augmented DS_1). We evaluate a range of ML models-Logistic Regression (LR), Decision Tree (DT), Random Forest (RF), Gradient Boosting (GB), XGBoost (XBG), Naive Bayes (NB), Support Vector Machine (SVM), and AdaBoost (AdB)-on these
datasets, analyzing prediction accuracy, model size, and prediction latency. The results show that while larger
datasets lead to increased model
memory footprints and prediction latencies, the Canny edge detection preprocessing supplemented by Hessian filtering (used in DS_1 and DS_3) degrades the performance of most models. In our experiment, we observe that Random Forest (RF) maintains a stable
memory footprint of 61 KB across all
datasets, while models like KNN and SVM show significant increases in
memory usage, from 5.7-7 KB on DS_0 to 102-220 KB on DS_2, and similar increases in prediction time. Logistic Regression, Decision Tree, and Naive Bayes show stable
memory footprints and fast prediction times across all
datasets. XGBoost's prediction time increases from 180-200 ms on DS_0 to 700-3000 ms on DS_2 (truncated)
Read the full abstract ▾
Machine Learning
Think Wider: Mitigating Latent Rank Collapse in Implicit Chain-of-Thought Reasoning
Yuwen Hao and Menglin Yang
Chain-of-thought (CoT)
reasoning improves the
reasoning ability of large
language models by introducing intermediate computation, but explicit rationales increase decoding length, latency, and context cost. Implicit CoT offers a more efficient alternative by moving intermediate
reasoning into continuous latent states. However, latent
reasoning can be unstable: successive latent states may become overly similar and collapse toward a shared dominant direction, reducing the diversity of the
reasoning trajectory. In this work, we identify $\textit{latent rank collapse}$ and propose $\textbf{WIDER}$, a lightweight spectral regularizer for implicit CoT. During training, WIDER estimates the shared direction of each latent trajectory and penalizes projections onto this direction, encouraging latent states to span a broader representational subspace. The method is plug-and-play and leaves the backbone model, latent schedule, and inference-time decoding procedure unchanged. We further formulate this collapse as a
geometric bottleneck in implicit
reasoning, casting its mitigation as a training-time regularization problem rather than an inference-time decoding change. Extensive experiments show that WIDER improves matched implicit CoT baselines, while mechanistic analyses reveal higher effective rank, lower dominant-direction energy, and reduced redundancy among latent steps. These results highlight latent subspace utilization as an important factor for efficient continuous
reasoning, providing a
geometric perspective for analyzing and improving implicit CoT. Code is available at https://github.com/whitesweater/WIDER.
Read the full abstract ▾
Machine Learning · Cross-listed from cs.CV
Federated Binary Gating with Server-Side Vision-Language Inference for Surveillance Anomaly Classification
Côme-Alexis Puech, Sébastien Thuau, Amira Gran et al. (+3 more)
Privacy-sensitive surveillance systems could benefit from large
vision-
language models (VLMs), but such models typically require centralized access to raw
video. In
federated learning settings, this challenge is amplified by non-independent and identically distributed (non-IID) client data, which can make direct multiclass anomaly classification unstable, especially for rare categories. We propose a hybrid two-stage architecture that combines a
federated binary convolutional neural network (CNN) gate with server-side zero-shot VLM inference. The lightweight LiteCNN3D gate performs local anomaly screening and forwards only flagged
videos to Qwen3-VL-8B, which assigns them to four anomaly metaclasses. We evaluate this design on UCF-Crime grouped into five coarse metaclasses and implement the
federated stage in a real three-node heterogeneous deployment. In the studied setting, direct
federated multiclass training collapses, whereas the proposed decomposition yields a better trade-off between classification quality and raw-
video transmission. With fixed-threshold routing, the
federated hybrid pipeline preserves nearly the same macro-averaged F1 score (F1-macro) as its centralized CNN+VLM counterpart while reducing the fraction of transmitted
videos to 51.4%, although with a lower proxy macro receiver operating characteristic area under the curve (ROC AUC) than the centralized hybrid system. A complementary sensitivity-oriented routing operating point increases macro ROC AUC from 0.673 to 0.692 and reduces the false negative rate from 29.3% to 22.9%, but decreases F1-macro from 0.503 to 0.485 while increasing transmission from 51.4% to 57.9%. These results suggest that federation is better suited to coarse local screening, while routing rules can be adjusted to trade server-side VLM usage for higher anomaly sensitivity.
Read the full abstract ▾
All authors Côme-Alexis Puech, Sébastien Thuau, Amira Gran, Arthur Mennessier, Siba Haidar, Rachid Chelouah
Machine Learning · Cross-listed from quant-ph
Riemannian Optimization for Multi-Player Quantum Games on Product Unitary Manifolds
Alireza Habibi and Setareh Maghsudi
Quantum game theory is an extension of classical game theory that uses
quantum principles in game theory. The Eisert-Wilkens-Lewenstein (EWL)
quantum game is an early example of the two-player classical Prisoner's Dilemma transformed into a
quantum Prisoner's Dilemma. In the EWL game, the players choose pure
quantum strategies represented by
unitary matrices. This extension can resolve the classical dilemma by enabling cooperative
equilibrium with higher payoff. In this paper, we first discuss the Extended EWL (EEWL) for multiplayer
quantum games with mixed strategies. In EEWL, each player controls a set of
unitary operators as
quantum actions and uses a classical mixed strategy over these actions. The payoffs are defined as expectation values of Hermitian reward operators acting on a shared
quantum state, which is generated and measured according to the EEWL protocol. We then propose the
Unitary Strategy
Matrix Exponential
Algorithm (USMEA), a geometry-aware sequential
algorithm for the EEWL mixed-strategy setting, in which each player jointly learns a trainable set of local
unitary actions and the associated classical mixing probabilities. Thereby it acts as a learning-and-control layer for multi-
agent quantum decision systems. We analyze the convergence properties of USMEA under standard smoothness and step-size conditions and validate the theory with numerical experiments. These results show how classical optimization methods can be systematically integrated into the design and analysis of engineered
quantum strategic interactions.
Read the full abstract ▾
Machine Learning · Cross-listed from stat.ML
Distributed Lag Neural Additive Models
Calle Helmersson, Shivang Pandey, Leonardo Olivetti et al. (+1 more)
We introduce Distributed Lag Neural Additive Models (DLNAMs), neural-additive analogues of Distributed Lag Non-linear Models (DLNMs) for learning nonlinear effects distributed over lags. DLNAMs replace a prespecified spline cross-basis with neural components that learn exposure–lag response surfaces, avoiding choices of basis family, dimension, and
knot placement while preserving additive interpretability and familiar distributed-lag summaries. Exp-centered input layers, smooth activations, and learned subnetwork mixtures produce smooth, locally adaptive representations; pointwise uncertainty combines a conditional last-layer Laplace approximation with between-member ensemble variation. In simulations, DLNAMs generally outperformed DLNM comparators, including penalized and treed variants, in recovering known response functions, with lower bias, stronger boundary recovery, and better-calibrated cumulative intervals; gains were largest for more demanding functions. The architecture performed consistently across sample sizes, outcome families, lag horizons, and jointly fitted multi-exposure settings, retaining recovery performance as exposures were added; fit-specific changes were largely confined to optimization, and applications recovered established empirical patterns.
Read the full abstract ▾
All authors Calle Helmersson, Shivang Pandey, Leonardo Olivetti, Elena Raffetti
Machine Learning
Inferring Urban Mobility Interactions from Aggregated Dynamics
Yi Wang, Jing Li, Jinliang Deng et al. (+5 more)
Real-time urban governance depends not only on knowing where people are, but on how they move between places, directional flows that could be conventionally resolved by tracking individuals through space, i.e., expensive to sustain and built on traces that are highly unique and readily re-identifiable. Here we show that this directional structure need not be observed to be known: aggregated counts which cities already collect retain enough information to reconstruct the temporal
evolution of origin-destination (OD)
matrix. Using an uncertainty-aware physics-informed framework, we infer future OD flows from area-level counts alone across twelve mobility
datasets from cities in the United States and China, reaching accuracy comparable to models that take historical OD
matrices as input. Probabilistic modeling corrects the systematic underestimation of sparse, high-value corridors and yields calibrated predictions consistent with observed flows. Architectures that respect the generation-before-assignment logic of transport planning recover interactions more faithfully, indicating that location-level spatial heterogeneity should be preserved before pairwise interactions are reconstructed. Because inference requires only aggregated observations after training, recovering interactions this way reduces reliance on continuous individual-level tracking, pointing toward a more deployable and less exposure-heavy basis for real-time urban intelligence.
Read the full abstract ▾
All authors Yi Wang, Jing Li, Jinliang Deng, Zhenghong Wang, Yizhi Zhang, Fan Zhang, Ivor W. Tsang, Yu Liu
Machine Learning · Cross-listed from cs.CL
Content-Based Addressing for Long Context
Mahesh Godavarti
Rotary position
embedding (RoPE) uses each
token's integer position to determine the rotation applied inside
attention. This works well for local
token order, but increasing context length creates a positional train-test mismatch: RoPE produces relative rotations at offsets not seen during training. Methods that rescale, interpolate, randomize, or bias positions specify how
attention handles those offsets, but still derive positional information from a growing
token counter. We instead divide a
token stream into units, retain ordinary RoPE positions within each unit, and assign every completed unit an address computed from its content. Adding units then applies the same learned map to new content rather than extending a positional range or an identifier table. We prove that this construction preserves local RoPE exactly, leaves the
attention comparison between two fixed
tokens unchanged when other units are inserted or reordered, and does not create new relative rotations merely because more units are added. In a character-level Tiny Shakespeare diagnostic, all-
token validation perplexity remains approximately constant from contexts of 256 to 4096 characters. A second diagnostic shows that content-based addressing can retrieve and use information from multiple serialized facts. These are controlled shallow experiments, not scale
benchmarks, but they support a direct prescription: use position to address locally and content to address across units.
Read the full abstract ▾
Machine Learning
Robust Decentralized Personalized Federated Learning via Prediction-Constrained Neighborhood Collaboration
Xiao Ma, Hong Shen, Hui Tian et al. (+2 more)
This paper proposes a robust decentralized personalized
federated learning method R-DPFL, that enables clients to reduce the impact of Byzantine attacks via robust neighborhood direction estimation and history-based update trend prediction, rather than purely aggregating client models as in the existing work. In R-DPFL, each client first computes the current-round model update by aggregating the received neighborhood update vectors. It then predicts what this update should be based on its historical values and local model changes. Finally, R-DPFL computes the difference between these two quantities, adaptively clips this difference, and adds it to the local update. We prove convergence of the learning process through rigorous analysis and show that honest clients maintain stable personalized descent dynamics under Byzantine neighbor perturbations without requiring consensus among neighboring models. Extensive experiments on CIFAR-10 demonstrate that RDPFL consistently outperforms state-of-the-art decentralized and personalized
federated learning baselines under heterogeneous and
adversarial settings.
Read the full abstract ▾
All authors Xiao Ma, Hong Shen, Hui Tian, Wenqi Lyu, Wei Ke
Machine Learning
Long-Horizon Language Model Reinforcement Learning via Progressive Point Matching
Preston Fu, Kevin Frans, Oleh Rybkin et al. (+2 more)
Current paradigms for training
language models via reinforcement learning rely heavily on sparse outcome rewards. However, as we pursue tasks that require longer and more complicated trajectories, such strategies result in slow learning. Prior work has attempted to address this problem by rewarding partial progress; however, naive formulations are often biased and converge to suboptimal policies. We show that a simple and unbiased dense reward formulation, which we term progressive point matching, scales exponentially more efficiently to long-horizon tasks by rewarding partial progress on a segment level, both theoretically and empirically via synthetic environments. We then show how progressive point matching can be practically instantiated using a single reference trajectory per task. On extremely hard math
reasoning problems, sparse outcome rewards cannot make any progress, whereas segment-level rewards enable improvements at larger test-time
token budgets when measured by success rate or pass@k.
Read the full abstract ▾
All authors Preston Fu, Kevin Frans, Oleh Rybkin, Sergey Levine, Aviral Kumar
Machine Learning
PCFlow: Physics-Conditioned Flow Matching for GPR B-Scan Image Synthesis
Zhijie Shen, Chenchen Fu, Xuanhao Chang et al. (+2 more)
Ground-penetrating radar (GPR) B-scan image synthesis is important for data augmentation,
algorithm validation, and simulation acceleration, yet generating radargrams with both visual realism and physical consistency remains challenging. Existing learning-based generative models often emphasize visual appearance but provide limited control over response geometry. In this paper, we propose PCFlow, a physics-conditioned flow matching framework for fast GPR B-scan image synthesis. The core of PCFlow is a Maxwell-informed dense physical condition field constructed from the parameterized physical model used for electromagnetic simulation, including material properties, target geometry, propagation cues, and response-domain priors. This condition field provides an interpretable interface between physical scene parameters and radar response geometry, and guides conditional flow matching in the VAE latent space toward physically feasible generation paths. We evaluate PCFlow on a gprMax-based buried-pipeline
dataset with both in-distribution and out-of-distribution test cases. Experimental results show that PCFlow generates images with more accurate response geometry and high visual fidelity, demonstrating its effectiveness for controllable and physically faithful radar image synthesis.
Read the full abstract ▾
All authors Zhijie Shen, Chenchen Fu, Xuanhao Chang, Hongtao Bai, Lili He
Machine Learning
Constitutive State-Space Modeling of Path-Dependent Plasticity: A Resolution-Consistent and Parallelizable Computational Framework
Rui Barreira, Taylan Soydan, Francesco Scipione et al. (+2 more)
Data-driven constitutive models for path-dependent plasticity are commonly formulated using nonlinear recurrent neural networks, whose sequential state
evolution limits parallel training and whose predictions may depend on the discretization of the applied strain path. We introduce a Constitutive State Space (CSS) model that reformulates structured state-space dynamics as an incremental constitutive operator. The strain increment is decomposed into magnitude and direction: the loading direction drives the latent state-space system, while the increment magnitude enters the zero-order-hold discretization of its continuous-time linear recurrence. This mechanics-tailored construction guarantees stationarity under zero increments, strongly reduces sensitivity to strain-path resolution, and retains the parallel-scan structure of S5 for efficient training on long constitutive histories. The CSS and Minimal State
Cell (MSC) architectures are compared for four multiaxial path-dependent material models including isotropic J2 plasticity, pressure-sensitive foam plasticity, and combined isotropic-kinematic hardening. CSS matches or exceeds the prediction accuracy of the MSC, including one order of magnitude lower validation losses for the plastically incompressible materials. Importantly, CSS maintains low errors across large changes in strain-path discretization, whereas the MSC error increases substantially when evaluated at coarser resolutions than used for training. CSS trains substantially faster and requires fewer strain-stress pairs to attain comparable or better accuracy. Analysis of the learned state further reveals latent structure consistent with the dimensionality of the underlying physical constitutive models. These results establish mechanics-tailored structured state-space dynamics as a computational framework for efficient and discretization-robust data-driven constitutive modeling.
Read the full abstract ▾
All authors Rui Barreira, Taylan Soydan, Francesco Scipione, Miguel A. Bessa, Dirk Mohr
Machine Learning · Cross-listed from cs.IR
Matryoshka Hash Representations for Model-Aware Compact Semantic Retrieval
Peichun Hua and Yunming Xiao
Retrieval-augmented generation (RAG) depends on dense
retrieval: each document is stored as a learned vector, and a query is answered by finding its nearest neighbors in that vector space. Keeping one full-precision vector per document is the dominant index cost at corpus scale, so
retrieval systems replace each vector with a short code of a few bytes—a step called quantization. Standard quantizers such as product quantization (PQ) pick the code that reconstructs the original vector most closely. A single code is even more useful if it serves several byte budgets at once: when its short prefixes are each directly searchable, a deployment can set its efficiency–quality operating point without re-encoding the corpus. But training all prefixes under one objective makes the early bits a compromise across budgets—short codes improve while the full-width code degrades. Quantization to low-bit representation, such as binary codes, further sharpens the conflict. We introduce Matryoshka Hash Representations (MHR), a two-stage procedure that separates full-width training from prefix organization. MHR first learns a longer binary code, then freezes the model and trains additional zero-initialized residual code adaptors for directly searchable prefixes. Documents are stored at one bit per coordinate, while queries keep continuous logits like PQ to attain sufficient expressivity. We implement the search process with FAISS FastScan. Trained on MS MARCO and zero-shot transferred to seven BEIR
datasets, MHR reaches .5561 NDCG@10 and .6535 Recall@100 at 32 bytes, surpassing the best baseline of the same budget. The advantage is more pronounced in lower budgets. The same code also strengthens two common pipelines: shortlisting candidates for full-precision reranking, and
pruning a low-storage graph index such as LEANN.
Read the full abstract ▾
Machine Learning
Dense Structural Compression of Transformers via Gauge-Correct Channel Removal
Jed A. Duersch, Naïm Es-Sebbani, Nathanaël Haas et al. (+1 more)
Inference energy per
token drives the cost and carbon footprint of deployed
transformers. It is dominated by dense
matrix products that incur fused multiply-accumulate (FMA) operations and
memory traffic. To reduce these computations while retaining dense tensors for high GPU throughput, we develop a methodology from first principles to adapt structural complexity during training to maximize inference utility per unit compute. Channel penalties drive entire tensor slices to zero to enable physical removal while preserving density and the network function. The natural approach, penalizing the norm of operator components acting through each channel, is provably destabilized by gauge freedom. We resolve this pathology with GaugeLasso: additive symmetric group-lasso penalties that recover a monotone function of product-norms when the network converges to gauge balance. Our
equilibrium analysis enables per-channel calibration to correctly suppress slices that under-perform in inference utility per unit compute. Under adaptive pressure, the network reorganizes into depth-dependent structural profiles that can be far smaller than the architecture required to learn the task. On
polynomial long division over $\mathbb{F}_{31}$, compute compresses from 148 to 255 times with perfect accuracy. On character-level
language modeling, compressed models outperform the hand-designed baseline at equal FMA. On masked autoencoding, a compression trial exposes which axes were over-provisioned and which saturated, guiding a better second design. Compaction also accelerates training monotonically as the model progresses. Post-hoc
pruning with the same utility ranking cannot reach these structures, showing that sustained pressure is central to discovery of efficient models. Retraining a discovered architecture recovers baseline quality on our statistical tasks, but fails on our exact algorithmic task.
Read the full abstract ▾
All authors Jed A. Duersch, Naïm Es-Sebbani, Nathanaël Haas, Zied Bouraoui
Machine Learning
Kolmogorov–Arnold stability for discontinuous functions
Sviatoslav V. Dzhenzher
Here we investigate the stability of the Kolmogorov–Arnold representation
theorem (KART) under
adversarial reparameterisations of the hidden layer for multivariate discontinuous and unbounded functions. Our results provide a rigorous mathematical foundation for the structural robustness of modern deep learning architectures, such as Kolmogorov–Arnold Networks (KANs), under
adversarial configurations.
Read the full abstract ▾
Machine Learning
Parallelism Strategy Chaining for Fast Training Convergence
Minchul Kang, Changyong Shin, Younghun Go et al. (+4 more)
Selecting a parallelism strategy - the configuration of data, tensor, and pipeline parallelism degrees together with micro- and global-batch sizes - largely determines the training efficiency of large
language models. State-of-the-art methods search for a parallelism strategy offline and select the single strategy that minimizes per-iteration time. But we find that they neglect the target validation perplexity and time-to-perplexity (TTP). In particular, our analysis reveals that the best strategy yielding the fastest perplexity improvement changes multiple times during training. As a result, state-of-the-art methods are 1.8-11.4x slower in TTP than the strategy sequence that selects the best strategy at each iteration. This paper proposes CONA, a new training method that introduces online strategy chaining. Instead of a single strategy selected offline, CONA ranks candidate strategies during training using a surrogate metric built from compute throughput and gradient statistics, and switches the current strategy to a new strategy with a higher metric. In our evaluation with GPT-3 1.3B, BERT-Large, and Llama-3.2-1B, CONA reaches the target validation perplexity 1.4-9.6x faster than state-of-the-art methods. Moreover, CONA closely tracks the perplexity achieved by the sequence that selects the best strategy at each iteration, within 2.6%.
Read the full abstract ▾
All authors Minchul Kang, Changyong Shin, Younghun Go, Hyunho Lee, Jinwoo Jeong, Chuck Yoo, Gyeongsik Yang
Machine Learning
Robust Decentralized Federated Distillation via Multi-Modality Knowledge Collaboration
Xiao Ma, Hong Shen, Hui Tian et al. (+2 more)
This paper propose a robust decentralized
federated distillation method that enables clients with heterogeneous models to collaborate through predictions on shared unlabeled public data. In the proposed method, each client first evaluates the received predictions in three modalities of class prediction, boundary decision, and prediction correlation. It then filters unreliable clients, assigns reliability-based weights to the retained clients, and constructs a teacher for each type of knowledge. Finally, the corresponding
distillation gradients are validated using a supervised gradient computed from private data. Conflicting prediction and boundary gradients are removed, and conflicting relation gradients are suppressed before the final model update. We prove the convergence of the proposed method by showing stable local optimization for honest clients under Byzantine
distillation. Particularly, we show that our method ensures a bounded Byzantine influence on both
distillation gradients and individual client private gradients after cross-modality fusion, thereby enabling stable local optimization for honest clienunder Byzantine
distillation. Extensive experiments on CIFAR-10 and CIFAR-100 demonstrate that the proposed method improves the prediction accuracy of heterogeneous models of clients under non-IID data and Byzantine attacks. As the booming demands of
federated learning in decentralized environments such as edge computing and mission-oriented
UAV collaborations, our method has a great potential for adoption of DFL in unreliable real-world scenarios where clients are exposed to receiver-specific Byzantine messages of malicious predictions.
Read the full abstract ▾
All authors Xiao Ma, Hong Shen, Hui Tian, Wei Ke, Wenqi Lyu
Machine Learning · Cross-listed from cs.SD
Iterative Audio Separation with Mixture Consistency via MIMO Model Extension
Yukara Ikemiya, WeiHsiang Liao and Yuki Mitsufuji
This paper proposes a general framework for stable and effective iterative audio separation with mixture consistency by extending source separation models to a multi-input multi-output (
MIMO) configuration. In the field of audio separation, mixture consistency is an essential property for many applications that require accurate phase and timbral information of target sources. While iterative approaches such as
diffusion models achieve perceptually superior results in
speech enhancement or user-guided target source separation tasks, most existing methods focus on single-step separation with a single-input single-output (SISO) or single-input multi-output (SIMO) configuration through architectural improvements, since mixture-consistent audio separation is generally regarded as a regression problem that admits a unique solution. By extending these architectures to a
MIMO configuration, we introduce iterative prediction without compromising the architectural advantages or the characteristics of mixture consistency. We conduct a comprehensive ablation study of combining the framework with discriminators and extending it to a generative model. Experimental results demonstrate significant performance improvements when applying the proposed framework to state-of-the-art separation models.
Read the full abstract ▾
Machine Learning
REFINE: Trajectory Representation Learning via Closed-Loop Transcription – Extended Version
Sean Bin Yang, Ying Sun, Jilin Hu et al. (+5 more)
Trajectory representation learning underpins a wide range of trajectory analytics tasks; however, most existing self-supervised approaches, whether discriminative or generative, adopt an open-loop paradigm, relying on fixed data augmentations or random masking without feedback, which limits their ability to generalize and scale. We propose REFINE, a simple yet effective Representation lEarning Framework vIa closed-loop traNscription rEfinement for trajectory data. Drawing upon feedback control theory, REFINE tightly couples road-network-aware generative reconstruction with feedback-driven contrastive learning, enabling the model to capture fine-grained local movement semantics and global spatio-temporal dependencies without manually designed augmentation views. We further provide a control-theoretic analysis that establishes convergence guarantees for the proposed closed-loop optimization. Extensive experiments on four real-world
datasets demonstrate that REFINE consistently outperforms state-of-the-art methods across multiple downstream tasks while remaining computationally efficient and scalable. This paper is an extended version of REFINE: Trajectory Representation Learning via Closed-Loop Transcription, to appear in KDD 2026.
Read the full abstract ▾
All authors Sean Bin Yang, Ying Sun, Jilin Hu, Zongyi Xu, Kristian Torp, Hua Lu, Bin Yang, Christian S. Jensen
Machine Learning
Protocol effects on feature-based hardware-Trojan detection across Trust-Hub families
Hang Xiao, Chuhong Xu, Kainan Zhou et al. (+2 more)
Trust-Hub reuses host
circuits: several files differ mainly in the inserted Trojan. When gates from sibling variants enter both training and test folds, a detector can benefit from host logic it has already seen. We measure that effect instead of proposing another classifier. The corpus contains 49,124 gates from 16 netlists grouped into five host families. We left the parser, 36 gate features, class weighting, model settings, threshold, and family-level aggregation unchanged and altered one choice: the test boundary. The three settings draw test gates from the pooled corpus, withhold a complete netlist, or withhold every variant of one host. The choice matters. Random forest records F1/AP of 0.914/0.978 with pooled gates, 0.636/0.851 with one netlist held out, and 0.460/0.577 with a host family held out. XGBoost falls from 0.946/0.976 to 0.464/0.544 across the same comparison. Logistic regression loses AP, although its fixed-threshold F1 is not monotonic. Each family shows the same pooled-to-family direction. Feature removal, repeated model and simulator seeds, score normalization, parser-related exclusions, and a smaller sample change the size of the gap without reversing it. Aggregation also matters: a gate-weighted average is dominated by the larger ISCAS files, so the headline values give each host family one vote. Bootstrap and jackknife summaries keep the gap positive, but their folds reuse training families. We treat the five family rows as descriptive evidence rather than independent trials. Five host families are too few for a population claim, and the experiment says nothing about transfer to a new
cell library or an industrial design. It supports a narrower conclusion: sibling
benchmark variants can inflate apparent transfer.
Benchmarks with several variants of one host
circuit should report family-aware holdouts and all five family results beside pooled scores.
Read the full abstract ▾
All authors Hang Xiao, Chuhong Xu, Kainan Zhou, Gangzhen Qian, Lu Yi
Machine Learning
FedRAW: Preserving Rare-Label Influence in Asynchronous Federated Learning
Prashant Bajpai, Divya Saxena, Philippe Lalanda et al. (+1 more)
Asynchronous
federated learning improves scalability by updating the global model from a server-side buffer of client updates as they arrive, rather than waiting for all selected clients to finish. While efficient, this arrival-driven aggregation can silently distort representation learning under heterogeneous participation. We identify silent rarity failure, a hidden failure mode in which clients holding rare labels contribute too weakly to the global model even though its overall accuracy appears largely unaffected. This failure arises from two coupled effects: rare-label clients may submit updates less frequently when they are slower or less available, creating participation bias; and once their updates enter the buffer, standard asynchronous aggregation assigns them no compensating influence, creating aggregation bias. We propose FedRAW, a fully server-side aggregation method that preserves rare-label influence without changing local training, client objectives, or communication protocols. FedRAW combines client-level update deduplication, which prevents frequently arriving clients from repeatedly dominating the update buffer, with rare-label-aware weighting, which increases the influence of clients carrying low-coverage labels. We formalize silent rarity failure through participation and aggregation bias, and show that FedRAW increases rare-label client influence over uniform aggregation while preserving convergence. Across EMNIST Balanced, CIFAR-10, HAM10000, and ISIC-2019, FedRAW improves rarelabel accuracy while preserving comparable global accuracy and adding negligible server-side computation.
Read the full abstract ▾
All authors Prashant Bajpai, Divya Saxena, Philippe Lalanda, German Vega
Machine Learning · Cross-listed from math.ST
Tensor network representations of discrete maximum entropy distributions via mean polytopes
Alex Goessmann and Martin Eigel
We present tensor network representations for discrete maximum
entropy distributions under expectation constraints. To this end, we introduce Computation-Activation Networks (CompActNets), a tensor network architecture that subsumes exponential families. By leveraging the geometry of the convex polytope of realizable expectation vectors, we represent any maximum
entropy distribution in the same architecture. We exploit the fact that proper faces of this polytope correspond to the boundary closure of exponential families, which restricts the distribution's support. We then derive explicit representations for the support within the CompActNet architecture. The proposed framework suggests tensor network ranks as complexity measures for faces. Finally, a case study on Boolean statistics links the geometry of 0/1-polytopes directly to propositional formulas.
Read the full abstract ▾
Machine Learning · Cross-listed from cs.CL
CircuitLens: Reasoning Circuits as Data Selection Signals for Reinforcement Learning with Verifiable Rewards
Zhuofan Chen, Ziqian Jiao, Yikai Cui et al. (+3 more)
Reinforcement learning with verifiable rewards (RLVR) is sensitive to which problems a model trains on, yet existing selection criteria–difficulty filtering, hand-curation, reward-trajectory scoring–assess data value as an intrinsic property of problems, independent of the model that will learn from them. We introduce
Circuit Reasoning Score (CRS), a selection signal derived from 46
reasoning-sensitive
attention heads identified via contrastive ablation, computed in a single forward pass on the frozen base model without reward labels or rollouts. CRS runs against the intuitive hypothesis that stronger
reasoning-
circuit engagement produces better training data: on Qwen2.5-Math-7B, the lowest-engagement decile improves over random selection on three medium-difficulty
benchmarks (GSM8K +2.0 pp, OlympiadBench +1.6 pp, Minerva +2.9 pp), while the highest-engagement decile gains less and is indistinguishable from the middle decile. The advantage has boundary conditions: on a domain-curated pool no selection method separates from the others; at 1.5B scale the useful direction differs; and the lowest-reward training condition produces the strongest downstream generalization. Within the Qwen2.5-Math settings tested, RLVR data selection appears regime-dependent rather than reducible to a static ranking of problem quality.
Read the full abstract ▾
All authors Zhuofan Chen, Ziqian Jiao, Yikai Cui, Zhixin Cai, Jun Bai, Wenge Rong
Machine Learning
The Oversight Gap: What LLM Safety Monitors Miss, and Why It Is Not Capability
Xin Xu
Several properties safety monitors are asked to certify, among them cross-tenant noninterference, sandbagging and evaluation awareness, are 2-safety hyperproperties, witnessed only by two executions. The standard consequence is a binary impossibility: one trace cannot decide them. We replace the binary with a measurement. A tight bound puts the balanced accuracy of any single-trace monitor at $\tfrac12+\tfrac12\,TV(P_0,P_1)$, turning undecidability into a graded detectability frontier and defining an oversight gap: a monitor's shortfall below it. On a leak family with closed-form $TV$, nine LLM monitors are optimal at $TV=0$ but capture little signal as $TV$ grows; at $TV=1$, where a 20-line membership check scores $100\%$, they average $60.9\%$. That shortfall is mostly not capability: naming what to check closes $61\%$ of it while leaving the $TV=0$ control at chance. The same split runs through a $2{\times}2$ factorial: an imagined second run leaves monitors at chance ($50.4\%$) while the same rule on an executed second run reaches $90.0\%$, and a stored oracle without a comparison procedure yields only $68.2\%$. Information and procedure are each necessary and neither is capability. Under nondeterminism, replay tracks a closed-form $k$-replay curve only under the right projection, and a projection frontier shows the resulting dilemma is forced: narrow misses $98.6\%$ of off-channel leaks, broad flags $75.7\%$ of clean traffic, and attainable accuracy decays like $1/(qm)$ in the benign-variation rate and the channel count. Finally, two frontier LLM judges certified an earlier version of our own
benchmark as sound while a sign test found a directional bias ($p=2.7\times10^{-5}$) that invalidated three of our findings. Construction validity for hyperproperty
benchmarks should be proved mechanically, not audited by models.
Read the full abstract ▾
Machine Learning · Cross-listed from cs.CV
Mind the Approximation: Fisher-Weighted SVD Compression for ViTs
Moritz Thoma, Maximilian Groezinger, Maximilian Forstenhäusler et al. (+7 more)
Model compression is key to mitigate deployment challenges of ever growing machine learning models. In this area of research, singular value decomposition (SVD)-based compression offers a compelling trade-off between computational efficiency and model accuracy. Fisher-weighted SVD in particular provides principled, loss-aware compression. However, we find that improving the fidelity of Fisher approximation used in the compression is poorly predictive of post-compression accuracy for
Vision Transformers (ViTs). Motivated by this observation, we propose FACTS, a structured Fisher Approximation tailored to Compressing ViTs with Fisher-weighted SVD, which enforces
token-local aggregation while preserving within-
token activation-gradient dependence. Additionally, we introduce a fast Constrained Rank Search (CoRS), that optimizes layer-wise rank allocation while adhering to a fixed floating point operation (FLOP) constraint. Extensive experiments across ViTs and hybrid architectures demonstrate that FACTS consistently improves accuracy-efficiency trade-offs without requiring finetuning. Notably, it outperforms the strongest SVD baseline by up to +5.8 percentage points (p.p.) Top-1 on Swin-B, with further gains driven by our search method. Code is available at https://github.com/MoritzTho/FACTS.
Read the full abstract ▾
All authors Moritz Thoma, Maximilian Groezinger, Maximilian Forstenhäusler, Emad Aghajanzadeh, Ryan Pegoud, Manoj Rohit Vemparala, Pierpaolo Mori, Alexander Frickenstein, Daniel Mueller-Gritschneder, Ulf Schlichtmann