Machine Learning
Hierarchical Sparse Plus Low Rank Compression of LLM
Pawan Kumar and Aditi Gupta
Modern large
language models (LLMs) place extraordinary pressure on
memory and compute budgets, making principled compression indispensable for both deployment and continued training. We present Hierarchical Sparse Plus Low-Rank (HSS) compression, a two-stage scheme that (i) removes the largest-magnitude weights into a sparse
matrix S and (ii) applies a recursive Hierarchically Sparse Separable (HSS) low-rank factorisation to the dense residual
matrix. A recursive rank-reducing strategy and a reverse Cuthill-Mckee (RCM) permutation are introduced to align high weights towards the diagonal with the block-diagonal hierarchy, maximising off-diagonal compressibility (because they are touched only once). HSS is hardware-friendly: its
matrix-vector multiply reduces to one sparse and a sequence of thin-
matrix multiplications and can be trained end-to-end with standard optimisers. Experiments on LLaMA-7B show that targeting only the self-
attention projections (1.6 B parameters of Q, K, and V
matrices out of a total 7B parameters) suffices to yield large
memory savings while retaining comparable state-of-the-art perplexity scores on test samples of the WikiText
dataset. For example, with a 30% sparsity budget and an outer rank of 512, sHSS-RCM achieves a perplexity of 1.64, outperforming dense baselines and classical sparse-plus-SVD variants, while also achieving significant
memory savings.
Read the full abstract ▾
Machine Learning · Cross-listed from cs.CL
Reinforcement Learning for Chain of Thought Compression with One-Domain-to-All Generalization
Hanyu Li, Jiangshan Duo, Bofei Gao et al. (+4 more)
Chain-of-thought
reasoning in large
language models can trigger an "overthinking trap": longer rollouts raise cost and latency yet often yield unreliable accuracy gains. Existing methods use global, static controls that may suppress needed
reasoning. We propose mastery-gated, sample-level, soft reinforcement learning compression that penalizes long rollouts only when the model already solves the problem and has produced a shorter rollout. Across
benchmarks, it cuts response length by 20-40% with comparable or higher accuracy and generalizes across domains: a model trained on math spontaneously shortens unseen tasks (code, instruction following, general-knowledge QA) without hurting accuracy. We further show two-way transfer between non-
agent CoT and tool-use
agents: non-
agent training reduces SWE-Bench Verified rounds by 13%, while compressing a thinking
agent cuts SWE trajectories by 67%
tokens and 52% rounds and shortens non-
agent outputs by up to 44%. Compression is thus not cosmetic brevity, but an inherent computation policy – what to keep, and what to forget.
Read the full abstract ▾
All authors Hanyu Li, Jiangshan Duo, Bofei Gao, Hailin Zhang, Sujian Li, Xiaotie Deng, Liang Zhao
Machine Learning · Cross-listed from cs.CL
WRAVAL – WRiting Assist eVALuation
Gabriel Benedict, Matthew Butler, Naved Merchant et al. (+1 more)
The emergence of Large
Language Models (LLMs) has shifted
language model evaluation toward
reasoning and problem-solving tasks as measures of general intelligence. Small
Language Models (SLMs) – defined here as models under 10B parameters – typically score 3-4 times lower than LLMs on these metrics. However, we demonstrate that these evaluations fail to capture SLMs' effectiveness in common industrial applications, such as tone modification tasks (e.g., funny, serious, professional). We propose an evaluation framework specifically designed to highlight SLMs' capabilities in non-
reasoning tasks where predefined evaluation
datasets don't exist. Our framework combines novel approaches in data generation,
prompt-tuning, and LLM-based evaluation to demonstrate the potential of task-specific finetuning. This work provides practitioners with tools to effectively
benchmark both SLMs and LLMs for practical applications, particularly in edge and private computing scenarios. Our implementation is available at: https://github.com/amazon-science/wraval.
Read the full abstract ▾
All authors Gabriel Benedict, Matthew Butler, Naved Merchant, Eetu Salama-Laine
Machine Learning · Cross-listed from q-bio.NC
Spike-Timing-Dependent Plasticity for Bernoulli Message Passing
Sepideh Adamiat, Wouter M. Kouw and Bert de Vries
Bayesian inference provides a principled framework for understanding
brain function, while neural activity in the
brain is inherently spike-based. This paper bridges these two perspectives by designing spiking neural networks that simulate
Bayesian inference through message passing for Bernoulli messages. To train the networks, we employ spike-timing-dependent plasticity, a biologically plausible mechanism for synaptic plasticity which is based on the Hebbian rule. Our results demonstrate that the network's performance closely matches the true numerical solution. We further demonstrate the versatility of our approach by implementing a factor graph example from coding theory, illustrating signal transmission over an unreliable channel.
Read the full abstract ▾
Machine Learning · Cross-listed from cs.CV
Enhancing Medical Data Analysis through AI-Enhanced Locally Linear Embedding: Applications in Medical Point Location and Imagery
Hassan Khalid, Muhammad Mahad Khaliq and Muhammad Jawad Bashir
The rapid
evolution of Artificial intelligence in healthcare has opened avenues for enhancing various processes, including medical billing and transcription. This paper introduces an innovative approach by integrating AI with Locally Linear
Embedding (LLE) to revolutionize the handling of high-dimensional medical data. This AI-enhanced LLE model is specifically tailored to improve the accuracy and efficiency of medical billing systems and transcription services. By automating these processes, the model aims to reduce human error and streamline operations, thereby facilitating faster and more accurate patient care documentation and financial transactions. This paper provides a comprehensive mathematical model of AI-enhanced LLE, demonstrating its application in real-world healthcare scenarios through a series of experiments. The results indicate a significant improvement in data processing accuracy and operational efficiency. This study not only underscores the potential of AI-enhanced LLE in medical data analysis but also sets a foundation for future research into broader healthcare applications.
Read the full abstract ▾
Machine Learning
Latent Sculpting for Zero-Shot Generalization: A Manifold Learning Approach to Out-of-Distribution Anomaly Detection
Rajeeb Thapa Chhetri, Saurab Thapa, Avinash Kumar et al. (+1 more)
Detecting previously unseen attacks remains a major challenge for machine learning-based intrusion detection systems. Deep models trained on network traffic often achieve high accuracy on known attacks but fail under distributional shift because their decision boundaries are tightly coupled to the training data distribution. We introduce Latent Sculpting, a two-stage anomaly detection framework that improves robustness by explicitly structuring the latent representation before density estimation. The first stage trains a
Transformer-based tabular encoder using a novel Binary Latent Sculpting loss, which encourages benign traffic to form a compact latent cluster while enforcing separation from anomalous patterns. The second stage fits a Masked Autoregressive Flow to the resulting latent space to produce calibrated probabilistic anomaly scores. Under a strict zero-shot evaluation protocol on the CIC-IDS-2017
benchmark, Stage 1 attains an F1-score of 0.98 on known attacks, while Stage 2 – evaluated at the balanced threshold (85th-percentile) – achieves a zero-shot OOD F1-score of 0.867 and AUROC of 0.913. The model successfully detects difficult distribution shifts including stealthy infiltration attacks (78.7% recall, peaking at 97.2%) and low-volume DoS variants (>94% recall), scenarios where conventional approaches often fail. Our results suggest that explicitly separating latent geometry learning from density modeling provides a stable approach for detecting zero-day cyber threats.
Read the full abstract ▾
All authors Rajeeb Thapa Chhetri, Saurab Thapa, Avinash Kumar, Zhixiong Chen
Machine Learning
Wireless Traffic Prediction with Large Language Model
Chuanting Zhang, Haixia Zhang, Jingping Qiao et al. (+2 more)
The growing demand for intelligent, adaptive resource management in next-generation wireless networks has underscored the importance of accurate and scalable wireless traffic prediction. While recent advancements in deep learning and foundation models such as large
language models (LLMs) have demonstrated promising
forecasting capabilities, they largely overlook the spatial dependencies inherent in city-scale traffic dynamics. In this paper, we propose TIDES (Traffic Intelligence with DeepSeek-Enhanced Spatial-temporal prediction), a novel LLM-based framework that captures spatial-temporal correlations for urban wireless traffic prediction. TIDES first identifies heterogeneous traffic patterns across regions through a clustering mechanism and trains personalized models for each region to balance generalization and specialization. To bridge the domain gap between numerical traffic data and
language-based models, we introduce a
prompt engineering scheme that embeds statistical traffic features as structured inputs. Furthermore, we design a DeepSeek module that enables spatial
alignment via cross-domain
attention, allowing the LLM to leverage information from spatially related regions. By fine-tuning only lightweight components while freezing core LLM layers, TIDES achieves efficient adaptation to domain-specific patterns without incurring excessive training overhead. Extensive experiments on real-world cellular traffic
datasets demonstrate that TIDES significantly outperforms state-of-the-art baselines in both prediction accuracy and robustness. Our results indicate that integrating spatial awareness into LLM-based predictors is the key to unlocking scalable and intelligent network management in future 6G systems.
Read the full abstract ▾
All authors Chuanting Zhang, Haixia Zhang, Jingping Qiao, Zongzhang Li, Mohamed-Slim Alouini
Machine Learning
Disentangling Fact from Sentiment: A Dynamic Conflict-Consensus Framework for Multimodal Fake News Detection
Weilin Zhou, Zonghao Ying, Rongchen Zhao et al. (+7 more)
Prevalent
multimodal fake news detection relies on consistency-based fusion, yet this paradigm fundamentally misinterprets critical cross-modal discrepancies as noise, leading to over-smoothing, which dilutes critical evidence of fabrication. Mainstream consistency-based fusion inherently minimizes feature discrepancies to align modalities, yet this approach fundamentally fails because it inadvertently smoothes out the subtle cross-modal contradictions that serve as the primary evidence of fabrication. To address this, we propose the Dynamic Conflict-Consensus Framework (DCCF), an inconsistency-seeking paradigm designed to amplify rather than suppress contradictions. First, DCCF decouples inputs into independent Fact and Sentiment spaces to distinguish objective mismatches from emotional dissonance. Second, we employ physics-inspired feature dynamics to iteratively polarize these representations, actively extracting maximally informative conflicts. Finally, a conflict-consensus mechanism standardizes these local discrepancies against the global context for robust deliberative judgment.Extensive experiments conducted on three real world
datasets demonstrate that DCCF consistently outperforms state-of-the-art baselines, achieving an average accuracy improvement of 3.52%.
Read the full abstract ▾
All authors Weilin Zhou, Zonghao Ying, Rongchen Zhao, Chunlei Meng, Quanchen Zou, Deyue Zhang, Enhao Gu, Mingze Liu, Dongdong Yang, Xiangzheng Zhang
Machine Learning
Improving Cardiac Risk Prediction Using Data Generation Techniques
Alexandre Cabodevila, Pedro Gamallo-Fernandez, Juan C. Vidal et al. (+1 more)
Cardiac rehabilitation constitutes a structured clinical process involving multiple interdependent phases, individualized medical decisions, and the coordinated participation of diverse healthcare professionals. This sequential and adaptive nature enables the program to be modeled as a business process, thereby facilitating its analysis. Nevertheless, studies in this context face significant limitations inherent to real-world medical databases: data are often scarce due to both economic costs and the time required for collection; many existing records are not suitable for specific analytical purposes; and, finally, there is a high prevalence of missing values, as not all patients undergo the same diagnostic tests. To address these limitations, this work proposes an architecture based on a Conditional Variational Autoencoder (CVAE) for the synthesis of realistic clinical records that are coherent with real-world observations. The primary objective is to increase the size and diversity of the available
datasets in order to enhance the performance of cardiac risk prediction models and to reduce the need for potentially hazardous diagnostic procedures, such as exercise stress testing. The results demonstrate that the proposed architecture is capable of generating coherent and realistic synthetic data, whose use improves the accuracy of the various classifiers employed for cardiac risk detection, outperforming state-of-the-art deep learning approaches for synthetic data generation.
Read the full abstract ▾
All authors Alexandre Cabodevila, Pedro Gamallo-Fernandez, Juan C. Vidal, Manuel Lama
Machine Learning
Forward Only Learning for Orthogonal Neural Networks of any Depth
Paul Caillon, Alex Colagrande, Erwan Fagnou et al. (+2 more)
Backpropagation is still the de facto
algorithm used today to train neural networks. With the exponential growth of recent architectures, the computational cost of this
algorithm also becomes a burden. The recent PEPITA and forward-only frameworks have proposed promising alternatives, but they failed to scale up to a handful of hidden layers, yet limiting their use. In this paper, we first analyze theoretically the main limitations of these approaches. It allows us the design of a forward-only
algorithm, which is equivalent to backpropagation under the linear and orthogonal assumptions. By relaxing the linear assumption, we then introduce FOTON (Forward-Only Training of Orthogonal Networks) that bridges the gap with the backpropagation
algorithm. Experimental results show that it outperforms PEPITA, enabling us to train neural networks of any depth, without the need for a backward pass. Moreover its performance on convolutional networks clearly opens up avenues for its application to more advanced architectures. The code is open-sourced at https://github.com/p0lcAi/FOTON .
Read the full abstract ▾
All authors Paul Caillon, Alex Colagrande, Erwan Fagnou, Blaise Delattre, Alexandre Allauzen
Machine Learning
Dominant vs. Dominated: Concept-Level Generative Collapse in Diffusion Models
Hayeon Jeong and Jong-Seok Lee
Text-to-image
diffusion models have attracted significant
attention for their ability to generate diverse, high-fidelity images. However, in multi-concept generation, one concept
token often dominates the output while others are suppressed-a phenomenon we term the Dominant-vs-Dominated (DvD) imbalance. To systematically study this failure mode, we introduce DominanceBench and examine its underlying causes from both data and internal-mechanistic perspectives. Our controlled fine-tuning study, which mimics concept learning during
diffusion-model training, shows that concepts learned from visually homogeneous (low-variation) concept-specific training images exhibit stronger dominance when composed with others. Cross-
attention analysis indicates that dominant
tokens concentrate
attention in early denoising steps, followed by reduced representation of competing concepts. Head-ablation analysis further shows that this dominance is distributed across
attention heads rather than localized. Overall, these findings characterize DvD as a systematic concept-level failure mode and provide a basis for more reliable and controllable multi-concept generation. DominanceBench will be released upon publication.
Read the full abstract ▾
Machine Learning · Cross-listed from cs.PL
Optimal Software Pipelining and Warp Specialization for Tensor Core GPUs
Rupanshu Soi, Rohan Yadav, Fredrik Kjolstad et al. (+4 more)
GPU architectures have continued to grow in complexity, with recent incarnations introducing increasingly powerful fixed-function units for
matrix multiplication and data movement to accompany highly parallel general-purpose cores. To fully leverage these machines, software must use sophisticated schedules that maximally utilize all hardware resources. Since realizing such schedules is complex, both programmers and compilers routinely employ program transformations, such as software pipelining (SWP) and warp specialization (WS), to do so in practice. However, determining how best to use SWP and WS in combination is a challenging problem that is currently handled through a mix of brittle compilation heuristics and fallible human intuition, with little insight into the space of solutions. To remedy this situation, we introduce a novel formulation of SWP and WS as a joint optimization problem that can be solved holistically by off-the-shelf constraint solvers. We reify our approach in Twill, the first system that automatically derives optimal SWP and WS schedules for a large class of iterative programs. Twill is heuristic-free, easily extensible to new GPU architectures, and guaranteed to produce optimal schedules. We show that Twill can rediscover, and thereby prove optimal, the SWP and WS schedules manually developed by experts for Flash
Attention on both the NVIDIA Hopper and Blackwell GPU architectures.
Read the full abstract ▾
All authors Rupanshu Soi, Rohan Yadav, Fredrik Kjolstad, Alex Aiken, Maryam Mehri Dehnavi, Michael Garland, Michael Bauer
Machine Learning
Grad: Guided Relation Diffusion Generation for Graph Augmentation in Graph Fraud Detection
Jie Yang, Rui Zhang, Ziyang Cheng et al. (+3 more)
Nowadays, Graph Fraud Detection (GFD) in financial scenarios has become an urgent research topic to protect online payment security. However, as organized crime groups are becoming more professional in real-world scenarios, fraudsters are employing more sophisticated camouflage strategies. Specifically, fraudsters disguise themselves by mimicking the behavioral data collected by platforms, ensuring that their key characteristics are consistent with those of benign users to a high degree, which we call Adaptive Camouflage. Consequently, this narrows the differences in behavioral traits between them and benign users within the platform's database, thereby making current GFD models lose efficiency. To address this problem, we propose a relation
diffusion-based graph augmentation model Grad. In detail, Grad leverages a supervised graph contrastive learning module to enhance the fraud-benign difference and employs a guided relation
diffusion generator to generate auxiliary homophilic relations from scratch. Based on these, weak fraudulent signals would be enhanced during the aggregation process, thus being obvious enough to be captured. Extensive experiments have been conducted on two real-world
datasets provided by WeChat Pay, one of the largest online payment platforms with billions of users, and three public
datasets. The results show that our proposed model Grad outperforms SOTA methods in both various scenarios, achieving at most 11.10% and 43.95% increases in AUC and AP, respectively. Our code is released at https://github.com/AI4Risk/antifraud and https://github.com/Muyiiiii/WWW25-Grad.
Read the full abstract ▾
All authors Jie Yang, Rui Zhang, Ziyang Cheng, Dawei Cheng, Guang Yang, Bo Wang
Machine Learning
TraCeR: Transformer-Based Competing Risk Analysis with Longitudinal Covariates
Maxmillan Ries and Sohan Seth
Survival analysis is a critical tool for modeling time-to-event data. Recent deep learning-based models have reduced various modeling assumptions including proportional hazard and linearity. However, a persistent challenge remains in incorporating longitudinal
covariates, with prior work largely focusing on cross-sectional features, and in assessing calibration of these models, with research primarily focusing on discrimination during evaluation. We introduce TraCeR, a
transformer-based survival analysis framework for incorporating longitudinal
covariates. Based on a factorized self-
attention architecture, TraCeR estimates the hazard function from a sequence of measurements, naturally capturing temporal
covariate interactions without assumptions about the underlying data-generating process. The framework is inherently designed to handle censored data and competing events. Experiments on multiple real-world
datasets demonstrate that TraCeR achieves substantial and statistically significant performance improvements over state-of-the-art methods. Furthermore, our evaluation extends beyond discrimination metrics and assesses model calibration, addressing a key oversight in literature.
Read the full abstract ▾
Machine Learning · Cross-listed from quant-ph
Exploring polymer classification with a hybrid single-photon quantum approach
Alexandrina Stoyanova and Bogdan Penkovsky
Polymers exhibit complex architectures and diverse properties that place them at the center of contemporary research in chemistry and materials science. As conventional computational techniques, even multi-scale ones, struggle to capture this complexity,
quantum computing offers a promising alternative framework for extracting structure-property relationships. Noisy Intermediate-Scale
Quantum (NISQ) devices are commonly used to explore the implementation of
algorithms, including
quantum neural networks for classification tasks, despite ongoing debate regarding their practical impact. We present a hybrid classical-
quantum formalism that couples a classical deep neural network for
polymer featurization with a single-
photon-based
quantum classifier native to photonic
quantum computing. This pipeline successfully classifies
polymer species by their optical gap, with performance in line between CPU-based noisy simulations and a proof-of-principle run on Quandela's Ascella
quantum processor. These findings demonstrate the effectiveness of the proposed computational workflow and indicate that chemistryfrelated classification tasks can already be tackled under the constraints of today's NISQ devices.
Read the full abstract ▾
Machine Learning
Learning Generalizable Neural Operators for Inverse Problems
Adam J. Thorpe, Stepan Tretiakov, Dibakar Roy Sarkar et al. (+2 more)
Inverse problems challenge existing neural operator architectures because ill-posed inverse maps violate continuity, uniqueness, and stability assumptions. We introduce B2B${}^{-1}$, an inverse basis-to-basis neural operator framework that addresses this limitation. Our key innovation is to decouple function representation from the inverse map. We learn neural basis functions for the input and output spaces, then train inverse models that operate on the resulting coefficient space. This structure allows us to learn deterministic, invertible, and probabilistic models within a single framework, and to choose models based on the degree of ill-posedness. We evaluate our approach on six inverse PDE
benchmarks, including two novel
datasets, and compare against existing invertible neural operator baselines. We learn probabilistic models that capture uncertainty and input variability, and remain robust to measurement noise due to implicit denoising in the coefficient calculation. Our results show consistent re-simulation performance across varying levels of ill-posedness. By separating representation from inversion, our framework enables scalable surrogate models for inverse problems that generalize across instances, domains, and degrees of ill-posedness.
Read the full abstract ▾
All authors Adam J. Thorpe, Stepan Tretiakov, Dibakar Roy Sarkar, Krishna Kumar, Ufuk Topcu
Machine Learning
Microstructure-based Variational Neural Networks for Robust Uncertainty Quantification in Materials Digital Twins
Andreas E. Robertson, Samuel B. Inman, Ashley T. Lenau et al. (+4 more)
Aleatoric uncertainties - irremovable variability in microstructure morphology, constituent behavior, and processing conditions - pose a major challenge to developing uncertainty-robust digital twins. We introduce the Variational Deep Material Network (VDMN), a physics-informed surrogate model that enables efficient and probabilistic forward and inverse predictions of material behavior. The VDMN captures microstructure-induced variability by
embedding variational distributions within its hierarchical, mechanistic architecture. Using an analytic propagation scheme based on Taylor-series expansion and automatic differentiation, the VDMN efficiently propagates uncertainty through the network during training and prediction. We demonstrate its capabilities in two digital-twin-driven applications: (1) as an uncertainty-aware materials digital twin, it predicts and experimentally validates the nonlinear mechanical variability in additively manufactured
polymer composites; and (2) as an inverse calibration engine, it disentangles and quantitatively identifies overlapping sources of uncertainty in constituent properties. Together, these results establish the VDMN as a foundation for uncertainty-robust materials digital twins.
Read the full abstract ▾
All authors Andreas E. Robertson, Samuel B. Inman, Ashley T. Lenau, Ricardo A. Lebensohn, Dongil Shin, Brad L. Boyce, Remi M. Dingreville
Machine Learning · Cross-listed from cs.SE
From Coverage to Causes: Data-Centric Fuzzing for JavaScript Engines
Kishan Kumar Ganguly and Tim Menzies
Context: Exhaustive fuzzing of modern JavaScript engines is infeasible due to the vast number of program states and execution paths. Coverage-guided fuzzers waste effort on low-risk inputs, often ignoring vulnerability-triggering ones that do not increase coverage. Existing heuristics proposed to mitigate this require expert effort, are brittle, and hard to adapt. Objective: We propose a data-centric, LLM-boosted alternative that learns from historical vulnerabilities to automatically identify minimal static (code) and dynamic (runtime) features for detecting high-risk inputs. Method: Guided by historical V8 bugs, iterative prompting generated 115 static and 49 dynamic features, with the latter requiring only five trace flags, minimizing instrumentation cost. After feature selection, 41 features remained to train an XGBoost model to predict high-risk inputs during fuzzing. Results: Combining static and dynamic features yields over 85% precision and under 1% false alarms. Only 25% of these features are needed for comparable performance, showing that most of the search space is irrelevant. Conclusion: This work introduces feature-guided fuzzing, an automated data-driven approach that replaces coverage with data-directed inference, guiding fuzzers toward high-risk states for faster, targeted, and reproducible vulnerability discovery. To support open science, all scripts and data are available at https://github.com/KKGanguly/DataCentricFuzzJS .
Read the full abstract ▾
Machine Learning · Cross-listed from cs.AI
Faithful and Stable Neuron Explanations for Trustworthy Mechanistic Interpretability
Ge Yan, Tuomas Oikarinen and Tsui-Wei Weng
Neuron identification is a popular tool in mechanistic interpretability, aiming to uncover the human-interpretable concepts represented by individual
neurons in deep networks. While
algorithms such as Network Dissection and CLIP-Dissect achieve great empirical success, a rigorous theoretical foundation remains absent, which is crucial to enable trustworthy and reliable explanations. In this work, we observe that
neuron identification can be viewed as the inverse process of machine learning, which allows us to derive guarantees for
neuron explanations. Based on this insight, we present the first theoretical analysis of two fundamental challenges: (1) Faithfulness: whether the identified concept faithfully represents the
neuron's underlying function and (2) Stability: whether the identification results are consistent across probing
datasets. We derive generalization bounds for widely used similarity metrics (e.g. accuracy, AUROC, IoU) to guarantee faithfulness, and propose a bootstrap ensemble procedure that quantifies stability along with BE (Bootstrap Explanation) method to generate concept prediction sets with guaranteed coverage probability. Experiments on both synthetic and real data validate our theoretical results and demonstrate the practicality of our method, providing an important step toward trustworthy
neuron identification.
Read the full abstract ▾
Machine Learning · Cross-listed from stat.ML
Causal Inference as Distribution Adaptation: Optimizing ATE Risk under Propensity Uncertainty
Ashley Zhang
Standard approaches to causal inference, such as Outcome Regression and Inverse Probability Weighted Regression Adjustment (IPWRA), are typically derived through the lens of missing data imputation and identification theory. In this work, we unify these methods from a Machine Learning perspective, reframing ATE estimation as a domain adaptation problem under distribution shift. We demonstrate that the canonical Hajek
estimator is a special case of IPWRA restricted to a constant hypothesis class, and that IPWRA itself is fundamentally Importance-Weighted Empirical Risk Minimization designed to correct for the
covariate shift between the treated sub-population and the target population. Leveraging this unified framework, we critically examine the optimization objectives of Doubly Robust
estimators. We argue that standard methods enforce sufficient but not necessary conditions for consistency by requiring outcome models to be individually unbiased. We define the true "ATE Risk Function" and show that minimizing it requires only that the biases of the treated and control models structurally cancel out. Exploiting this insight, we propose the Joint Robust
Estimator (JRE). Instead of treating propensity estimation and outcome modeling as independent stages, JRE utilizes bootstrap-based uncertainty quantification of the propensity score to train outcome models jointly. By optimizing for the expected ATE risk over the distribution of propensity scores, JRE leverages model degrees of freedom to achieve robustness against propensity misspecification. Simulation studies demonstrate that JRE achieves up to a 15% reduction in MSE compared to standard IPWRA in finite-sample regimes with misspecified outcome models.
Read the full abstract ▾
Machine Learning · Cross-listed from cs.DS
Graph-based Nearest Neighbors with Dynamic Updates via Random Walks
Nina Mishra, Yonatan Naamad, Tal Wagner et al. (+1 more)
Approximate nearest neighbor search (ANN) is a common way to retrieve relevant search results, especially now in the context of large
language models and
retrieval augmented generation. One of the most widely used
algorithms for ANN is based on constructing a multi-layer graph over the
dataset, called the Hierarchical Navigable Small World (HNSW). While this
algorithm supports insertion of new data, it does not support deletion of existing data. Moreover, deletion
algorithms described by prior work come at the cost of increased query latency, decreased recall, or prolonged deletion time. In this paper, we propose a new theoretical framework for graph-based ANN based on random walks. We then utilize this framework to analyze a randomized deletion approach that preserves hitting time statistics compared to the graph before deleting the point. We then turn this theoretical framework into a deterministic deletion
algorithm, and show that it provides better tradeoff between query latency, recall, deletion time, and
memory usage through an extensive collection of experiments.
Read the full abstract ▾
All authors Nina Mishra, Yonatan Naamad, Tal Wagner, Lichen Zhang
Machine Learning · Cross-listed from math.NA
Approximation and learning with compositional tensor trains
Martin Eigel, Charles Miranda, Anthony Nouy et al. (+1 more)
We introduce compositional tensor trains (CTTs) for the approximation of multivariate functions, a class of models obtained by composing low-rank functions in the tensor-train format. This format can encode standard approximation tools, such as (sparse)
polynomials, deep neural networks (DNNs) with fixed width, or tensor networks with arbitrary permutation of the inputs, or more general affine coordinate transformations, with similar complexities. This format can be viewed as a DNN with width exponential in the input dimension and structured weights
matrices. Compared to DNNs, this format enables controlled compression at the layer level using efficient tensor
algebra. On the optimization side, we derive a layerwise
algorithm inspired by natural gradient descent, allowing to exploit efficient low-rank tensor
algebra. This relies on low-rank estimations of Gram
matrices, and tensor structured random sketching. Viewing the format as a discrete dynamical system, we also derive an optimization
algorithm inspired by numerical methods in optimal control. Numerical experiments on regression tasks demonstrate the expressivity of the new format and the relevance of the proposed optimization
algorithms. Overall, CTTs combine the expressivity of compositional models with the algorithmic efficiency of tensor
algebra, offering a scalable alternative to standard deep neural networks.
Read the full abstract ▾
All authors Martin Eigel, Charles Miranda, Anthony Nouy, David Sommer
Machine Learning · Cross-listed from cs.CV
FOODER: Real-time Facial Authentication and Expression Recognition
Sabri Mustafa Kahya, Muhammet Sami Yavuz, Boran Hamdi Sivrikaya et al. (+1 more)
Out-of-distribution (OOD) detection is essential for the safe deployment of neural networks, as it enables the identification of samples outside the training domain. We present FOODER, a real-time,
privacy-preserving radar-based framework that integrates OOD-based facial authentication with facial expression recognition. FOODER operates using low-cost frequency-modulated continuous-wave (FMCW) radar and exploits both range-Doppler and micro range-Doppler representations. The authentication module employs a multi-encoder multi-decoder architecture with Body Part (BP) and Intermediate Linear Encoder-Decoder (ILED) components to classify a single enrolled individual as in-distribution while detecting all other faces as OOD. Upon successful authentication, an expression recognition module is activated. Concatenated radar representations are processed by a ResNet block to distinguish between dynamic and static facial expressions. Based on this categorization, two specialized MobileViT networks are used to classify dynamic expressions (smile, shock) and static expressions (neutral, anger). This hierarchical design enables robust facial authentication and fine-grained expression recognition while preserving user
privacy by relying exclusively on radar data. Experiments conducted on a
dataset collected with a 60 GHz short-range FMCW radar demonstrate that FOODER achieves an AUROC of 94.13% and an FPR95 of 18.12% for authentication, along with an average expression recognition accuracy of 94.70%. FOODER outperforms state-of-the-art OOD detection methods and several
transformer-based architectures while operating efficiently in real time.
Read the full abstract ▾
All authors Sabri Mustafa Kahya, Muhammet Sami Yavuz, Boran Hamdi Sivrikaya, Eckehard Steinbach
Machine Learning
Probabilistic Digital Twins of Users: Latent Representation Learning with Statistically Validated Semantics
Daniel David
Understanding user identity and behavior is central to applications such as personalization, recommendation, and decision support. Most existing approaches rely on deterministic
embeddings or black-box predictive models, offering limited uncertainty quantification and little insight into what latent representations encode. We propose a probabilistic digital twin framework in which each user is modeled as a latent
stochastic state that generates observed behavioral data. The digital twin is learned via amortized variational inference, enabling scalable posterior estimation while retaining a fully probabilistic interpretation. We instantiate this framework using a variational autoencoder (VAE) applied to a user-response
dataset designed to capture stable aspects of user identity. Beyond standard reconstruction-based evaluation, we introduce a statistically grounded interpretation pipeline that links latent dimensions to observable behavioral patterns. By analyzing users at the extremes of each latent dimension and validating differences using nonparametric hypothesis tests and effect sizes, we demonstrate that specific dimensions correspond to interpretable traits such as opinion strength and decisiveness. Empirically, we find that user structure is predominantly continuous rather than discretely clustered, with weak but meaningful structure emerging along a small number of dominant latent axes. These results suggest that probabilistic digital twins can provide interpretable, uncertainty-aware representations that go beyond deterministic user
embeddings.
Read the full abstract ▾