Machine Learning
Unsupervised learning of multiscale switching dynamical system models from multimodal neural data
DongKyu Kim, Han-Lin Hsieh and Maryam M. Shanechi
Neural population activity often exhibits regime-dependent non-stationarity in the form of switching dynamics. Learning accurate switching dynamical system models can reveal how behavior is encoded in neural activity. Existing switching approaches have primarily focused on learning models from a single neural modality, either continuous Gaussian signals or discrete Poisson signals. However, multiple neural modalities are often recorded simultaneously to measure different spatiotemporal scales of
brain activity, and all these modalities can encode behavior. Moreover, regime labels are typically unavailable in training data, posing a significant challenge for learning models of regime-dependent switching dynamics. To address these challenges, we develop a novel unsupervised learning
algorithm that learns the parameters of switching multiscale dynamical system models using only multiscale neural observations. We demonstrate our method using both simulations and two distinct experimental
datasets with
multimodal spike-LFP observations during different motor tasks. We find that our switching multiscale dynamical system models more accurately decode behavior than switching single-scale dynamical models, showing the success of multiscale neural fusion. Further, our models outperform stationary multiscale models, illustrating the importance of tracking regime-dependent non-stationarity in
multimodal neural data. The developed unsupervised learning framework enables more accurate modeling of complex multiscale neural dynamics by leveraging information in
multimodal recordings while incorporating regime switches. This approach holds promise for improving the performance and robustness of
brain-computer interfaces over time and for advancing our understanding of the neural basis of behavior.
Read the full abstract ▾
Machine Learning
Improving Recursive Transformers with Mixture of LoRAs
Mohammadmahdi Nouriborji, Morteza Rohanian and Omid Rohanian
Parameter sharing in recursive
transformers reduces model size but collapses layer-wise expressivity. We propose Mixture of LoRAs (MoL), a lightweight conditional-computation mechanism that inserts Low-Rank Adaptation (LoRA) experts inside a shared feed-forward network (FFN). MoL enables
token-conditional weight-space modulation of the shared FFN without untying backbone parameters, unlike prior approaches that add fixed or externally attached adapters. We pretrain a modernised recursive architecture, ModernALBERT, integrating rotary
embeddings, GeGLU, FlashAttention, and a
distillation-based initialisation. Across GLUE, SQuAD-v2, and BEIR, ModernALBERT (50M–120M) achieves state-of-the-art performance among compact models and surpasses larger fully parameterised baselines. We also propose an expert-merging procedure that compresses MoL into a single adapter at inference while preserving accuracy, enabling efficient deployment. Our results show that conditional weight-space modulation effectively restores the expressivity lost under aggressive parameter sharing in recursive
transformers.
Read the full abstract ▾
Machine Learning
Active Learning with Imperfect Labels: Optimal Labeler Assignment and Sample Selection
Pouya Ahadi, Blair Winograd, Camille Zaug et al. (+3 more)
Active Learning (AL) is commonly used in applications where labeling data is expensive or time-consuming. In practice, however, labels are often noisy due to varying labeler expertise and annotation uncertainty, especially for complex or ambiguous samples. Learning from such imperfectly labeled data can degrade classifier performance. We propose an AL framework that explicitly accounts for label noise by optimally assigning labelers and selecting samples to minimize labeling error. Our approach, called OLAS (Optimal Labeler Assignment and Sampling), uses a noise model that depends on both labeler accuracy and model uncertainty to guide these decisions. We develop two tractable optimization formulations: one for assigning samples to labelers to minimize worst-case noise, and another for selecting samples while controlling overall label noise. Theoretical results provide closed-form solutions under mild conditions. Empirical evaluations on
benchmark datasets and a real-world warranty claim classification problem show that OLAS achieves the highest or near-highest classification accuracy among existing AL strategies across most settings, using only a single label per sample.
Read the full abstract ▾
All authors Pouya Ahadi, Blair Winograd, Camille Zaug, Karunesh Arora, Lijun Wang, Kamran Paynabar
Machine Learning · Cross-listed from cs.CL
Counting Clues: A Lightweight Probabilistic Baseline Can Match an LLM
Furong Jia, Yuan Pu, Finn Guo et al. (+1 more)
Large
language models (LLMs) excel on multiple-choice clinical diagnosis
benchmarks, yet it is unclear how much of this performance reflects underlying probabilistic
reasoning. We study this through questions from MedQA that ask for the most likely diagnosis. We introduce Frequency-Based Probabilistic Ranker (FBPR), a lightweight method that scores options with a smoothed Naive Bayes over concept-diagnosis co-occurrence statistics from a large corpus. When co-occurrence statistics are sourced from Dolma, the pretraining corpus for OLMo, and RedPajama, an open reproduction of the LLaMA pretraining-data recipe, FBPR achieves comparable performance to the corresponding LLMs. Furthermore, direct LLM inference and FBPR exhibit complementary strengths, correctly answering largely different subsets of questions with overlap barely above chance. Combining them with a simple hybrid method yields a notable gain over either method alone. These findings show that explicit probabilistic baselines can account for a substantial portion of the
benchmark performance on diagnosis questions, while serving as historically grounded reference points and complementary signals for hybridization.
Read the full abstract ▾
All authors Furong Jia, Yuan Pu, Finn Guo, Monica Agrawal
Machine Learning
Information-Consistent Language Model Recommendations through Group Relative Policy Optimization
Sonal Prabhune, Balaji Padmanabhan and Kaushik Dutta
Large
Language Models (LLMs) are increasingly deployed in business-critical domains such as finance, education, healthcare, and customer support, where users expect consistent and reliable recommendations. Yet LLMs often exhibit variability when
prompts are phrased with minor differences, even when semantically equivalent. Such inconsistency undermines trust, complicates compliance, and disrupts user experience. While personalization is desirable in certain contexts, many enterprise scenarios, such as HR onboarding, customer support, or policy disclosure, require invariant information delivery regardless of phrasing or prior conversational history. Existing approaches, including
retrieval-augmented generation (RAG) and temperature tuning, improve factuality or reduce stochasticity, but cannot guarantee stability across equivalent
prompts. In this paper, we propose a reinforcement learning framework based on Group Relative Policy Optimization (GRPO) to directly optimize for consistency. Unlike prior applications of GRPO, which have been limited to
reasoning and code generation, we adapt GRPO to enforce the stability of information content across groups of semantically equivalent
prompts. We introduce
entropy-based helpfulness and stability rewards, treating
prompt variants as groups and resetting conversational context to isolate phrasing effects. Experiments on investment and job recommendation tasks show that our GRPO-fine-tuned model reduces variability compared to the baseline LLM model. To our knowledge, this is a novel application of GRPO for aligning LLMs toward information consistency, reframing variability not as an acceptable feature of generative diversity, but as a correctable flaw in enterprise deployments.
Read the full abstract ▾
Machine Learning · Cross-listed from cs.AI
Forgetful but Faithful: A Cognitive Memory Architecture and Benchmark for Privacy-Aware Generative Agents
Saad Alqithami
As generative
agents become increasingly sophisticated and deployed in long-term interactive scenarios, their
memory management capabilities emerge as a critical bottleneck for both performance and
privacy. Current approaches either maintain unlimited
memory stores, leading to computational intractability and
privacy concerns, or employ simplistic forgetting mechanisms that compromise
agent coherence and functionality. This paper introduces the
Memory-Aware Retention Schema (MaRS), a novel framework for human-centered
memory management in generative
agents, coupled with six theoretically-grounded forgetting policies that balance performance,
privacy, and computational efficiency. We present the Forgetful but Faithful
Agent (FiFA)
benchmark, a comprehensive evaluation framework that assesses
agent performance across narrative coherence, goal completion, social recall accuracy,
privacy preservation, and cost efficiency. Through extensive experimentation involving 300 evaluation runs across multiple
memory budgets and
agent configurations, we demonstrate that our hybrid forgetting policy achieves superior performance (composite score: 0.911) while maintaining computational tractability and
privacy guarantees. Our work establishes new
benchmarks for
memory-budgeted
agent evaluation and provides practical guidelines for deploying generative
agents in resource-constrained,
privacy-sensitive environments. The theoretical foundations, implementation framework, and empirical results contribute to the emerging field of human-centered AI by addressing fundamental challenges in
agent memory management that directly impact user trust, system scalability, and regulatory compliance.
Read the full abstract ▾
Machine Learning · Cross-listed from cs.AR
KANELÉ: Kolmogorov-Arnold Networks for Efficient LUT-based Evaluation
Duc Hoang, Aarush Gupta and Philip Harris
Low-latency, resource-efficient neural network inference on FPGAs is essential for applications demanding real-time capability and low power. Lookup table (LUT)-based neural networks are a common solution, combining strong representational power with efficient FPGA implementation. In this work, we introduce KANELÉ, a framework that exploits the unique properties of Kolmogorov-Arnold Networks (KANs) for FPGA deployment. Unlike traditional multilayer perceptrons (MLPs), KANs employ learnable one-dimensional splines with fixed domains as edge activations, a structure naturally suited to discretization and efficient LUT mapping. We present the first systematic design flow for implementing KANs on FPGAs, co-optimizing training with quantization and
pruning to enable compact, high-throughput, and low-latency KAN architectures. Our results demonstrate up to a 2700x speedup and orders of magnitude resource savings compared to prior KAN-on-FPGA approaches. Moreover, KANELÉ matches or surpasses other LUT-based architectures on widely used
benchmarks, particularly for tasks involving symbolic or physical formulas, while balancing resource usage across FPGA hardware. Finally, we showcase the versatility of the framework by extending it to real-time, power-efficient control systems.
Read the full abstract ▾
Machine Learning · Cross-listed from cs.AR
HaShiFlex: A High-Throughput Hardened Shifter DNN Accelerator with Fine-Tuning Flexibility
Jonathan Herbst, Michael Pellauer and Sherief Reda
We introduce a high-throughput neural network accelerator that embeds most network layers directly in hardware, minimizing data transfer and
memory usage while preserving a degree of flexibility via a small neural processing unit for the final classification layer. By leveraging power-of-two (Po2) quantization for weights, we replace multiplications with simple rewiring, effectively reducing each convolution to a series of additions. This streamlined approach offers high-throughput, energy-efficient processing, making it highly suitable for applications where model parameters remain stable, such as continuous sensing tasks at the edge or large-scale data center deployments. Furthermore, by including a strategically chosen reprogrammable final layer, our design achieves high throughput without sacrificing fine-tuning capabilities. We implement this accelerator in a 7nm ASIC flow using MobileNetV2 as a baseline and report throughput, area, accuracy, and sensitivity to quantization and
pruning - demonstrating both the advantages and potential trade-offs of the proposed architecture. We find that for MobileNetV2, we can improve inference throughput by 20x over fully programmable GPUs, processing 1.21 million images per second through a full forward pass while retaining fine-tuning flexibility. If absolutely no post-deployment fine tuning is required, this advantage increases to 67x at 4 million images per second.
Read the full abstract ▾
Machine Learning
Selective Conformal Risk Control
Yunpeng Xu, Wenge Guo and Zhi Wei
Reliable uncertainty quantification is essential for deploying machine learning systems in high-stakes domains. Conformal prediction provides distribution-free coverage guarantees but often produces overly large prediction sets, limiting its practical utility. To address this issue, we propose Selective Conformal Risk Control (SCRC), a unified framework that integrates conformal prediction with selective classification. The framework formulates uncertainty control as a two-stage problem: the first stage selects confident samples for prediction, and the second stage applies conformal risk control on the selected subset to construct calibrated prediction sets. We develop two
algorithms under this framework. The first, SCRC-T, preserves exchangeability by computing thresholds jointly over calibration and test samples, offering exact finite-sample guarantees. The second, SCRC-I, is a calibration-only variant that provides PAC-style probabilistic guarantees while being more computational efficient. Experiments on two public
datasets show that both methods achieve the target coverage and risk levels, with nearly identical performance, while SCRC-I exhibits slightly more conservative risk control but superior computational practicality. Our results demonstrate that selective conformal risk control offers an effective and efficient path toward compact, reliable uncertainty quantification.
Read the full abstract ▾
Machine Learning · Cross-listed from cs.RO
SAGA: Open-World Mobile Manipulation via Structured Affordance Grounding
Kuan Fang, Yuxin Chen, Xinghao Zhu et al. (+3 more)
We present SAGA, a versatile and adaptive framework for visuomotor control that can generalize across various environments, task objectives, and user specifications. To efficiently learn such capability, our key idea is to disentangle high-level semantic intent from low-level visuomotor control by explicitly grounding task objectives in the observed environment. Using an affordance-based task representation, we express diverse and complex behaviors in a unified, structured form. By leveraging
multimodal foundation models, SAGA grounds the proposed task representation to the
robot's visual observation as 3D affordance heatmaps, highlighting task-relevant entities while abstracting away spurious appearance variations that would hinder generalization. These grounded affordances enable us to effectively train a conditional policy on multi-task demonstration data for whole-body control. In a unified framework, SAGA can solve tasks specified in different forms, including
language instructions, selected points, and example demonstrations, enabling both zero-shot execution and few-shot adaptation. We instantiate SAGA on a quadrupedal manipulator and conduct extensive experiments across eleven real-world tasks. SAGA consistently outperforms end-to-end and modular baselines by substantial margins. Together, these results demonstrate that structured affordance grounding offers a scalable and effective pathway toward generalist mobile manipulation.
Read the full abstract ▾
All authors Kuan Fang, Yuxin Chen, Xinghao Zhu, Farzad Niroui, Lingfeng Sun, Jiuguang Wang
Machine Learning
PRIVEE: Privacy-Preserving Vertical Federated Learning Against Feature Inference Attacks
Sindhuja Madabushi, Haider Ali, Ahmad Faraz Khan et al. (+5 more)
Vertical
Federated Learning (VFL) enables collaborative model training across organizations that share common user samples but hold disjoint feature spaces. Despite its potential, VFL is susceptible to feature inference attacks, in which
adversarial parties exploit shared confidence scores (prediction probabilities) during inference to reconstruct private input features of other participants. To counter this threat, we propose PRIVEE (
PRIvacy-preserving Vertical
fEderated lEarning), a novel defense mechanism named after the French word privée, meaning "private." PRIVEE obfuscates confidence scores while preserving critical properties such as relative ranking and inter-score distances. Rather than exposing raw scores, PRIVEE only shares transformed representations, mitigating risk of reconstruction attacks without degrading model prediction accuracy. Extensive experiments show that PRIVEE achieves up to a 30 times increase in reconstruction error (MSE) against feature inference attacks, compared to the strongest competing defense, while preserving full predictive performance against advanced feature inference attacks.
Read the full abstract ▾
All authors Sindhuja Madabushi, Haider Ali, Ahmad Faraz Khan, Rui Ning, Hongyi Wu, Chunsheng Xin, Ali. R. Butt, Jin-Hee Cho
Machine Learning
Network Level Evaluation of Hangup Susceptibility of HRGCs using Deep Learning and Sensing Techniques: A Goal Towards Safer Future
Kaustav Chatterjee, Joshua Li, Kundan Parajulee et al. (+1 more)
Steep-profiled Highway Railway Grade Crossings (HRGCs) pose safety hazards to vehicles with low ground clearance, which may become stranded on the tracks, creating risks of train vehicle collisions. This research develops a framework for network level evaluation of hang-up susceptibility of HRGCs. Profile data from different crossings in Oklahoma were collected using both a walking profiler and the Pave3D8K
Laser Imaging System. A hybrid deep learning model, combining Long Short Term
Memory (LSTM) and
Transformer architectures, was developed to reconstruct accurate HRGC profiles from Pave3D8K
Laser Imaging System data. Vehicle dimension data from around 350 specialty vehicles were collected at various locations across Oklahoma to enable up-to-date statistical design dimensions. Hang-up susceptibility was analyzed using three vehicle dimension scenarios: (a) median dimension (median wheelbase and ground clearance), (b) 75-25 percentile dimension (75 percentile wheelbase, 25 percentile ground clearance), and (c) worst case dimension (maximum wheelbase and minimum ground clearance). Results indicate 70, 80, and 95 crossings at the highest hang-up risk levels under these scenarios, respectively. An ArcGIS database and a software interface were developed to support transportation agencies in mitigating crossing hazards. This framework advances safety evaluation by integrating next-generation sensing, deep learning, and infrastructure
datasets into practical decision support tools.
Read the full abstract ▾
All authors Kaustav Chatterjee, Joshua Li, Kundan Parajulee, Jared Schwennesen
Machine Learning
GradID: Adversarial Detection via Intrinsic Dimensionality of Gradients
Mohammad Mahdi Razmjoo, Mohammad Mahdi Sharifian and Saeed Bagheri Shouraki
Despite their remarkable performance, deep neural networks exhibit a critical vulnerability: small, often imperceptible,
adversarial perturbations can lead to drastically altered model predictions. Given the stringent reliability demands of applications such as medical diagnosis and autonomous driving, robust detection of such
adversarial attacks is paramount. In this paper, we investigate the
geometric properties of a model's input loss landscape. We analyze the Intrinsic Dimensionality (ID) of the model's gradient parameters, which quantifies the minimal number of coordinates required to describe the data points on their underlying
manifold. We reveal a distinct and consistent difference in the ID for natural and
adversarial data, which forms the basis of our proposed detection method. We validate our approach across two distinct operational scenarios. First, in a batch-wise context for identifying malicious data groups, our method demonstrates high efficacy on
datasets like MNIST and SVHN. Second, in the critical individual-sample setting, we establish new state-of-the-art results on challenging
benchmarks such as CIFAR-10 and MS COCO. Our detector significantly surpasses existing methods against a wide array of attacks, including CW and AutoAttack, achieving detection rates consistently above 92% on CIFAR-10. The results underscore the robustness of our
geometric approach, highlighting that intrinsic dimensionality is a powerful fingerprint for
adversarial detection across diverse
datasets and attack strategies.
Read the full abstract ▾
Machine Learning
On the continuity of flows
Congzhou M Sha
Flow matching has emerged as a powerful framework for generative modeling through continuous normalizing flows. We investigate a potential topological constraint: when the prior distribution and target distribution have mismatched
topology (e.g., unimodal to
multimodal), the optimal velocity field under standard flow matching objectives may exhibit spatial discontinuities. We suggest that this discontinuity arises from the requirement that continuous flows must bifurcate to map a single mode to multiple modes, forcing particles to make discrete routing decisions at intermediate times. Through theoretical analysis on bimodal Gaussian mixtures, we demonstrate that the optimal velocity field exhibits jump discontinuities along decision boundaries, with magnitude approaching infinity as time approaches the target distribution. Our analysis suggests that this phenomenon is not specific to $L^2$ loss, but rather may be a consequence of topological mismatch between distributions. We validate our theory empirically and discuss potential implications for flow matching on
manifolds, connecting our findings to recent work on Riemannian flow matching and the challenge of learning discontinuous representations in neural networks.
Read the full abstract ▾
Machine Learning · Cross-listed from cs.CL
Hindsight is 20/20: Building Agent Memory that Retains, Recalls, and Reflects
Chris Latimer, Nicoló Boschi, Andrew Neeser et al. (+4 more)
Agent memory has been touted as a dimension of growth for LLM-based applications, enabling
agents that can accumulate experience, adapt across sessions, and move beyond single-shot question answering. The current generation of
agent memory systems treats
memory as an external layer that extracts salient snippets from conversations, stores them in vector or graph-based stores, and retrieves top-k items into the
prompt of an otherwise stateless model. While these systems improve personalization and context carry-over, they still blur the line between evidence and inference, struggle to organize information over long horizons, and offer limited support for
agents that must explain their
reasoning. We present Hindsight, a
memory architecture that treats
agent memory as a structured, first-class substrate for
reasoning by organizing it into four logical networks that distinguish world facts,
agent experiences, synthesized entity summaries, and evolving beliefs. This framework supports three core operations – retain, recall, and reflect – that govern how information is added, accessed, and updated. Under this abstraction, a temporal, entity aware
memory layer incrementally turns conversational streams into a structured, queryable
memory bank, while a reflection layer reasons over this bank to produce answers and to update information in a traceable way. On key long-horizon conversational
memory benchmarks like LongMemEval and LoCoMo, Hindsight with an open-source 20B model lifts overall accuracy from 39% to 83.6% over a full-context baseline with the same backbone and outperforms full context GPT-4o. Scaling the backbone further pushes Hindsight to 91.4% on LongMemEval and up to 89.61% on LoCoMo (vs. 75.78% for the strongest prior open system), consistently outperforming existing
memory architectures on multi-session and open-domain questions.
Read the full abstract ▾
All authors Chris Latimer, Nicoló Boschi, Andrew Neeser, Chris Bartholomew, Gaurav Srivastava, Xuan Wang, Naren Ramakrishnan
Machine Learning
Optimal Resource Allocation for ML Model Training and Deployment under Concept Drift
Hasan Burhan Beytur, Haris Vikalo, Kevin S Chan et al. (+1 more)
We study how to allocate resources for training and deployment of machine learning (ML) models under concept drift and limited budgets. We consider a setting in which a model provider distributes trained models to multiple clients whose devices support local inference but lack the ability to retrain those models, placing the burden of performance maintenance on the provider. We introduce a model-agnostic framework that captures the interaction between resource allocation, concept drift dynamics, and deployment timing. We show that optimal training policies depend critically on the aging properties of concept durations. Under sudden concept changes, we derive optimal training policies subject to budget constraints when concept durations follow distributions with Decreasing Mean Residual Life (DMRL), and show that intuitive heuristics are provably suboptimal under Increasing Mean Residual Life (IMRL). We further study model deployment under communication constraints, prove that the associated optimization problem is quasi-convex under mild conditions, and propose a randomized scheduling strategy that achieves near-optimal client-side performance. These results offer theoretical and algorithmic foundations for cost-efficient ML model management under concept drift, with implications for continual learning, distributed inference, and adaptive ML systems.
Read the full abstract ▾
All authors Hasan Burhan Beytur, Haris Vikalo, Kevin S Chan, Gustavo de Veciana
Machine Learning
From Small to Large: Generalization Bounds for Transformers on Variable-Size Inputs
Anastasiia Alokhina and Pan Li
Transformers exhibit a notable property of size generalization, demonstrating an ability to extrapolate from smaller
token sets to significantly longer ones. This behavior has been documented across diverse applications, including point clouds, graphs, and natural
language. Despite its empirical success, this capability still lacks some rigorous theoretical characterizations. In this paper, we develop a theoretical framework to analyze this phenomenon for
geometric data, which we represent as discrete samples from a continuous source (e.g., point clouds from
manifolds, graphs from graphons). Our core contribution is a bound on the error between the
Transformer's output for a discrete sample and its continuous-domain equivalent. We prove that for
Transformers with stable positional encodings, this bound is determined by the sampling density and the intrinsic dimensionality of the data
manifold. Experiments on graphs and point clouds of various sizes confirm the tightness of our theoretical bound.
Read the full abstract ▾
Machine Learning
TRACER: Transfer Learning based Real-time Adaptation for Clinical Evolving Risk
Mengying Yan, Ziye Tian, Siqi Li et al. (+4 more)
Clinical decision support tools built on electronic health records often experience performance drift due to temporal population shifts, particularly when changes in the clinical environment initially affect only a subset of patients, resulting in a transition to mixed populations. Such case-mix changes commonly arise following system-level operational updates or the emergence of new diseases, such as COVID-19. We propose TRACER (Transfer Learning-based Real-time Adaptation for Clinical Evolving Risk), a framework that identifies encounter-level transition membership and adapts predictive models using transfer learning without full retraining. In simulation studies, TRACER outperformed static models trained on historical or contemporary data. In a real-world application predicting hospital admission following emergency department visits across the COVID-19 transition, TRACER improved both discrimination and calibration. TRACER provides a scalable approach for maintaining robust predictive performance under evolving and heterogeneous clinical conditions.
Read the full abstract ▾
All authors Mengying Yan, Ziye Tian, Siqi Li, Nan Liu, Benjamin A. Goldstein, Molei Liu, Chuan Hong
Machine Learning
Liquid Reasoning Transformers: A Sudoku-Based Prototype for Chess-Scale Algorithmic Tasks
Shivansh Sahni and Wenzhi Zhang
The
Liquid Reasoning Transformer (LRT) is a
transformer architecture designed for inference with adaptive depths using iterative changes, discard-based correction, and a learned stopping mechanism. Instead of relying on a single feedforward pass, the model updates a recurrent
reasoning token across multiple internal steps, allowing it to correct early errors and allocate computation based on input difficulty. We evaluate the LRT on Sudoku as a controlled testbed for structured
reasoning and show that it achieves strong performance, reaching 98.68% digit accuracy and 36.30% full-puzzle accuracy without using symbolic rules or search. Analyzing internal patterns shows that the discard and stop gates play different, important roles in stabilizing inferences and adjusting computational depth. We discuss how these mechanisms extend naturally to chess-scale
reasoning tasks and outline extensions for multi-
token reasoning and larger domains.
Read the full abstract ▾
Machine Learning
Unveiling Statistical Significance of Online Regression over Multiple Datasets
Mohammad Abu-Shaira and Weishi Shi
Despite extensive focus on techniques for evaluating the performance of two learning
algorithms on a single
dataset, the critical challenge of developing statistical tests to compare multiple
algorithms across various
datasets has been largely overlooked in most machine learning research. Additionally, in the realm of Online Learning, ensuring statistical significance is essential to validate continuous learning processes, particularly for achieving rapid convergence and effectively managing concept drifts in a timely manner. Robust statistical methods are needed to assess the significance of performance differences as data evolves over time. This article examines the state-of-the-art online regression models and empirically evaluates several suitable tests. To compare multiple online regression models across various
datasets, we employed the Friedman test along with corresponding post-hoc tests. For thorough evaluations, utilizing both real and synthetic
datasets with 5-fold cross-validation and seed averaging ensures comprehensive assessment across various data subsets. Our tests generally confirmed the performance of competitive baselines as consistent with their individual reports. However, some statistical test results also indicate that there is still room for improvement in certain aspects of state-of-the-art methods.
Read the full abstract ▾
Machine Learning
OLC-WA: Drift Aware Tuning-Free Online Classification with Weighted Average
Mohammad Abu Shaira, Yunhe Feng, Heng Fan et al. (+1 more)
Real-world data sets often exhibit temporal dynamics characterized by evolving data distributions. Disregarding this phenomenon, commonly referred to as concept drift, can significantly diminish a model's predictive accuracy. Furthermore, the presence of hyperparameters in online models exacerbates this issue. These parameters are typically fixed and cannot be dynamically adjusted by the user in response to the evolving data distribution. This paper introduces Online Classification with Weighted Average (OLC-WA), an adaptive, hyperparameter-free online classification model equipped with an automated optimization mechanism. OLC-WA operates by blending incoming data streams with an existing base model. This blending is facilitated by an exponentially weighted moving average. Furthermore, an integrated optimization mechanism dynamically detects concept drift, quantifies its magnitude, and adjusts the model based on the observed data stream characteristics. This approach empowers the model to effectively adapt to evolving data distributions within streaming environments. Rigorous empirical evaluation across diverse
benchmark datasets shows that OLC-WA achieves performance comparable to batch models in stationary environments, maintaining accuracy within 1-3%, and surpasses leading online baselines by 10-25% under drift, demonstrating its effectiveness in adapting to dynamic data streams.
Read the full abstract ▾
All authors Mohammad Abu Shaira, Yunhe Feng, Heng Fan, Weishi Shi
Machine Learning
Credit Risk Estimation with Non-Financial Features: Evidence from a Synthetic Istanbul Dataset
Atalay Denknalbant, Emre Sezdi and Zeki Furkan Kutlu
Financial exclusion constrains entrepreneurship, increases income
volatility, and widens wealth gaps. Underbanked consumers in Istanbul often have no bureau file because their earnings and payments flow through informal channels. To study how such borrowers can be evaluated we create a synthetic
dataset of one hundred thousand Istanbul residents that reproduces first quarter 2025 TÜİK (TURKSTAT) census marginals and telecom usage patterns.
Retrieval augmented generation feeds these public statistics into the OpenAI o3 model, which synthesises realistic yet private records. Each profile contains seven socio demographic variables and nine alternative attributes that describe phone specifications, online shopping rhythm, subscription spend, car ownership, monthly rent, and a credit card flag. To test the impact of the alternative financial data CatBoost, LightGBM, and XGBoost are each trained in two versions. Demo models use only the socio demographic variables; Full models include both socio demographic and alternative attributes. Across five fold stratified validation the alternative block raises area under the curve by about one point three percentage and lifts balanced F 1 from roughly 0.84 to 0.95, a fourteen percent gain. We contribute an open Istanbul 2025 Q1 synthetic
dataset, a fully reproducible modeling pipeline, and empirical evidence that a concise set of behavioural attributes can approach bureau level discrimination power while serving borrowers who lack formal credit records. These findings give lenders and regulators a transparent blueprint for extending fair and safe credit access to the underbanked.
Read the full abstract ▾
Machine Learning
OLR-WAA: Adaptive and Drift-Resilient Online Regression with Dynamic Weighted Averaging
Mohammad Abu-Shaira and Weishi Shi
Real-world
datasets frequently exhibit evolving data distributions, reflecting temporal variations and underlying shifts. Overlooking this phenomenon, known as concept drift, can substantially degrade the predictive performance of the model. Furthermore, the presence of hyperparameters in online models exacerbates this issue, as these parameters are typically fixed and lack the flexibility to dynamically adjust to evolving data. This paper introduces "OLR-WAA: An Adaptive and Drift-Resilient Online Regression with Dynamic Weighted Average", a hyperparameter-free model designed to tackle the challenges of non-stationary data streams and enable effective, continuous adaptation. The objective is to strike a balance between model stability and adaptability. OLR-WAA incrementally updates its base model by integrating incoming data streams, utilizing an exponentially weighted moving average. It further introduces a unique optimization mechanism that dynamically detects concept drift, quantifies its magnitude, and adjusts the model based on real-time data characteristics. Rigorous evaluations show that it matches batch regression performance in static settings and consistently outperforms or rivals state-of-the-art online models, confirming its effectiveness. Concept drift
datasets reveal a performance gap that OLR-WAA effectively bridges, setting it apart from other online models. In addition, the model effectively handles confidence-based scenarios through a conservative update strategy that prioritizes stable, high-confidence data points. Notably, OLR-WAA converges rapidly, consistently yielding higher R2 values compared to other online models.
Read the full abstract ▾
Machine Learning · Cross-listed from cs.CV
CoRe3D: Collaborative Reasoning as a Foundation for 3D Intelligence
Tianjiao Yu, Xinzhuo Li, Yifan Shen et al. (+2 more)
Recent advances in large
multimodal models suggest that explicit
reasoning mechanisms play a critical role in improving model reliability, interpretability, and cross-modal
alignment. While such
reasoning-centric approaches have been proven effective in
language and
vision tasks, their extension to 3D remains underdeveloped. CoRe3D introduces a unified 3D understanding and generation
reasoning framework that jointly operates over semantic and spatial abstractions, enabling high-level intent inferred from
language to directly guide low-level 3D content formation. Central to this design is a spatially grounded
reasoning representation that decomposes 3D latent space into localized regions, allowing the model to reason over geometry in a compositional and procedural manner. By tightly coupling semantic chain-of-thought inference with structured spatial
reasoning, CoRe3D produces 3D outputs that exhibit strong local consistency and faithful
alignment with linguistic descriptions.
Read the full abstract ▾
All authors Tianjiao Yu, Xinzhuo Li, Yifan Shen, Yuanzhe Liu, Ismini Lourentzou
Machine Learning · Cross-listed from q-bio.NC
Random matrix theory of sparse neuronal networks with heterogeneous timescales
Thiparat Chotibut, Oleg Evnin and Weerawit Horinouchi
Training recurrent neuronal networks consisting of excitatory (E) and inhibitory (I) units with additive noise for working
memory computation slows and diversifies inhibitory timescales, leading to improved task performance that is attributed to emergent marginally stable
equilibria [PNAS 122 (2025) e2316745122]. Yet the link between trained network characteristics and their roles in shaping desirable dynamical landscapes remains unexplored. Here, we investigate the Jacobian
matrices describing the dynamics near these
equilibria and show that they are sparse, non-Hermitian rectangular-block
matrices modified by heterogeneous synaptic decay timescales and activation-function gains. We specify a random
matrix ensemble that faithfully captures the spectra of trained Jacobian
matrices, arising from the inhibitory core - excitatory periphery network motif (pruned E weights, broadly distributed I weights) observed post-training. An analytic theory of this ensemble is developed using statistical field theory methods: a Hermitized resolvent representation of the spectral density is processed with a supersymmetry-based treatment in the style of Fyodorov and Mirlin. In this manner, an analytic description of the spectral edge is obtained, relating statistical parameters of the Jacobians (sparsity, weight variances, E/I ratio, and the distributions of timescales and gains) to near-critical features of the
equilibria essential for robust working
memory computation.
Read the full abstract ▾
Machine Learning
Incheol Baek, Hyungbin Kim, Minseo Kim et al. (+1 more)
Federated Learning (FL) enables collaborative training across multiple clients while preserving data
privacy, yet it struggles with data heterogeneity, where clients' data are not distributed independently and identically (non-IID). This causes local drift, hindering global model convergence. To address this, we introduce
Federated Learning with Feedback
Alignment (FLFA), a novel framework that integrates feedback
alignment into FL. FLFA uses the global model's weights as a shared feedback
matrix during local training's backward pass, aligning local updates with the global model efficiently. This approach mitigates local drift with minimal additional computational cost and no extra communication overhead. Our theoretical analysis supports FLFA's design by showing how it alleviates local drift and demonstrates robust convergence for both local and global models. Empirical evaluations, including accuracy comparisons and measurements of local drift, further illustrate that FLFA can enhance other FL methods demonstrating its effectiveness.
Read the full abstract ▾
All authors Incheol Baek, Hyungbin Kim, Minseo Kim, Yon Dohn Chung
Machine Learning · Cross-listed from eess.SY
An End-to-End Approach for Microgrid Probabilistic Forecasting and Robust Operation via Decision-focused Learning
Tingwei Cao and Yan Xu
High penetration of renewable energy sources (RES) introduces significant uncertainty and intermittency into microgrid operations, posing challenges to economic and reliable scheduling. To address this, this paper proposes an end-to-end decision-focused framework that jointly optimizes probabilistic
forecasting and robust operation for microgrids. A multilayer encoder-decoder (MED) probabilistic
forecasting model is integrated with a two-stage robust optimization (TSRO) model involving direct load control (DLC) through a differentiable decision pathway, enabling gradient-based feedback from operational outcomes to improve
forecasting performance. Unlike conventional sequential approaches, the proposed method aligns
forecasting accuracy with operational objectives by directly minimizing decision regret via a surrogate smart predict-then-optimize (SPO) loss function. This integration ensures that probabilistic forecasts are optimized for downstream decisions, enhancing both economic efficiency and robustness. Case studies on modified IEEE 33-bus and 69-bus systems demonstrate that the proposed framework achieves superior
forecasting accuracy and operational performance, reducing total and net operation costs by up to 18% compared with conventional
forecasting and optimization combinations. The results verify the effectiveness and scalability of the end-to-end decision-focused approach for resilient and cost-efficient microgrid management under uncertainty.
Read the full abstract ▾
Machine Learning · Cross-listed from stat.CO
Residual-augmented flow matching operators for probabilistic partial differential equations
Sahil Bhola and Karthik Duraisamy
Learning surrogate models for physical systems with latent uncertainty remains challenging in data-scarce regimes: deterministic neural operators fail to characterize uncertainty, while generative approaches require large ensembles of high-fidelity solution operator simulations and often sacrifice resolution generalizability. In this work, we propose a residual-augmented probabilistic operator learning framework that casts flow-matching-based generative modeling in infinite-dimensional function spaces while leveraging inexpensive low-fidelity solution operators as an inductive bias. Rather than learning the full high-fidelity
stochastic solution operator directly, the proposed framework learns probabilistic residual operators that characterize the discrepancy between low- and high-fidelity solutions. By parameterizing the vector field in flow matching using neural operators conditioned on both the known system input and low-fidelity solution, the framework amortizes probabilistic inference across input conditions while enabling uncertainty-aware and resolution-generalizable predictions across spatial discretizations. Numerical experiments on
stochastic advection, Burgers', and Darcy flow systems demonstrate that the residual-augmented formulation improves predictive accuracy under the same high-fidelity data budget, while the probabilistic operator learning formulation enables accurate characterization of uncertainty in low-data regimes compared to learning high-fidelity
stochastic operators directly from data.
Read the full abstract ▾
Machine Learning
Resting Neurons, Active Insights: Robustifying Activation Sparsity in LLMs via Spontaneity
Haotian Xu, Jiannan Yang, Tian Gao et al. (+2 more)
Activation sparsity offers a compelling route to accelerate large
language model (LLM) inference by selectively suppressing hidden activations, yet existing approaches exhibit severe accuracy degradation at high sparsity. We show that this failure stems from representational instability: *activation sparsity disrupts input-dependent activation learned during pretraining, inducing distribution shifts in hidden states.* We address this issue by reframing activation sparsity as a representational
alignment problem and introducing **Spontaneous
Neurons (SPON)**, a lightweight mechanism inspired by spontaneous neural activity in biological systems. SPON injects a small set of learnable, input-independent activation vectors that act as persistent representational anchors for sparse computation. These vectors are trained via distribution matching to the dense model and can be absorbed into bias terms after training, incurring negligible inference overhead. Across multiple LLM backbones, SPON consistently restores performance, stabilizes latent representations, and preserves generalization. Our results establish SPON as an effective and principled solution for reliable activation-sparse inference, and offer new insights into knowledge retention in LLMs.
Read the full abstract ▾
All authors Haotian Xu, Jiannan Yang, Tian Gao, Tsui-Wei Weng, Tengfei Ma
Machine Learning · Cross-listed from stat.ML
A Novel Framework Using Variational Inference with Normalizing Flows to Train Transport Reversible Jump Proposals
Pingping Yin and Xiyun Jiao
We propose a unified framework that employs variational inference (VI) with (conditional) normalizing flows (NFs) to train both between-model and within-model proposals for reversible jump Markov chain Monte Carlo, enabling efficient trans-dimensional
Bayesian inference. In contrast to the transport reversible jump (TRJ) of Davies et al. (2023), which optimizes forward KL divergence using pilot samples from the complex target distribution, our approach minimizes the reverse KL divergence, requiring only samples from a simple base distribution and largely reducing computational cost. Especially, we develop a novel trans-dimensional VI method with conditional NFs to fit the conditional transport proposal of Davies et al. (2023). We use RealNVP flows to learn the model-specific transport maps used for constructing proposals so that the calculation is parallelizable. Our framework also provides accurate estimates of marginal likelihoods, which may facilitate efficient model comparison and help design rejection-free proposals. Extensive numerical studies demonstrate that the TRJ method trained under our framework achieves faster mixing compared to existing baselines.
Read the full abstract ▾