Machine Learning
VNN-LIB 2.0: Rigorous Foundations for Neural Network Verification
Ann Roy, Allen Antony, Andrea Gimelli et al. (+1 more)
Neural network verification is an active and rapidly maturing research area, with a growing ecosystem of solvers and tools. The VNN-LIB standard was introduced to support interoperability in this ecosystem, but Version~1.0 has several serious short-comings as a formal foundation: it lacks a precise syntax, semantics, and type system, offers limited expressivity, and relies on externally defined ONNX models whose semantics are informal and constantly evolving. The latter distinguishes VNN-LIB from established standards such as SMT-LIB, where queries are self-contained and have fixed semantics. In this paper we address these challenges by developing the theoretical foundations of VNN-LIB~2.0. Our key contribution is the introduction of the notion of a network theory, which abstractly characterises the minimal semantic interface required from a neural network model format. This abstraction enables VNN-LIB to be defined independently of any specific ONNX version while remaining compatible with evolving model representations. Building on this foundation, we present a formal syntax for a more expressive query
language, a type system for it over the numeric domains provided by the network theory, and finally a formal semantics. To ensure internal consistency, the standard is mechanised in the Agda
theorem prover. VNN-LIB~2.0 therefore provides robust and rigorous foundations for trustworthy neural network verification.
Read the full abstract ▾
All authors Ann Roy, Allen Antony, Andrea Gimelli, Matthew L. Daggitt
Machine Learning · Cross-listed from cs.CV
Sparse Autoencoders as Plug-and-Play Firewalls for Adversarial Attack Detection in VLMs
Hao Wang, Yiqun Sun, Pengfei Wei et al. (+2 more)
Vision-
language models (VLMs) have advanced rapidly and are increasingly deployed in real-world applications, especially with the rise of
agent-based systems. However, their safety has received relatively limited
attention. Even the latest proprietary and open-weight VLMs remain highly vulnerable to
adversarial attacks, leaving downstream applications exposed to significant risks. In this work, we propose a novel and lightweight
adversarial attack detection framework based on sparse autoencoders (SAEs), termed SAEgis. By inserting an SAE module into a pretrained VLM and training it with standard reconstruction objectives, we find that the learned sparse latent features naturally capture attack-relevant signals. These features enable reliable classification of whether an input image has been adversarially perturbed, even for previously unseen samples. Extensive experiments show that SAEgis achieves strong performance across in-domain, cross-domain, and cross-attack settings, with particularly large improvements in cross-domain generalization compared to existing baselines. In addition, combining signals from multiple layers further improves robustness and stability. To the best of our knowledge, this is the first work to explore SAE as a plug-and-play mechanism for
adversarial attack detection in VLMs. Our method requires no additional
adversarial training, introduces minimal overhead, and provides a practical approach for improving the safety of real-world VLM systems.
Read the full abstract ▾
All authors Hao Wang, Yiqun Sun, Pengfei Wei, Lawrence B. Hsieh, Daisuke Kawahara
Machine Learning · Cross-listed from cs.CL
SSP-based construction of evaluation-annotated data for fine-grained aspect-based sentiment analysis
Suwon Choi, Shinwoo Kim, Changhoe Hwang et al. (+3 more)
We report the construction of a Korean evaluation-annotated corpus, hereafter called 'Evaluation Annotated
Dataset (EVAD)', and its use in Aspect-Based Sentiment Analysis (ABSA) extended in order to cover e-commerce reviews containing sentiment and non-sentiment linguistic patterns. The annotation process uses Semi-Automatic Symbolic Propagation (SSP). We built extensive linguistic resources formalized as a Finite-State Transducer (FST) to annotate corpora with detailed ABSA components in the fashion e-commerce domain. The ABSA approach is extended, in order to analyze user opinions more accurately and extract more detailed features of targets, by including aspect values in addition to topics and aspects, and by classifying aspectvalue pairs depending whether values are unary, binary, or multiple. For evaluation, the KoBERT and KcBERT models are trained on the annotated
dataset, showing robust performances of F1 0.88 and F1 0.90, respectively, on recognition of aspect-value pairs.
Read the full abstract ▾
All authors Suwon Choi, Shinwoo Kim, Changhoe Hwang, Gwanghoon Yoo, Eric Laporte, Jeesun Nam
Machine Learning
GameGen-Verifier: Parallel Keypoint-Based Verification for LLM-Generated Games via Runtime State Injection
Chaobo Jia, Ruipeng Wan, Ting Sun et al. (+5 more)
LLM-based game generation promises to turn natural-
language specifications into executable games, but progress is limited by the lack of reliable automated verification. Unlike conventional code generation, game correctness is defined over long-horizon interaction: a game may appear correct while violating core mechanics such as state updates, interaction rules, and phase transitions. Existing
Agent-as-a-Verifier approaches collapse verification into open-ended gameplay, making verdicts reachability-bound, time-consuming, coverage-limited, and sensitive to the
agent's gameplay ability. We present GameGen-Verifier, an automated verification paradigm for LLM-generated games that decomposes a specification into verifiable keypoints and grounds them into independent verification units. Each unit patches the game runtime into a concrete target state, executes a bounded interaction, and judges the outcome against the keypoint assertion. We implement GGV-Harness, a scalable agentic harness providing concurrency management, runtime isolation, and fault recovery. On VeriGame, our
dataset of 100 games across seven genres, GameGen-Verifier achieves up to 92.2% accuracy against human judgments versus 58.8% for the coverage-enforced
Agent-as-a-Verifier baseline, while reducing wall-clock time by up to 16.6x.
Read the full abstract ▾
All authors Chaobo Jia, Ruipeng Wan, Ting Sun, Weihao Tan, Borui Wan, Yuxuan Tong, Guangming Sheng, Hong Xu
Machine Learning · Cross-listed from q-bio.MN
Inference of Qualitative Models from Steady-State Data via Weighted MaxSMT
Ondřej Huvar, Nikola Beneš, Martin Jonáš et al. (+2 more)
Qualitative models provide crucial instruments for modelling complex biological systems. While advances in automated
reasoning and symbolic encodings have enabled rigorous inference of these models from data, the process remains highly fragile. First, biological measurement errors inevitably propagate into formal model specifications. Second, when a specification becomes unsatisfiable, distinguishing between fundamental design flaws and minor technical errors is notoriously difficult. This uncertainty often leads to under-specification, as it is unclear which observations are still “safe” to incorporate. To overcome these challenges, we introduce a robust inference method based on weighted MaxSMT. By encoding uncertain biological observations as weighted soft constraints, our approach enables the solver to identify a model best reflecting the observations, even with some conflicting constraints. Our method allows for Boolean and multi-valued variable domains, alongside observations derived from discretisation (level constraints) and differential expression (ordering constraints). We show our approach can be used to successfully infer neural
cell differentiation models from prior-knowledge networks with 200–1300 genes using ordering constraints on all included genes.
Read the full abstract ▾
All authors Ondřej Huvar, Nikola Beneš, Martin Jonáš, David Šafránek, Samuel Pastva
Machine Learning · Cross-listed from cs.CL
Generating training datasets for legal chatbots in Korean
Changhoe Hwang, Jee-Sun Nam and Eric Laporte
Chatbots are
robots that can communicate with humans using text or voice signals. Legal chatbots improve access to justice, since legal representation and legal advice by lawyers come with a high cost that excludes disadvantaged and vulnerable people. However, capturing the diversity of actual user input in
datasets for deep-learning dialog systems (chatbots) is a technical challenge. Diversity requires large volumes of data, which must also be labelled in order to classify the user's intent, while the cost of labelling
datasets increases with volume. Instead of labelling large volumes of authentic data from users, our approach consists in jointly generating large volumes of utterances and high-quality labels. The generator of labelled
datasets is based on
language resources that take the form of local grammar graphs (LGG), which capture and generalize the vocabulary and local syntax observed by linguists in text. The LGGs associate labels to the utterances according to a domain-specific classification system. We tested this approach by implementing LIGA, a legal chatbot in Korean. The chatbot answers users' conversational queries on legal situations by providing information on similar legal cases, made publicly available by the Korean government. We generated labelled utterances from the LGGs with the aid of the open-source Unitex platform. This process produced 700 million utterances. We trained a DIET classifier on a
dataset made of these utterances, and the trained model reached 91% f1-score performance. We implemented a chatbot called LIGA, which uses the results of the model to select a link to a web page that documents similar legal cases.
Read the full abstract ▾
Machine Learning
A Flexible Adaptive Stable Clustering Algorithm for Archive-Scale Online Mass Spectrometry
Shao Shi, Xin Yang, Huiran Feng et al. (+8 more)
Modern online mass spectrometry generates multi-terabyte data streams critical for understanding Earth's environmental systems. However, extracting actionable chemical insights from these repositories is impeded by a computational bottleneck: existing clustering methods force a compromise among scalability, metric flexibility, and algorithmic stability. Here, we introduce Flexible Adaptive Stable Clustering (FASC), a dynamical systems framework that resolves these constraints by architecturally decoupling the similarity kernel from rigorous optimization logic. Unlike legacy heuristics that suffer from
stochastic drift and algorithmic blending, FASC employs a Density-Augmented Similarity Selection rule and
geometric constraints to guarantee deterministic, order-independent convergence. After validating FASC on canonical machine-learning ground truths (achieving >99.5% cluster purity and 0.99 Adjusted Rand Index), we deployed the framework on 25 million mass spectra of atmospheric aerosols. Demonstrating strictly linear empirical runtime scaling (O(N)), FASC autonomously mapped atmospheric aging pathways of secondary inorganic aerosols while isolating ultra-rare industrial tracers (<0.2% abundance), providing a scalable infrastructure for mining environmental big data.
Read the full abstract ▾
All authors Shao Shi, Xin Yang, Huiran Feng, Jianhuai Ye, Tianlong Hu, Yaling Zeng, Tzung-May Fu, Lei Zhu, Huizhong Shen, Chen Wang, Shu Tao
Machine Learning
SR$^2$-LoRA: Self-Rectifying Inter-layer Relations in Low-Rank Adaptation for Class-Incremental Learning
Fengqiang Wan, Yipeng Lin, Kan Lv et al. (+1 more)
Pre-trained models with parameter-efficient fine-tuning (PEFT) have demonstrated promising potential for class-incremental learning (CIL), yet catastrophic forgetting still persists when adapting models to new tasks. In this paper, we present a novel perspective on catastrophic forgetting through the analysis of inter-layer relation drift, i.e., the progressive disruption of relationships among layer-wise representations during the learning of new tasks. We theoretically show that the increase of such drift reduces the classification margins of previously learned tasks, thereby degrading overall model performance. To address this issue, we propose \underline{S}elf-\underline{R}ectifying inter-layer \underline{R}elation Low-Rank Adaptation~(SR$^2$-LoRA), a simple yet effective method that mitigates catastrophic forgetting by constraining inter-layer relation drift. Specifically, SR$^2$-LoRA constructs the relation
matrices induced by the previous and current models on current-task samples, and aligns the corresponding singular values. We further theoretically show that this
alignment exhibits greater robustness to estimation perturbations than direct entry-wise
alignment. Extensive experiments on standard CIL
benchmarks demonstrate that SR$^2$-LoRA effectively mitigates catastrophic forgetting, with its advantages becoming more pronounced as the number of tasks increases. Code is available in the \href{https://github.com/FqWan24/SR-2-LoRA}{repository}.
Read the full abstract ▾
All authors Fengqiang Wan, Yipeng Lin, Kan Lv, Yang Yang
Machine Learning · Cross-listed from cs.AR
Effective and Memory-Efficient Alternatives to ECC for Reliable Large-Scale DNNs
Mohammad Hasan Ahmadilivani, Marten Roots, Marco Restifo et al. (+3 more)
Modern Deep Learning (DL) workloads are increasingly deployed in safety-critical domains, such as automotive systems and hyperscale data centers, where transient hardware faults pose a serious threat to system reliability. These workloads are highly
memory-intensive, and their correct functionality strongly depends on model parameters stored in
memory, which are typically protected using Error Correction Codes (ECCs). In this work, we study ECC's impact on such models and propose two lightweight alternatives to ECCs that achieve superior reliability. The first approach, MSET, selectively hardens the most vulnerable bits in CNN and ViT parameters, while the second approach, CEP, provides fine-grained protection for all parameter bits. Experimental results demonstrate that both methods significantly enhance the reliability of large CNNs and ViTs, mostly outperforming conventional Single Error Detection Double Error Correction (SECDED) ECC schemes, with no
memory overhead and, in fact, with considerably lower area and delay characteristics when compared to SECDEC. Experimental results indicate that ViTs can be effectively protected by merely protecting their highest exponent bits in FP16 and FP32 representations. Furthermore, applying the CEP technique can guarantee the resilience of DNNs by up to one order of magnitude higher BERs, with a 3.5x lower area overhead and 7x faster decoder compared to SECDED ECC.
Read the full abstract ▾
All authors Mohammad Hasan Ahmadilivani, Marten Roots, Marco Restifo, Sven-Markus Loorits, Luca Di Mauro, Jaan Raik
Machine Learning
Risk-Consistent Multiclass Learning from Random Label-Subset Membership Queries
Jiaxu Su, Junpeng Li, Changchun Hua et al. (+1 more)
Obtaining accurate class labels is often costly or unreliable, and may also be limited by
privacy or other practical conditions. Compared with asking an annotator to provide the exact class, it is often easier to ask whether the true label belongs to a certain label subset. This query-response form defines a distinct weak-supervision mechanism: weak supervision information is generated through feedback on a label subset. Although weakly supervised learning has studied many learning frameworks, most existing work starts from established weak label objects. A systematic characterization is still lacking for weakly supervised learning generated directly by such query response observations. This paper proposes a multiclass learn ing framework under random label-subset queries. We model the data-generating distribution of query-response observations and derive an unbiased
estimator of the target risk under the empirical risk minimization (ERM) framework. To address negative empirical risk and the associated overfitting problem, we introduce corrected risk
estimators based on non-negative and absolute-value corrections. Theoretical analysis establishes a conditional generalization and excess-risk bound for the unbiased
estimator, and a bias-and-consistency result for the corrected risk
estimator. Experiments under the matched random-query mechanism demonstrate the feasibility of direct query-response learning and the stabilization effect of risk correction.
Read the full abstract ▾
All authors Jiaxu Su, Junpeng Li, Changchun Hua, Yana Yang
Machine Learning
Tracking Large-scale Shared Bikes with Inertial Motion Learning in GNSS Blocked Environments
Feng Liu, Kejia Li, Zhiwei Yang et al. (+5 more)
Although Global
Navigation Satellite Systems (GNSS) provide a general solution for bike tracking outdoors, there still exist complex riding environments where only inertial
navigation systems work, such as urban canyons. Despite decades of research, localization using only low-cost inertial sensors still faces challenges such as cumulative drifts and poor robustness caused by filtering methods. Furthermore, sensors such as visual and
LiDAR could provide reliable measurements, but they are not suitable for large-scale deployment. In this paper, we propose an inertial tracking framework that integrates bicycle mechanical constraints with a mixture-of-experts model. Specifically, we leverage multiple expert modules to capture shared representations and weight them through the gating mechanism, thus improving multi-task learning performance and enabling uncertainty-aware trajectory estimation. Furthermore, based on the mechanical transmission between the pedal and the rear wheel of a bike, we explore the intrinsic relationship between the rider's periodic pedalling behaviors and acceleration variations, and convert such patterns into bike's wheel speed for dynamic calibration. Experiments with real-world riding data from shared bikes of the DiDi ride-hailing platform demonstrate that our system improves the accuracy of baselines by at least 12%, with wheel speed errors below 0.5 m/s at 95-percentile.
Read the full abstract ▾
All authors Feng Liu, Kejia Li, Zhiwei Yang, Chunwei Yang, Qun Li, Guobin Wu, Qiang Ni, Ruipeng Gao
Machine Learning · Cross-listed from cs.CL
The Proxy Presumption: From Semantic Embeddings to Valid Social Measures
Baishi Li, Ta Yu, Kelvin J.L. Koa et al. (+1 more)
Natural
Language Processing is rapidly evolving into a primary instrument for Computational Social Science, with researchers increasingly using
embeddings to measure latent constructs such as novelty, creativity, and bias. However, this transition faces a fundamental validity challenge: the ”Proxy Presumption,” or the reliance on
geometric properties (e.g., cosine distance) as direct measures of social concepts. We argue that without explicit validation, unsupervised representations remain entangled mixtures of the target construct ($C$) and confounding attributes ($Z$) like topic, style, and authorship. To bridge the gap between semantic
embeddings and valid social measures, we introduce the Construct Validity Protocol (CVP). Drawing on causal representation learning and psychometrics, the CVP offers a rigorous pipeline from conceptualization to quantitative verification. We further propose Counterfactual Neutralization, a novel method using LLMs to reduce confounding in
embedding space. By providing a standardized Validity Suite – including tests for discriminant, incremental, and predictive validity – this work offers the community a toolkit to transform heuristic proxies into robust, scientifically defensible instruments.
Read the full abstract ▾
All authors Baishi Li, Ta Yu, Kelvin J.L. Koa, Ke-Wei Huang
Machine Learning
Emergent Symbolic Structure in Health Foundation Models: Extraction, Alignment, and Cross-Modal Transfer
Gajendra Katuwal, Advait Koparkar, Salar Abbaspourazad et al. (+2 more)
We show that information can be transferred post-hoc across independently trained health foundation models (FMs), each pretrained on ~20M minutes of wearable sensor data from ~172K participants, by aligning their data-dependent coordinate systems. From frozen
embeddings we extract candidate symbol-like components using linear decomposition methods, and align them across models with simple linear maps. Aligned symbols associate selectively with health conditions and physiological attributes, with associations similar across modalities and architectures. A classifier trained on one model's symbols and applied to another retains more than 95% of its in-domain performance, with similar retention in both directions. Overall, our results indicate that independently trained health FMs converge toward a common representation of the same underlying physiology.
Read the full abstract ▾
All authors Gajendra Katuwal, Advait Koparkar, Salar Abbaspourazad, Anshuman Mishra, Sarvesh Kirthivasan
Machine Learning
Have Graph – Will Lift? The Case for Higher-Order Benchmarks
Bastian Rieck
After a somewhat rocky start, geometry and
topology have established a foothold in machine learning. Message passing, either on graphs or higher-order complexes, is one of the main drivers of
geometric deep learning, and paradigms that were once considered to be firmly in the realm of the abstract-like sheaves-have been "tamed" to serve as novel inductive biases for model architectures in topological deep learning. The veritable diversity of models, however, is in stark contrast to the scarcity of suitable
benchmark datasets. As a result, researchers often resort to lifting existing graph
datasets to include higher-order information. In this opinion paper, I want to encourage the community to also source new
datasets, which may be used to prop up the foundations of our research field.
Read the full abstract ▾
Machine Learning
Rethinking Black-Box On-Policy Distillation: A Rubric-based Analysis
Zhepei Hong, Yukai Wang, Mao Zheng et al. (+7 more)
On-policy
distillation (OPD) is a powerful paradigm for LLM post-training, yet its reliance on teacher logits restricts it to white-box teachers. In this paper, we study whether rubric-based supervision can provide a text-only feedback interface for black-box OPD, and what makes it effective. Our instantiation, Rubric-based On-Policy
Distillation (ROPD), induces rubrics by contrasting teacher responses with the student's current rollouts and uses them to score these rollouts for on-policy optimization. We find that, under matched settings, text-only ROPD outperforms logit-based OPD methods on four competition-math
benchmarks, reaching the best AIME24 accuracy of standard logit-based OPD with 9.6x fewer training samples. Our analysis yields three findings: rubric rewards track correctness far better than teacher log-likelihood (AUC 0.90 vs. 0.35); training with them repairs more failed criteria and regresses fewer passed ones than logit-based supervision; and contrasting teacher responses with student rollouts is central to rubric quality. These results position rubric-based OPD as a flexible, black-box-compatible alternative to logit-based OPD. Our implementation is available at https://github.com/Peregrine123/ROPD_official.
Read the full abstract ▾
All authors Zhepei Hong, Yukai Wang, Mao Zheng, Mingyang Song, Gengsheng Li, Houcheng Jiang, Dan Zhang, Haiyun Guo, Junfeng Fang, Tat-Seng Chua
Machine Learning
Unsolvability Ceiling in Multi-LLM Routing: An Empirical Study of Evaluation Artifacts
Saloni Garg and Amit Sagtani
Efficient routing across multiple LLMs enables cost-quality tradeoffs by directing queries to the cheapest capable model. Prior work attributes routing headroom to an "unsolvability ceiling", queries no model in the pool can solve. We present a large-scale study of multi-tier LLM routing with 206,000 query-model pairs across six
benchmarks (MMLU, MedQA, HumanEval, MBPP, Alpaca, ShareGPT) using the Gemma 4 and Llama 3.1 families. Evaluating with both LLM-as-a-judge and exact-match metrics, we show that a substantial portion of reported unsolvability stems from evaluation artifacts: (i) systematic judge biases favoring verbosity over correctness, (ii) truncation under fixed generation budgets, and (iii) output format mismatches. Through dual-judge validation and exact-match grounding, we reduce measured unsolvability across tasks. We introduce a decomposition framework attributing failures to these artifacts, revealing consistent patterns across domains and model families. These artifacts also distort router training signals: standard routers collapse to majority-class prediction (~79% smallest-tier optimal), confirmed via random-feature and shuffled-label controls, incurring a 13-17 percentage point opportunity cost. We provide actionable recommendations including dual-judge validation, exact-match anchoring, and cost-sensitive objectives. Our findings suggest existing routing headroom estimates are substantially inflated, underscoring the need for reliable evaluation protocols in multi-LLM systems.
Read the full abstract ▾
Machine Learning · Cross-listed from cs.SE
Exploring CoCo Challenges in ML Engineering Teams: Insights From the Semiconductor Industry
A. Azamnouri, M. Haug, L. Woltmann et al. (+3 more)
The integration of machine learning (ML) into complex software systems has increased challenges in collaboration and communication (CoCo) of the teams building these systems. ML engineering (MLE) teams often involve diverse roles, ML engineers, data scientists, software engineers, and domain experts, each bringing unique goals, experiences, and jargon. These interdisciplinary dynamics can make it challenging to deploy, reproduce, and maintain ML-enabled systems over the long term. Previous studies have uncovered several CoCo challenges and practices, but most have focused on software-centric companies, leaving limited empirical understanding of how these dynamics unfold in hardware-centric contexts. In hardware-centric environments, CoCo challenges are shaped by additional constraints such as strict data governance, long development cycles, and tight coupling with physical processes, which amplify coordination complexity and reduce flexibility. To strengthen empirical understanding in such settings, we present a qualitative investigation of MLE teams within a global
semiconductor company, where ML-enabled systems and manufacturing processes introduce additional complexity. We interviewed 12 practitioners regarding CoCo practices, tools, challenges, and approaches. Through analysis, we identified 16 recurring challenges, with unclear roles and responsibilities emerging as the most critical, and common practices and recommendations practitioners considered effective in mitigating CoCo problems. While grounded in a single organizational context, our findings align with known issues in interdisciplinary ML-enabled systems development, but also demonstrate how these challenges manifest differently under hardware-driven constraints. Our results highlight directions for future research and tool support to strengthen CoCo in MLE projects and ensure the success of ML-enabled systems.
Read the full abstract ▾
All authors A. Azamnouri, M. Haug, L. Woltmann, M. Fritz, J. Bogner, S. Wagner
Machine Learning
Convex Optimization with Nested Evolving Feasible Sets
Karthick Krishna M., Haricharan Balasundaram and Rahul Vaze
Convex Optimization with Nested Evolving Feasible Sets (CONES) is considered where the objective function \(f\) remains fixed but the feasible region evolves over time as a nested sequence \(S_1 \supseteq S_2 \supseteq \cdots \supseteq S_T\). The goal of an online
algorithm is to simultaneously minimize the regret with respect to hindsight static optimal
benchmark and the total movement cost $M_\cA(T)$ while ensuring feasibility at all times. CONES is an optimization-oriented generalization of the well-known nested convex body chasing (NCBC). When the loss function is convex, we propose a lazy-
algorithm and show that it achieves $O(T^{1-\beta}), O(T^\beta)$ simultaneous regret and movement cost for any $\beta \in (0,1]$, over a time horizon of $T$. When the loss function is strongly convex, we propose a Frugal
algorithm that simultaneously achieves zero regret and a movement cost of $O(\log T)$. To complement this, we show that any online
algorithm with $o(T)$ regret has a movement cost of $\Omega\left(\sqrt{\frac{\log{T}}{\log \log T}}\right)$.
Read the full abstract ▾
Machine Learning
StreamPhy: Streaming Inference of High-Dimensional Physical Dynamics via State Space Models
Panqi Chen, Yifan Sun, Shikai Fang et al. (+2 more)
Inferring the
evolution of high-dimensional and multi-modal (e.g., spatio-temporal) physical fields from irregular sparse measurements in real time is a fundamental challenge in science and engineering. Existing approaches, including
diffusion-based generative models and functional tensor methods, typically operate in offline settings, depend on full temporal observations, or incur substantial inference cost. We propose StreamPhy, an end-to-end framework that enables efficient and accurate streaming inference of full-field physical dynamics from incoming irregular sparse measurements. The framework integrates a data-adaptive observation encoder that is robust to arbitrary observation patterns, a structured state-space model that supports
memory-efficient online updates across irregular time intervals, and an expressive Functional Tensor Feature-wise Linear Modulation (FT-FiLM) decoder for continuous-field generation. We prove that FT-FiLM is more expressive than the functional Tucker model, admitting a richer function class for handling complex dynamics. Experiments on three representative physical systems under challenging sampling patterns show that StreamPhy consistently outperforms state-of-the-art baselines, with at least 48% improvement in accuracy and up to 20–100X faster inference than
diffusion-based methods.
Read the full abstract ▾
All authors Panqi Chen, Yifan Sun, Shikai Fang, Xiao Fu, Lei Cheng
Machine Learning
Zero-Shot Neural Network Evaluation with Sample-Wise Activation Patterns
Yameng Peng, Andy Song, HaythamM. Fayek et al. (+2 more)
Zero-shot proxies, also known as training-free metrics, are widely adopted to reduce the computational overhead in neural network evaluation for scenarios such as Neural Architecture Search (NAS), as they do not require any training. Existing zero-shot metrics have several limitations, including weak correlation with the true performance and poor generalisation across different networks or downstream tasks. For example, most of these metrics apply only to either convolutional neural networks (CNNs) or
Transformers, but not both. To address these limitations, we propose Sample-Wise Activation Patterns (SWAP), and its derivative, SWAP-Score, a novel and highly effective zero-shot metric. SWAP-Score is broadly applicable across both architecture families and task domains, demonstrating strong predictive performance in the majority of tasks. This metric measures the expressivity of neural networks over a mini-batch of samples, showing a high correlation with the neural networks' ground-truth performance. For both CNNs and
Transformers, the SWAP-Score outperforms existing zero-shot metrics across computer
vision and natural
language processing tasks. For instance, Spearman's correlation coefficient between the SWAP-Score and CIFAR-10 validation accuracy for DARTS CNNs is 0.93, and 0.71 for FlexiBERT
Transformers on GLUE tasks. Moreover, SWAP-Score is label-independent, hence can be applied at the pre-training stage of
language models to estimate their performance for downstream tasks. When applied to NAS, SWAP-empowered NAS, SWAP-NAS can achieve competitive performance using only approximately 6 and 9 minutes of GPU time, on CIFAR-10 and ImageNet respectively. Our code is available at: https://github.com/pym1024/SWAP_Universal
Read the full abstract ▾
All authors Yameng Peng, Andy Song, HaythamM. Fayek, Vic Ciesielski, Xiaojun Chang
Machine Learning
QuadNorm: Resolution-Robust Normalization for Neural Operators
Bum Jun Kim, Makoto Kawano, Yusuke Iwasawa et al. (+1 more)
Normalization layers in neural operators usually compute statistics by uniformly averaging discrete grid values, making the normalization itself discretization-dependent and thereby a source of transfer error across different resolutions or meshes. To enable discretization robustness, we introduce a quadrature normalization family that replaces existing uniform averaging in normalization layers with numerical quadrature: QuadNorm and BlendQuadNorm. On endpoint-inclusive uniform grids, the proposed quadrature moments are $O(h^2)$-consistent across discretizations, meaning that their cross-resolution mismatch decays quadratically with grid spacing. A transfer-error bound then predicts how normalization-induced mismatch scales with both the resolution gap and network depth. The experiments show the same gap- and depth-scaling trends predicted by the transfer-error bound. On Darcy, QuadNorm delivers the best cross-resolution performance at every tested target resolution from $64^2$ to $256^2$; on real-data
benchmarks, Transolver with QuadNorm achieves nearly resolution-invariant transfer. The largest gains appear on nonperiodic PDEs and nonspectral architectures, where native-resolution improvements also emerge. We also validate BlendQuadNorm, which stays close to LayerNorm behavior and serves as a conservative default for periodic FNO settings. These results identify normalization as a previously overlooked source of resolution dependence in neural operators.
Read the full abstract ▾
All authors Bum Jun Kim, Makoto Kawano, Yusuke Iwasawa, Yutaka Matsuo
Machine Learning
FlightSense: An End-to-End MLOps Platform for Real-Time Flight Delay Prediction via Rotation-Chain Propagation Features and Agentic Conversational AI
Aditi J. Shelke, Renuka J. Shelke, Yash M. Kamerkar et al. (+1 more)
Flight delays impose cascading operational and financial burdens across the aviation network, costing the U.S. economy billions of dollars annually by disrupting interconnected aircraft rotation systems. While prior machine learning approaches have demonstrated strong predictive performance, most treat upstream delays as static input variables rather than explicitly modeling how delays propagate dynamically through aircraft rotation chains, and none have deployed such systems alongside a live weather-aware conversational AI interface for end-user interaction. This paper presents FlightSense, an end-to-end MLOps platform for real-time flight delay prediction built through a progressive three-version feature engineering framework. Version 1 trains an XGBoost classifier on 11 schedule-based features establishing a baseline ROC AUC of 0.732 on 7.07 million BTS 2018 On-Time Performance records. Version 2 introduces 11 delay propagation features derived from aircraft rotation chains via tail-number tracking, yielding the dominant performance gain (AUC 0.732 to 0.875) and surpassing the single-stage XGBoost baseline reported by Zhou (2025). Version 3 integrates five NOAA meteorological features across 10 major U.S. airports, achieving a final test set AUC of 0.879. FlightSense is deployed as a production AWS MLOps pipeline incorporating live weather ingestion via Lambda, real-time SageMaker inference, an interactive Streamlit dashboard, and an Amazon Bedrock Nova Micro conversational assistant answering natural-
language delay queries via a tool-use architecture.
Read the full abstract ▾
All authors Aditi J. Shelke, Renuka J. Shelke, Yash M. Kamerkar, Nitin P. Hazarani
Machine Learning
MISA: Mixture of Indexer Sparse Attention for Long-Context LLM Inference
Ruijie Zhou, Fanxu Meng, Yufei Xu et al. (+4 more)
DeepSeek Sparse
Attention (DSA) sets the state of the art for fine-grained inference-time sparse
attention by introducing a learned
token-wise indexer that scores every prefix
token and selects the most relevant ones for the main
attention. To remain expressive, the indexer uses many query heads (for example, 64 on DeepSeek-V3.2) that share the same selected
token set; this multi-head design is precisely what makes the indexer the dominant cost on long contexts. We propose MISA (Mixture of Indexer Sparse
Attention), a drop-in replacement for the DSA indexer that treats its indexer heads as a pool of mixture-of-experts. A lightweight router uses cheap block-level statistics to pick a query-dependent subset of only a few active heads, and only those heads run the heavy
token-level scoring. This preserves the diversity of the original indexer pool while reducing the per-query cost from scoring every prefix
token with every head to scoring it with only a handful of routed heads, plus a negligible router term computed on a small set of pooled keys. We further introduce a hierarchical variant of MISA that uses the routed pass to keep an enlarged candidate set and then re-ranks it with the original DSA indexer to recover the final selected
tokens almost exactly. With only eight active heads and no additional training, MISA matches the dense DSA indexer on LongBench across DeepSeek-V3.2 and GLM-5 while running with eight and four times fewer indexer heads respectively, and outperforms HISA on average. It also preserves fully green Needle-in-a-Haystack heatmaps up to a 128K-
token context and recovers more than 92% of the
tokens selected by the DSA indexer per layer. Our TileLang kernel delivers roughly a 3.82 times speedup over DSA's original indexer kernel on a single NVIDIA H200 GPU.
Read the full abstract ▾
All authors Ruijie Zhou, Fanxu Meng, Yufei Xu, Tongxuan Liu, Guangming Lu, Muhan Zhang, Wenjie Pei
Machine Learning · Cross-listed from cs.CL
Mean-Pooled Cosine Similarity is Not Length-Invariant: Theory and Cross-Domain Evidence for a Length-Invariant Alternative
Sibayan Mitra and Dhruv Kumar
Mean-pooled cosine similarity is the default metric for comparing neural representations across
languages, modalities, and tasks. We establish that this metric is not length-invariant: under the anisotropy that characterizes modern
transformer representations, mean-pooled cosine grows monotonically in sequence length, independent of representational content. Empirically, on HumanEvalPack across four code LLMs, the length ratio alone explains $R^2 = 0.52$–$0.75$ of cross-
language "Python proximity," while AST depth and shared-
token fraction add less than 3% of explained variance beyond length. Substituting Centered Kernel
Alignment (CKA) reduces explained variance by 83% and reverses the sign of the length coefficient ($\beta_{\mathrm{len}}: +0.86 \to -0.37$). The same pattern holds in Mistral-7B on parallel WMT pairs ($R^2 = 0.23$ EN-FR, $R^2 = 0.33$ EN-DE for cosine; $R^2 < 0.01$ for CKA). In CLIP ViT-B/32, mean-pooling reduces the length effect relative to EOS-pooling ($R^2: 0.21 \to {<}0.01$), as predicted by the theory's dependence on anisotropy. We argue that length-invariant metrics such as CKA should be the default for cross-representation comparisons, and that recent claims of cross-lingual representational convergence built on mean-pooled cosine warrant re-examination.
Read the full abstract ▾
Machine Learning
Mage: Multi-Axis Evaluation of LLM-Generated Executable Game Scenes Beyond Compile-Pass Rate
Hugh Xuechen Liu and K{\i}vanç Tatar
Compile-pass rate is the dominant evaluation signal for LLM code generation, yet for multi-component domain-specific artifacts it can be actively misleading. We demonstrate this on executable game scene synthesis with a four-axis evaluation protocol (named `Mage') – compile success, runtime success, structural fidelity, and mechanism adherence – applied to 858 generation attempts across four open-weight LLMs (7B–30B), 26~hand-crafted Unity goal pattern playable concepts, and two automatically extracted IR granularity levels. Direct NL-to-C\# generation achieves the highest runtime-pass rate (43% mean) yet produces structurally vacuous scenes (mechanism $F_1 \approx 0.12$). Structural IR conditioning halves the runtime rate but recovers domain-faithful structure ($F_1$ up to 1.00). Within IR conditioning, behavior-only and full-scene granularity are statistically indistinguishable (McNemar $p = 1.0$), indicating input-level granularity saturation. These results show that compile rate is anti-correlated with functional correctness in this domain and that multi-axis evaluation is necessary to detect the divergence. We release the
benchmark, replay logs, and per-record metrics for independent verification.
Read the full abstract ▾
Machine Learning
CellScientist: From Execution Feedback to Auditable Model-Revision Trajectories for Cellular Perturbation Prediction
Mengran Li, Bo Li, Jiaying Wang et al. (+7 more)
Cellular perturbation-response modeling requires coordinated choices of representations, fusion mechanisms, objectives, and training procedures. Large
language models (LLMs) can propose executable candidates, but unconstrained revision can produce invalid implementations, change task semantics, or discard useful components. We present CellScientist, a protocol-constrained workflow that converts execution and validation feedback into auditable model-revision trajectories. It records design states and outcomes, routes discrepancies to specific components, and applies local revisions under a fixed task contract. A matched-budget study fixes the candidate
language, predictor, fitting, and evaluator: structured revision finds better held-out predictors at small budgets under two LLM backbones. Operational audits link history to fewer repeated proposals, contract checks to contained violations, and discrepancy routing to targeted repairs. Refits of two frozen designs on an independently acquired cohort evaluate external predictive utility. Open-workflow trajectories retain improvements, regressions, and failures, while transcriptomic and single-
cell searches extend application to additional response spaces. CellScientist produces both a selected predictor and an inspectable record of its development. Project page: https://limengran98.github.io/CellScientist/.
Read the full abstract ▾
All authors Mengran Li, Bo Li, Jiaying Wang, Wenbin Xing, Chengyang Zhang, Jinlin Wu, Zhen Lei, Jiebo Luo, Stan Z. Li, Zelin Zang
Machine Learning
Beyond Linear Attention: Softmax Transformers Implement In-Context Reinforcement Learning
Zixuan Xie, Xinyu Liu, Claire Chen et al. (+3 more)
In-context reinforcement learning (ICRL) studies
agents that, after pretraining, adapt to new tasks by conditioning on additional context without parameter updates. Existing theoretical analyses of ICRL largely rely on linear
attention, which replaces the softmax function in the standard
attention with an identity mapping. This paper provides the first theoretical understanding of ICRL without making the unrealistic linear
attention simplification. In particular, we consider the standard softmax
attention used in practice. We show that, with certain parameters, the layerwise forward pass of a
Transformer with such softmax
attention is equivalent to iterative updates of a weighted softmax temporal difference (TD) learning
algorithm. Here, weighted softmax TD is a new RL
algorithm that performs policy evaluation in kernel space and adopts both linear TD and tabular TD as special cases. We also prove that under a certain contraction condition, the policy evaluation error decays as the number of layers grows, with the identified parameters above. Finally, we prove that those parameters are a global minimizer of a pretraining loss, explaining their emergence in our numerical experiments.
Read the full abstract ▾
All authors Zixuan Xie, Xinyu Liu, Claire Chen, Shuze Daniel Liu, Rohan Chandra, Shangtong Zhang
Machine Learning
Rethinking Importance Sampling in LLM Policy Optimization: A Cumulative Token Perspective
Yuheng Zhang, Chenlu Ye, Shuowei Jin et al. (+4 more)
Reinforcement learning, including reinforcement learning with verifiable rewards (RLVR), has emerged as a powerful approach for LLM post-training. Central to these approaches is the design of the importance sampling (IS) ratio used in off-policy policy-gradient estimation. Existing methods face a fundamental bias-variance dilemma:
token-level IS ratios, as adopted by PPO (Schulman et al., 2017) and GRPO (Shao et al., 2024), introduce bias by ignoring prefix state distribution mismatch; full sequence ratios provide exact trajectory-level correction but suffer from high variance due to the multiplicative accumulation of per-
token ratios, while GSPO (Zheng et al., 2025) improves numerical stability via length normalization at the cost of deviating from the exact full-sequence IS correction. In this work, we identify the cumulative
token IS ratio, the product of per-
token ratios up to position $t$, as a theoretically principled solution to this dilemma. We prove that, under the
token-level policy-gradient formulation, this ratio provides an unbiased prefix correction for each
token-level gradient term and has strictly lower variance than the full sequence ratio. Building on this insight, we propose CTPO (Cumulative
Token Policy Optimization), which combines the cumulative
token IS ratio with position-adaptive clipping that scales log-space clip bounds according to the natural $\sqrt{t}$ growth of the cumulative log-ratio. This yields more consistent regularization across
token positions. We implement and evaluate CTPO in the tool-integrated
reasoning setting on several challenging mathematical
reasoning benchmarks, achieving the best average performance across both model scales compared with strong GRPO and GSPO baselines. Code will be available at https://github.com/horizon-llm/CTPO.
Read the full abstract ▾
All authors Yuheng Zhang, Chenlu Ye, Shuowei Jin, Changlong Yu, Wei Xiong, Saurabh Sahu, Nan Jiang
Machine Learning
SparseRL-Sync: Lossless Weight Synchronization with ~100x Less Communication
Lucas Hu, Ranchi Zhao, Isaac Zhu et al. (+4 more)
In large-scale reinforcement learning (RL) systems with decoupled Trainer-Rollout execution, the Trainer must regularly synchronize policy weights to the Rollout side to limit policy staleness. When inter-node bandwidth is abundant, such synchronization is usually only a small fraction of end-to-end cost. As model size grows, however, the communication demand rises rapidly. In bandwidth-constrained or network-variable deployments – for example, cross-datacenter or cross-cluster settings, heterogeneous resource pools, and online RL – weight synchronization can become a dominant bottleneck for throughput and tail latency. We observe that, in mainstream large-model RL training, the locations where parameters actually change are highly sparse at the element level (often 99%+ sparsity). Building on this observation, we propose and implement SparseRL-Sync, which replaces full-weight transfers with a lossless sparse update payload (indices and values) that can be exactly reconstructed on the inference side, thereby preserving 100% fidelity. Under a simplified cost model, sparse synchronization reduces the per-update communication volume from S to approximately S/X; with 99% sparsity (X ~ 100), this yields about a 100x reduction in transmitted data. Combined with appropriate bucketing, SparseRL-Sync also reduces launch and control-plane overhead, significantly improving scalability and end-to-end efficiency in bandwidth-limited and highly asynchronous RL settings.
Read the full abstract ▾
All authors Lucas Hu, Ranchi Zhao, Isaac Zhu, Zach Zhang, Hscos Zhang, Hugh Yin, Jason Zhao
Machine Learning · Cross-listed from cs.CL
Activation Differences Reveal Backdoors: A Comparison of SAE Architectures
Sachin Kumar
Backdoor attacks on
language models pose a significant threat to AI safety, where models behave normally on most inputs but exhibit harmful behavior when triggered by specific patterns. Detecting such
backdoors through mechanistic interpretability remains an open challenge. We investigate two sparse autoencoder architectures – Crosscoders and Differential SAEs (Diff-SAE) – for isolating
backdoor-related features in fine-tuned models. Using a controlled SQL injection
backdoor triggered by year-based context ("2024" triggers vulnerable code, "2023" triggers safe code), we evaluate both approaches across LoRA and full-rank fine-tuning regimes on SmolLM2-360M. We find that Diff-SAE consistently and substantially outperforms Crosscoders for
backdoor isolation. Diff-SAE achieves a
Backdoor Isolation Score (BIS) of 0.40 with perfect precision (1.0) and zero false positive rate across most experimental conditions, while Crosscoders fail almost entirely with BIS below 0.02 in most cases. This performance gap holds across multiple
transformer layers (14, 18, 22, 26) and both fine-tuning regimes, with full-rank fine-tuning producing particularly clean
backdoor signals. Our results suggest that
backdoors manifest as directional activation shifts rather than sparse feature activations, making difference-based representations fundamentally more effective for detection. These findings have important implications for AI safety monitoring and the development of interpretability tools for detecting model manipulation.
Read the full abstract ▾