Machine Learning · Cross-listed from cs.CV
AGMA: Adaptive Gaussian Mixture Anchors for Prior-Guided Multimodal Human Trajectory Forecasting
Chao Li, Rui Zhang, Siyuan Huang et al. (+2 more)
Human trajectory
forecasting requires capturing the
multimodal nature of pedestrian behavior. However, existing approaches suffer from prior misalignment. Their learned or fixed priors often fail to capture the full distribution of plausible futures, limiting both prediction accuracy and diversity. We theoretically establish that prediction error is lower-bounded by prior quality, making prior modeling a key performance bottleneck. Guided by this insight, we propose AGMA (Adaptive Gaussian Mixture Anchors), which constructs expressive priors through two stages: extracting diverse behavioral patterns from training data and distilling them into a scene-adaptive global prior for inference. Extensive experiments on ETH-UCY, Stanford Drone, and JRDB
datasets demonstrate that AGMA achieves state-of-the-art performance, confirming the critical role of high-quality priors in trajectory
forecasting.
Read the full abstract ▾
All authors Chao Li, Rui Zhang, Siyuan Huang, Xian Zhong, Hongbo Jiang
Machine Learning
From Sparse Sensors to Continuous Fields: STRIDE for Spatiotemporal Reconstruction
Yanjie Tong and Peng Chen
Reconstructing high-dimensional spatiotemporal fields from sparse point-sensor measurements is a central challenge in learning parametric PDE dynamics. Existing approaches often struggle to generalize across trajectories and parameter settings, or rely on discretization-tied decoders that do not naturally transfer across meshes and resolutions. We propose STRIDE (Spatio-Temporal Recurrent Implicit DEcoder), a two-stage framework that maps a short window of sensor measurements to a latent state with a temporal encoder and reconstructs the field at arbitrary query locations with a modulated implicit neural representation (INR) decoder. Using the
Fourier Multi-Component and Multi-Layer Neural Network (FMMNN) as the INR backbone improves representation of complex spatial fields and yields more stable optimization than sine-based INRs. We provide a conditional theoretical justification: under stable delay observability of point measurements on a low-dimensional parametric invariant set, the reconstruction operator factors through a finite-dimensional
embedding, making STRIDE-type architectures natural approximators. Experiments on four challenging
benchmarks spanning chaotic dynamics and wave propagation show that STRIDE outperforms strong baselines under extremely sparse sensing, supports super-resolution, and remains robust to noise.
Read the full abstract ▾
Machine Learning · Cross-listed from cs.CL
The Missing Half: Unveiling Training-time Implicit Safety Risks Beyond Deployment
Zhexin Zhang, Yida Lu, Junfeng Fang et al. (+8 more)
Safety risks of AI models have been widely studied at deployment time, such as jailbreak attacks that elicit harmful outputs. In contrast, safety risks emerging during training remain largely unexplored. Beyond explicit reward hacking that directly manipulates explicit reward functions in reinforcement learning, we study implicit training-time safety risks: harmful behaviors driven by a model's internal incentives and contextual background information. For example, during code-based reinforcement learning, a model may covertly manipulate logged accuracy for self-preservation. We present the first systematic study of this problem, introducing a taxonomy with five risk levels, ten fine-grained risk categories, and three incentive types. Extensive experiments reveal the prevalence and severity of these risks: notably, Llama-3.1-8B-Instruct exhibits risky behaviors in 74.4% of training runs when provided only with background information. We further analyze factors influencing these behaviors and demonstrate that implicit training-time risks also arise in multi-
agent training settings. Our results identify an overlooked yet urgent safety challenge in training.
Read the full abstract ▾
All authors Zhexin Zhang, Yida Lu, Junfeng Fang, Junxiao Yang, Shiyao Cui, Hao Zhou, Fandong Meng, Jie Zhou, Hongning Wang, Minlie Huang, Tat-Seng Chua
Machine Learning
LORE: Jointly Learning the Intrinsic Dimensionality and Relative Similarity Structure From Ordinal Data
Vivek Anand, Alec Helbling, Mark A. Davenport et al. (+3 more)
Learning the intrinsic dimensionality of subjective perceptual spaces such as taste, smell, or aesthetics from ordinal data is a challenging problem. We introduce LORE (Low Rank Ordinal
Embedding), a scalable framework that jointly learns both the intrinsic dimensionality and an ordinal
embedding from noisy triplet comparisons of the form, "Is A more similar to B than C?". Unlike existing methods that require the
embedding dimension to be set apriori, LORE regularizes the solution using the nonconvex Schatten-$p$ quasi norm, enabling automatic joint recovery of both the ordinal
embedding and its dimensionality. We optimize this joint objective via an iteratively reweighted
algorithm and establish convergence guarantees. Extensive experiments on synthetic
datasets, simulated perceptual spaces, and real world crowdsourced ordinal judgements show that LORE learns compact, interpretable and highly accurate low dimensional
embeddings that recover the latent geometry of subjective percepts. By simultaneously inferring both the intrinsic dimensionality and ordinal
embeddings, LORE enables more interpretable and data efficient perceptual modeling in psychophysics and opens new directions for scalable discovery of low dimensional structure from ordinal data in machine learning.
Read the full abstract ▾
All authors Vivek Anand, Alec Helbling, Mark A. Davenport, Gordon J. Berman, Sankaraleengam Alagapan, Christopher John Rozell
Machine Learning
Beyond Accuracy: Evaluating Posterior Fidelity of Diffusion Inverse Solvers
Xiaoyu Qiu, Taewon Yang, Zhanhao Liu et al. (+2 more)
Uncertainty evaluation is critical in scientific and engineering inverse problems. However, existing
benchmarks on
Diffusion Inverse Solvers (DIS) primarily focus on reconstruction accuracy but overlook uncertainty and distributional behavior. Since
stochastic inverse solvers represent uncertainty through
diffusion-based posterior samples, evaluating how well their generated samples capture the target posterior distribution becomes an important aspect of uncertainty quantification. To address this limitation and better understand the distributional behavior of
diffusion samplers, we conduct a systematic study to investigate the posterior fidelity of a broad range of existing DIS methods in controlled simulation settings with a known analytical true posterior. Furthermore, to enable posterior-aware evaluation on real-world inverse problems where ground-truth posterior is unavailable, we propose score-based Kernel Stein Discrepancy (score-KSD), a theoretically-grounded and ground-truth-free metric that measures the consistency of the distribution of generated samples from a DIS method with the target posterior score field, induced by the forward model and learned
diffusion prior. Through both simulation experiments and real-world inverse problem solving, we validate the effectiveness of the proposed score-KSD and demonstrate that it provides meaningful posterior fidelity diagnostics beyond reconstruction accuracy, revealing that higher reconstruction accuracy does not necessarily imply better posterior consistency.
Read the full abstract ▾
All authors Xiaoyu Qiu, Taewon Yang, Zhanhao Liu, Guanyang Wang, Liyue Shen
Machine Learning · Cross-listed from cs.CV
Natural Language Instructions for Scene-Responsive Human-in-the-Loop Motion Planning in Autonomous Driving using Vision-Language-Action Models
Angel Martinez-Sanchez, Parthib Roy and Ross Greer
Instruction-grounded driving, where passenger
language guides trajectory planning, requires vehicles to understand intent before motion. However, most prior instruction-following planners rely on simulation or fixed command vocabularies, limiting real-world generalization. doScenes, the first real-world
dataset linking free-form instructions (with referentiality) to nuScenes ground-truth motion, enables instruction-conditioned planning. In this work, we adapt OpenEMMA, an open-source MLLM-based end-to-end driving framework that ingests front-camera views and ego-state and outputs 10-step speed-
curvature trajectories, to this setting, presenting a reproducible instruction-conditioned baseline on doScenes and investigate the effects of human instruction
prompts on predicted driving behavior. We integrate doScenes directives as passenger-style
prompts within OpenEMMA's
vision-
language interface, enabling linguistic conditioning before trajectory generation. Evaluated on 849 annotated scenes using ADE, we observe that instruction conditioning substantially improves robustness by preventing extreme baseline failures, yielding a 98.7% reduction in mean ADE. When such outliers are removed, instructions still influence trajectory
alignment, with well-phrased
prompts improving ADE by up to 5.1%. We use this analysis to discuss what makes a "good" instruction for the OpenEMMA framework. We release the evaluation
prompts and scripts to establish a reproducible baseline for instruction-aware planning. GitHub: https://github.com/Mi3-Lab/doScenes-VLM-Planning
Read the full abstract ▾
Machine Learning · Cross-listed from cs.DB
Piece of CAKE: Adaptive Execution Engines via Microsecond-Scale Learning
Zijie Zhao and Ryan Marcus
Low-level database operators often admit multiple physical implementations ("kernels") that are semantically equivalent but have vastly different performance characteristics depending on the input data distribution. Existing database systems typically rely on static heuristics or worst-case optimal defaults to select these kernels, often missing significant performance opportunities. In this work, we propose CAKE (Counterfactual Adaptive Kernel Execution), a system that learns to select the optimal kernel for each data "morsel" using a microsecond-scale contextual multi-armed bandit. CAKE circumvents the high latency of traditional reinforcement learning by exploiting the cheapness of counterfactuals – selectively running multiple kernels to obtain full feedback – and compiling policies into low-latency regret trees. Experimentally, we show that CAKE can reduce end-to-end workload latency by up to 2x compared to state-of-the-art static heuristics.
Read the full abstract ▾
Machine Learning
Topology-Aware Revival for Efficient Sparse Training
Meiling Jin, Fei Wang, Xiaoyun Yuan et al. (+2 more)
Static sparse training is a promising route to efficient learning by committing to a fixed mask pattern, yet the constrained structure reduces robustness. Early
pruning decisions can lock the network into a brittle structure that is difficult to escape, especially in deep reinforcement learning (RL) where the evolving policy continually shifts the training distribution. We propose
Topology-Aware Revival (TAR), a lightweight one-shot post-
pruning procedure that improves static sparsity without dynamic rewiring. After static
pruning, TAR performs a single revival step by allocating a small reserve budget across layers according to
topology needs, randomly uniformly reactivating a few previously pruned connections within each layer, and then keeping the resulting connectivity fixed for the remainder of training. Across multiple continuous-control tasks with SAC and TD3, TAR improves final return over static sparse baselines by up to +37.9% and also outperforms dynamic sparse training baselines with a median gain of +13.5%.
Read the full abstract ▾
All authors Meiling Jin, Fei Wang, Xiaoyun Yuan, Chen Qian, Yuan Cheng
Machine Learning
BPDQ: Bit-Plane Decomposition Quantization on a Variable Grid for Large Language Models
Junyu Chen, Jungang Li, Jing Xiong et al. (+11 more)
Large
language model inference is often bounded by
memory footprint and bandwidth in resource-constrained deployments, making quantization fundamental to efficient serving. While post-training quantization (PTQ) maintains high fidelity at 4-bit, it deteriorates at 2-3 bits. In essence, existing methods enforce a shape-invariant quantization grid (e.g., the fixed uniform intervals of UINT2) for each group, severely restricting the feasible set for error minimization. To address this, we propose Bit-Plane Decomposition Quantization (BPDQ), which constructs a variable quantization grid via bit-planes and scalar coefficients, and iteratively refines them using second-order information while progressively compensating for quantization errors to minimize output discrepancy. In the 2-bit regime, BPDQ enables serving Qwen2.5-72B on a single RTX 3090 with 83.85% GSM8K accuracy (vs. 90.83% at 16-bit). Moreover, we theoretically show that the variable grid expands the feasible set, and that the quantization process consistently aligns with the optimization objective in Hessian-induced geometry. The code is available at https://github.com/KingdalfGoodman/BPDQ.
Read the full abstract ▾
All authors Junyu Chen, Jungang Li, Jing Xiong, Wenjie Wang, Qingyao Yang, He Xiao, Zhen Li, Taiqiang Wu, Mengzhao Chen, Zhen Peng, Chaofan Tao, Long Shi, Hongxia Yang, Ngai Wong
Machine Learning · Cross-listed from stat.ML
Maximin Relative Improvement: Fair Learning as a Bargaining Problem
Jiwoo Han, Moulinath Banerjee and Yuekai Sun
When deploying a single predictor across multiple subpopulations, we propose a fundamentally different approach: interpreting group
fairness as a bargaining problem among subpopulations. This game-theoretic perspective reveals that existing robust optimization methods such as minimizing worst-group loss or regret correspond to classical bargaining solutions and embody different
fairness principles. We propose relative improvement, the ratio of actual risk reduction to potential reduction from a baseline predictor, which recovers the Kalai-Smorodinsky solution. Unlike absolute-scale methods that may not be comparable when groups have different potential predictability, relative improvement provides axiomatic justification including scale invariance and individual monotonicity. We establish finite-sample convergence guarantees under mild conditions.
Read the full abstract ▾
Machine Learning
Pruning for Generalization: A Transfer-Oriented Spatiotemporal Graph Framework
Zihao Jing, Yuxi Long and Ganlin Feng
Multivariate time series
forecasting in graph-structured domains is critical for real-world applications, yet existing spatiotemporal models often suffer from performance degradation under data scarcity and cross-domain shifts. We address these challenges through the lens of structure-aware context selection. We propose TL-GPSTGN, a transfer-oriented spatiotemporal framework that enhances sample efficiency and out-of-distribution generalization by selectively
pruning non-optimized graph context. Specifically, our method employs information-theoretic and correlation-based criteria to extract structurally informative subgraphs and features, resulting in a compact, semantically grounded representation. This optimized context is subsequently integrated into a spatiotemporal convolutional architecture to capture complex multivariate dynamics. Evaluations on large-scale traffic
benchmarks demonstrate that TL-GPSTGN consistently outperforms baselines in low-data transfer scenarios. Our findings suggest that explicit context
pruning serves as a powerful inductive bias for improving the robustness of graph-based
forecasting models.
Read the full abstract ▾
Machine Learning
Training Data Efficiency in Multimodal Process Reward Models
Jinyuan Li, Chengsong Huang, Langlin Huang et al. (+4 more)
Multimodal Process Reward Models (MPRMs) are central to step-level supervision for visual
reasoning in MLLMs. Training MPRMs typically requires large-scale Monte Carlo (MC)-annotated corpora, incurring substantial training cost. This paper studies the data efficiency for MPRM training. Our preliminary experiments reveal that MPRM training quickly saturates under random subsampling of the training data, indicating substantial redundancy within existing MC-annotated corpora. To explain this, we formalize a theoretical framework and reveal that informative gradient updates depend on two factors: label mixtures of positive/negative steps and label reliability (average MC scores of positive steps). Guided by these insights, we propose the Balanced-Information Score (BIS), which prioritizes both mixture and reliability based on existing MC signals at the rollout level, without incurring any additional cost. Across two backbones (InternVL2.5-8B and Qwen2.5-VL-7B) on VisualProcessBench, BIS-selected subsets consistently match and even surpass the full-data performance at small fractions. Notably, the BIS subset reaches full-data performance using only 10% of the training data, improving over random subsampling by a relative 4.1%.
Read the full abstract ▾
All authors Jinyuan Li, Chengsong Huang, Langlin Huang, Shaoyang Xu, Haolin Liu, Wenxuan Zhang, Jiaxin Huang
Machine Learning · Cross-listed from cs.AI
OMG-Agent: Toward Robust Missing Modality Generation with Decoupled Coarse-to-Fine Agentic Workflows
Ruiting Dai, Zheyu Wang, Haoyu Yang et al. (+6 more)
Data incompleteness severely impedes the reliability of
multimodal systems. Existing reconstruction methods face distinct bottlenecks: conventional parametric/generative models are prone to
hallucinations due to over-reliance on internal
memory, while
retrieval-augmented frameworks struggle with
retrieval rigidity. Critically, these end-to-end architectures are fundamentally constrained by Semantic-Detail
Entanglement – a structural conflict between logical
reasoning and signal synthesis that compromises fidelity. In this paper, we present \textbf{\underline{O}}mni-\textbf{\underline{M}}odality \textbf{\underline{G}}eneration
Agent (OMG-
Agent), a novel framework that shifts the paradigm from static mapping to a dynamic coarse-to-fine Agentic Workflow. By mimicking a deliberate-then-act cognitive process, OMG-
Agent explicitly decouples the task into three synergistic stages: (1) an MLLM-driven Semantic Planner that resolves input ambiguity via Progressive Contextual
Reasoning, creating a deterministic structured semantic plan; (2) a non-parametric Evidence Retriever that grounds abstract semantics in external knowledge; and (3) a
Retrieval-Injected Executor that utilizes retrieved evidence as flexible feature
prompts to overcome
rigidity and synthesize high-fidelity details. Extensive experiments on multiple
benchmarks demonstrate that OMG-
Agent consistently surpasses state-of-the-art methods, maintaining robustness under extreme missingness, e.g., a $2.6$-point gain on CMU-MOSI at $70$% missing rates.
Read the full abstract ▾
All authors Ruiting Dai, Zheyu Wang, Haoyu Yang, Yihan Liu, Chengzhi Wang, Zekun Zhang, Zishan Huang, Jiaman Cen, Lisi Mo
Machine Learning
Generative Neural Operators through Diffusion Last Layer
Sungwon Park, Anthony Zhou, Hongjoong Kim et al. (+1 more)
Neural operators provide a powerful framework for learning discretization invariant mappings between function spaces, but standard deterministic models do not capture predictive uncertainty. We introduce
diffusion last layer (DLL), a modular probabilistic output head for neural operator backbones. DLL represents target fields through an input dependent low rank expansion inspired by the Karhunen-Loéve expansion and learns a conditional
diffusion model over the corresponding coefficient space. This design enables efficient distributional modeling while preserving the structural advantages of operator learning. On
stochastic PDE
benchmarks with random forcing, DLL achieves strong distributional fidelity and performs competitively with pixel space and conventional latent
diffusion baselines. In deterministic long horizon rollout tasks, DLL improves rollout stability over the underlying backbone and provides useful estimates of predictive uncertainty under compounding autoregressive errors. These results suggest that
diffusion modeling in learned coefficient spaces offers a practical route to uncertainty aware neural operators.
Read the full abstract ▾
All authors Sungwon Park, Anthony Zhou, Hongjoong Kim, Amir Barati Farimani
Machine Learning · Cross-listed from eess.SY
LC-SAC: Lyapunov-Constrained Soft Actor-Critic via Koopman Operator Theory for Trajectory Tracking and Stabilization
Dhruv S. Kushwaha and Zoleikha A. Biron
Reinforcement Learning (RL) has achieved remarkable success in solving complex sequential decision-making problems. However, its application to safety-critical physical systems remains constrained by the lack of stability guarantees. Standard RL
algorithms prioritize reward maximization, often yielding policies that may induce oscillations or unbounded state divergence. In this work we propose a Lyapunov-Constrained Soft Actor-Critic (LC-SAC)
algorithm using Koopman operator theory. We learn a linear lifted surrogate of the error dynamics via Extended Dynamic Mode Decomposition (EDMD) and solve the Discrete Algebraic Riccati Equation (DARE) to obtain a closed-form quadratic candidate Control Lyapunov Function (CLF). This CLF is incorporated into the SAC actor update as a Lagrangian penalty that aggregates the worst-case tail of violations via a Conditional Value-at-Risk (CVaR) objective, concentrating constraint pressure on rare but severe instability events. We further introduce three structural EDMD refinements spectral-radius normalization of the lifted A-
matrix prior to the DARE solve, a physically meaningful LQR state cost, and a value-bias anchor enforcing V(0)=0 that make the closed-form CLF well-posed for higher-dimensional lifted models such as the cartpole and 3D quadrotor. The ablation study shows that a hard Lagrangian constraint is essential, replacing it with reward shaping (Lyap-RS-SAC) destabilizes learning and collapses return on quadrotor tasks.
Read the full abstract ▾
Machine Learning
Decoupling Time and Risk: Risk-Sensitive Reinforcement Learning with General Discounting
Mehrdad Moghimi, Anthony Coache and Hyejin Ku
Distributional reinforcement learning (RL) is a powerful framework increasingly adopted in safety-critical domains for its ability to optimize risk-sensitive objectives. However, the role of the discount factor is often overlooked, as it is typically treated as a fixed parameter of the Markov decision process or tunable hyperparameter, with little consideration of its effect on the learned policy. In the literature, it is well-known that the discounting function plays a major role in characterizing time preferences of an
agent, which an exponential discount factor cannot fully capture. Building on this insight, we propose a novel framework that supports flexible discounting of future rewards and optimization of risk measures in distributional RL. We provide a technical analysis of the optimality of our
algorithms, show that our multi-horizon extension fixes issues raised with existing methodologies, and validate the robustness of our methods through extensive experiments. Our results highlight that discounting is a cornerstone in decision-making problems for capturing more expressive temporal and risk preferences profiles, with potential implications for real-world safety-critical applications.
Read the full abstract ▾
Machine Learning · Cross-listed from stat.ML
Attack-Resistant Uniform Fairness for Linear and Smooth Contextual Bandits
Qingwen Zhang and Wenjia Wang
Modern digital platforms use contextual bandits to allocate valuable exposure and opportunities among competing participants. Fair treatment is therefore an important concern, yet reward maximization alone does not ensure that preferential allocation reflects participants' merits. We develop
algorithms for linear and smooth contextual bandits under uniform merit-based
fairness, requiring the reward ordering to justify preferential allocation across all contexts and rounds, and study how these guarantees are affected by
adversarial reward corruption. Our
algorithms achieve \((1-\widetilde O(1/T))\)-
fairness, with regret that is minimax optimal among fair policies for linear rewards and nearly minimax optimal for smooth rewards. In the linear setting, matching lower bounds identify the price of
fairness exactly: minimax regret increases from \(\log T\) to \(\log^2 T\). For smooth rewards, the cost of
fairness is at most polylogarithmic. We further establish a separation between regret and
fairness robustness: an \(\widetilde O(1)\) corruption budget can cause substantial
fairness violations without worsening the regret order. We therefore develop robust
algorithms that adapt sampling, estimation, and
fairness certification to corruption, which preserve uniform
fairness and achieve minimax-optimal and nearly optimal regrets for linear and smooth rewards, respectively. Numerical and semi-synthetic experiments illustrate these findings.
Read the full abstract ▾
Machine Learning
Scalable Explainability-as-a-Service (XaaS) for Edge AI Systems
Samaresh Kumar Singh and Joyjit Roy
Though Explainable AI (XAI) has made significant advancements, its inclusion in edge and IoT systems is typically ad-hoc and inefficient. Most current methods are "coupled" in such a way that they generate explanations simultaneously with model inferences. As a result, these approaches incur redundant computation, high latency and poor scalability when deployed across heterogeneous sets of edge devices. In this work we propose Explainability-as-a-Service (XaaS), a distributed architecture for treating explainability as a first-class system service (as opposed to a model-specific feature). The key innovation in our proposed XaaS architecture is that it decouples inference from explanation generation allowing edge devices to request, cache and verify explanations subject to resource and latency constraints. To achieve this, we introduce three main innovations: (1) A distributed explanation cache with a semantic similarity based explanation
retrieval method which significantly reduces redundant computation; (2) A lightweight verification protocol that ensures the fidelity of both cached and newly generated explanations; and (3) An adaptive explanation engine that chooses explanation methods based upon device capability and user requirement. We evaluated the performance of XaaS on three real-world edgeAI use cases: (i) manufacturing quality control; (ii) autonomous vehicle perception; and (iii) healthcare diagnostics. Experimental results show that XaaS reduces latency by 38% while maintaining high explanation quality across three real-world deployments. Overall, this work enables the deployment of transparent and accountable AI across large scale, heterogeneous IoT systems, and bridges the gap between XAI research and edge-practicality.
Read the full abstract ▾
Machine Learning
Synthesizable Molecular Generation via Soft-constrained GFlowNets with Rich Chemical Priors
Hyeonah Kim, Minsu Kim, Celine Roget et al. (+5 more)
The application of generative models for experimental drug discovery campaigns is severely limited by the difficulty of designing molecules de novo that can be synthesized in practice. Previous works have leveraged Generative Flow Networks (GFlowNets) to impose hard synthesizability constraints through the design of state and action spaces based on predefined reaction templates and building blocks. Despite the promising prospects of this approach, it currently lacks flexibility and scalability. As an alternative, we propose S3-GFN, which generates synthesizable SMILES molecules via simple soft regularization of a sequence-based GFlowNet. Our approach leverages rich molecular priors learned from large-scale SMILES corpora to steer molecular generation towards high-reward, synthesizable chemical spaces. The model induces constraints through off-policy replay training with a contrastive learning signal based on separate buffers of synthesizable and unsynthesizable samples. Our experiments show that S3-GFN learns to generate synthesizable molecules ($\geq 95\%$) with higher rewards in diverse tasks.
Read the full abstract ▾
All authors Hyeonah Kim, Minsu Kim, Celine Roget, Dionessa Biton, Louis Vaillancourt, Yves V. Brun, Yoshua Bengio, Alex Hernandez-Garcia
Machine Learning
Learning to Reason in 13 Parameters
John X. Morris, Niloofar Mireshghallah, Mark Ibrahim et al. (+1 more)
Recent research has shown that
language models can learn to reason, often via reinforcement learning. Some work even trains low-rank parameterizations for
reasoning, but conventional LoRA cannot scale below the model dimension. We question whether even rank=1 LoRA is necessary for learning to reason and propose TinyLoRA, a method for scaling low-rank adapters to sizes as small as one parameter. Within our new parameterization, we are able to train the 8B parameter size of Qwen2.5 to 91% accuracy on GSM8K with only 13 trained parameters in bf16 (26 total bytes). We find this trend holds in general: we are able to recover 90% of performance improvements while training $1000x$ fewer parameters across a suite of more difficult learning-to-reason
benchmarks such as AIME, AMC, and MATH500. Notably, we are only able to achieve such strong performance with RL: models trained using SFT require $100-1000x$ larger updates to reach the same performance.
Read the full abstract ▾
All authors John X. Morris, Niloofar Mireshghallah, Mark Ibrahim, Saeed Mahloujifar
Machine Learning
Toward Effective Multimodal Graph Foundation Model: A Divide-and-Conquer Based Approach
Sicheng Liu, Xunkai Li, Daohan Su et al. (+4 more)
Graph Foundation Models (GFMs) have achieved remarkable success in generalizing across diverse domains. However, they mainly focus on Text-Attributed Graphs (TAGs), leaving
Multimodal-Attributed Graphs (MAGs) largely untapped. Developing
Multimodal Graph Foundation Models (MGFMs) allows for leveraging the rich
multimodal information in MAGs, and extends applicability to broader types of downstream tasks. While recent MGFMs integrate diverse modality information, our empirical investigation reveals two fundamental limitations of existing MGFMs: (1)they fail to explicitly model modality interaction, essential for capturing intricate cross-modal semantics beyond simple aggregation, and (2)they exhibit sub-optimal modality
alignment, which is critical for bridging the significant semantic disparity between distinct modal spaces. To address these challenges, we propose
PLANET (graPh
topoLogy-aware modAlity iNteraction and
alignmEnT), a novel framework employing a Divide-and-Conquer strategy to decouple modality interaction and
alignment across distinct granularities. At the
embedding granularity, (1)
Embedding-wise Domain Gating (EDG) performs local semantic enrichment by adaptively infusing
topology-aware cross-modal context, achieving modality interaction. At the node granularity, (2)Node-wise Discretization
Retrieval (NDR) ensures global modality
alignment by constructing a Discretized Semantic Representation Space (DSRS) to bridge modality gaps. Extensive experiments demonstrate that
PLANET significantly outperforms state-of-the-art baselines across diverse graph-centric and
multimodal generative tasks.
Read the full abstract ▾
All authors Sicheng Liu, Xunkai Li, Daohan Su, Ru Zhang, Hongchao Qin, Ronghua Li, Guoren Wang
Machine Learning
Turning mechanistic models into forecasters by using machine learning
Amit K. Chakraborty, Hao Wang and Pouria Ramazi
The equations of complex dynamical systems may not be identified by expert knowledge, especially if the underlying mechanisms are unknown. Data-driven discovery methods address this challenge by inferring governing equations from time-series data using a library of functions constructed from the measured variables. However, these methods typically assume time-invariant coefficients, which limits their ability to capture evolving system dynamics. To overcome this limitation, we allow some of the parameters to vary over time, learn their temporal
evolution directly from data, and infer a system of equations that incorporates both constant and time-varying parameters. We then transform this framework into a
forecasting model by predicting the time-varying parameters and substituting these predictions into the learned equations. The model is validated using
datasets for Susceptible-Infected-Recovered, Consumer–Resource, greenhouse gas concentration, and Cyanobacteria
cell count. By dynamically adapting to temporal shifts, our proposed model achieved a mean absolute error below 3% for learning a time series and below 6% for
forecasting up to a month ahead. We additionally compare
forecasting performance against CNN-LSTM and Gradient Boosting Machine (GBM), and show that our model outperforms these methods across most
datasets. Our findings demonstrate that integrating time-varying parameters into data-driven discovery of differential equations improves both modeling accuracy and
forecasting performance.
Read the full abstract ▾
Machine Learning · Cross-listed from cs.CR
ZKBoost: Zero-Knowledge Verifiable Training for XGBoost
Nikolas Melissaris, Antigoni Polychroniadou, Akira Takahashi et al. (+2 more)
Gradient boosted decision trees, particularly XGBoost, are among the most effective methods for tabular data. As deployment in sensitive settings increases, cryptographic guarantees of model integrity become essential. We present ZKBoost, the first zero-knowledge proof of training (zkPoT) protocol for XGBoost, enabling model owners to prove correct training on a committed
dataset without revealing data or model parameters. Naively re-executing XGBoost training in ZK would incur prohibitive costs, primarily due to the oblivious partitioning of training samples and unknown tree splits. Moreover, previous work on ZKP of training and inference had subtle security issues, such as leakage of tree
topology and soundness gaps allowing cheating model providers to deviate from the correct execution of training and inference. We make two key contributions to address these challenges: (1) a generic zkPoT template for XGBoost that can be instantiated with any general-purpose ZKP backend, significantly improving prover costs compared to naive re-execution of the training process; and (2) a VOLE-based instantiation that overcomes the security issues of previous ZK proofs of training at minimal costs. To maximize efficiency, we develop a fixed-point version of XGBoost, which is particularly well suited for efficient instantiation of ZKP, and show it matches standard XGBoost accuracy to within 1% on real-world
datasets.
Read the full abstract ▾
All authors Nikolas Melissaris, Antigoni Polychroniadou, Akira Takahashi, Chenkai Weng, Jiayi Xu
Machine Learning
Rate-Optimal Noise Annealing in Semi-Dual Neural Optimal Transport: Tangential Identifiability, Off-Manifold Ambiguity, and Guaranteed Recovery
Raymond Chu, Jaewoong Choi and Dohyun Kwon
Semi-dual neural optimal transport learns a transport map via a max-min objective, yet training can converge to incorrect or degenerate maps. We fully characterize these spurious solutions in the common regime where data concentrate on low-dimensional
manifold: the objective is underconstrained off the data
manifold, while the on-
manifold transport signal remains identifiable. Following Choi, Choi, and Kwon (2025), we study additive-noise smoothing as a remedy and prove new map recovery guarantees as the noise vanishes. Our main practical contribution is a computable terminal noise level $\varepsilon_{\mathrm{stat}}(N)$ that attains the optimal statistical rate, with scaling governed by the intrinsic dimension $m$ of the data. The formula arises from a theoretical unified analysis of (i) quantitative stability of optimal plans, (ii) smoothing-induced bias, and (iii) finite-sample error, yielding rates that depend on $m$ rather than the ambient dimension. Finally, we show that the reduced semi-dual objective becomes increasingly ill-conditioned as $\varepsilon \downarrow 0$. This provides a principled stopping rule: annealing below $\varepsilon_{\mathrm{stat}}(N)$ can $\textit{worsen}$ optimization conditioning without improving statistical accuracy.
Read the full abstract ▾
Machine Learning
Supervised Learning as Lossy Compression: Characterizing Generalization and Sample Complexity via Finite Blocklength Analysis
Kosuke Sugiyama and Masato Uchida
This paper presents a novel information-theoretic perspective on generalization in machine learning by framing the learning problem within the context of lossy compression and applying finite blocklength analysis. In our approach, the sampling of training data formally corresponds to an encoding process, and the model construction to a decoding process. By leveraging finite blocklength analysis, we derive lower bounds on sample complexity and generalization error for a fixed randomized learning
algorithm and its associated optimal sampling strategy. Our bounds explicitly characterize the degree of overfitting of the learning
algorithm and the mismatch between its inductive bias and the task as distinct terms. This separation provides a significant advantage over existing frameworks. Additionally, we decompose the overfitting term to show its theoretical connection to existing metrics found in information-theoretic bounds and stability theory, unifying these perspectives under our proposed framework.
Read the full abstract ▾
Machine Learning
Rethinking Perplexity: Revealing the Impact of Input Length on Perplexity Evaluation in LLMs
Letian Cheng, Junyan Wang, Yan Gao et al. (+3 more)
Perplexity is a widely adopted metric for assessing the predictive quality of large
language models (LLMs) and often serves as a reference metric for downstream evaluations. However, recent evidence shows that perplexity can be unreliable, especially when irrelevant long inputs are used, raising concerns for both benchmarking and system deployment. While prior efforts have employed selective input filtering and curated
datasets, the impact of input length on perplexity has not been systematically studied from a systems perspective and input length has rarely been treated as a first-class system variable affecting both
fairness and efficiency. In this work, we close this gap by introducing LengthBenchmark, a system-conscious evaluation framework that explicitly integrates input length, evaluation protocol design, and system-level costs, evaluating representative LLMs under two scoring protocols (direct accumulation and fixed window sliding) across varying context lengths. Unlike prior work that focuses solely on accuracy-oriented metrics, LengthBenchmark additionally measures latency,
memory footprint, and evaluation cost, thereby linking predictive metrics to deployment realities. We further incorporate quantized variants not as a main contribution, but as robustness checks, showing that length-induced biases persist across both full-precision and compressed models. This design disentangles the effects of evaluation logic, quantization, and input length, and demonstrates that length bias is a general phenomenon that undermines fair cross-model comparison. Our analysis yields two key observations: (i) sliding window evaluation consistently inflates performance on short inputs, and (ii) both full-precision and quantized models appear to realise gains as the evaluated segment length grows.
Read the full abstract ▾
All authors Letian Cheng, Junyan Wang, Yan Gao, Elliott Wen, Ting Dang, Hong Jia
Machine Learning
CORE: Context-Robust Remasking for Diffusion Language Models
Kevin Zhai, Sabbir Mollah, Zhenyi Wang et al. (+1 more)
Standard decoding in Masked
Diffusion Models (MDMs) is hindered by context
rigidity:
tokens are retained based on transient high confidence, often ignoring that early predictions lack full context. This creates cascade effects where initial inconsistencies misguide the remaining generation. Existing revision strategies attempt to mitigate this by relying on static confidence scores, but these signals are inherently myopic; inconsistent
tokens can appear confident to the model itself. We propose Context-Robust Remasking (CORE), a training-free framework for inference-time revision. Rather than trusting static
token probabilities, CORE identifies context-brittle
tokens by probing their sensitivity to targeted masked-context perturbations. We formalize revision as a robust optimization objective over context shifts and efficiently approximate this objective to prioritize unstable
tokens for revision. On LLaDA-8B-Base, CORE delivers consistent improvements across
reasoning and code
benchmarks, outperforming compute-matched baselines and improving MBPP by up to 9.2 percentage points.
Read the full abstract ▾
All authors Kevin Zhai, Sabbir Mollah, Zhenyi Wang, Mubarak Shah
Machine Learning
Federated Concept-Based Models: Interpretable models with distributed supervision
Dario Fenoglio, Arianna Casanova, Francesco De Santis et al. (+6 more)
Concept-based Models (CMs) enhance interpretability in deep learning by grounding predictions in human-understandable concepts. However, concept annotations are costly and rarely available at scale within a single data source.
Federated Learning (FL) could alleviate this limitation by enabling cross-institutional training over concept annotations distributed across multiple data owners. Yet, FL lacks interpretable modeling paradigms. Integrating CMs with FL is non-trivial: although FL supports heterogeneous and non-stationary client participation, it typically assumes a fixed shared architecture, whereas CMs may require architectural adaptation as the available concept set evolves. We propose
Federated Concept-based Models (F-CMs), a new methodology for deploying CMs in evolving FL settings. F-CMs aggregate concept-level information across institutions and efficiently adapt the model architecture to changes in concept supervision while preserving
privacy. Empirically, F-CMs maintain accuracy and intervention effectiveness comparable to training settings with full concept supervision, while outperforming on average non-adaptive
federated baselines. Notably, F-CMs enable interpretable inference on concepts unavailable to a given institution, a key novelty over existing approaches.
Read the full abstract ▾
All authors Dario Fenoglio, Arianna Casanova, Francesco De Santis, Gabriele Dominici, Johannes Schneider, Pietro Barbiero, Giovanni De Felice, Marc Langheinrich, Martin Gjoreski