Machine Learning
Compressed Video Aggregator: Content-driven Module for Efficient Micro-Video Recommendation
Yang Xiao, Huiyuan Chen, Kaiyuan Deng et al. (+6 more)
We propose Compressed
Video Aggregator (CVA), a lightweight micro-
video recommendation module that decouples
video information from preference learning. CVA first summarizes frozen VFM frame
embeddings into a semantic-consensus anchor through masked mean pooling, projects this anchor into a compact latent space, and refines the projected representation with residual self-
attention and feedforward blocks before producing a single
video embedding for the recommender. Due to the redundancy in the frame count of the original
benchmark and its overly coarse sampling, we used titles to re-select key frames based on CLIP. Experiments on MicroLens and Short-
Video show consistent gains with orders-of-magnitude reductions in training time and GPU
memory, and re-selected frames can further enhance the performance of all methods, including CVA. Furthermore, we also discussed the impact of several scenarios involving erroneous titles on our method.
Read the full abstract ▾
All authors Yang Xiao, Huiyuan Chen, Kaiyuan Deng, Chao Jiang, Zinan Ling, Ruimeng Ye, Fei Wang, Xiaolong Ma, Bo Hui
Machine Learning · Cross-listed from cs.CV
Curvature-Aware Captioning:Leveraging Geodesic Attention for 3D Scene Understanding
Ziyao He, Yingjie Liu, ZhangYangRui et al. (+3 more)
Accurate 3D scene description is fundamental to robotic
navigation and augmented reality, yet current dense captioning methods face significant limitations in processing sparse point cloud data. % Existing approaches that apply Euclidean
embedding spaces struggle to simultaneously preserve fine-grained local
geometric details and model exponentially growing global semantic hierarchies, leading to either inaccurate localization or disjointed, shallow scene descriptions. % In this work, we propose a novel \textbf{
Curvature-Aware Captioning} framework, integrating novel non-Euclidean geodesic
attention mechanisms, to resolve the localization-contextualization conflict. % Specifically, self-
attention within Oblique space enforces dimensional homogeneity while establishing long-range dependencies. Bidirectional geodesic cross-
attention within Lorentz space models hierarchical semantic relationships across scene instances, enabling simultaneous precision in object localization and coherence in scene descriptions. % Theoretical analysis confirms that the
curvature complementarity between the Oblique
manifold and Lorentz hyperboloid resolves the Euclidean-
hyperbolic conflict, ensuring feature stability via isotropic optimization while preserving inherent hierarchical relationships. Extensive experiments on ScanRefer and Nr3D
benchmarks demonstrate state-of-the-art performance, with significant gains in both localization accuracy and descriptive richness.
Read the full abstract ▾
All authors Ziyao He, Yingjie Liu, ZhangYangRui, Mingsong Chen, Xuan Tang, Xian Wei
Machine Learning
Data-driven transport modelling without overfit
Peter Vanya, Katarína Šimková and Rastislav Farkaš
Macroscopic transport modelling aims to predict traffic flows after proposed public policy interventions, such as a new road or railway section or a temporary road closure. As such, it is a vital step in infrastructure planning and development. Traditionally, building a transport model has relied on complex understanding of socio-economic characteristics of the population requiring expensive data collection via surveys, which are prone to biases. Previous numerical frameworks to optimize transport models to fit observed traffic flows are not easily-interpretable and can lead to overfit. We present here an alternative: a data-driven modelling protocol with objective function based on traffic counts, which can be nowadays cheaply and reliably obtained; explainable model weights; and a controlled path to increase model complexity and accuracy. We demonstrate our approach on several toy and realistic examples, and suggest ways to generalize to
multimodal systems including public transport.
Read the full abstract ▾
Machine Learning
Deterministic Decomposition of Stochastic Generative Dynamics
Xingyu Song, Yuan Mei and Naoya Takeishi
Modern generative models can be understood as probability transport from a simple base distribution to a target data distribution. Deterministic transport models offer tractable velocity-field parameterizations, whereas
stochastic generative models capture richer density
evolution through drift and
diffusion. Yet when
stochastic dynamics are described through deterministic velocity fields, the effects of drift and
diffusion are often compressed into a single effective field, obscuring the distinct roles of deterministic
evolution and
stochastic fluctuation. In this work, we show that the deterministic field \(b_t\) of a
stochastic generative process admits a natural transport–osmotic decomposition that separates deterministic transport from
stochastic,
diffusion-induced effects: \(b_t = u_t + d_t\), where \(u_t\) governs marginal probability transport and \(d_t\) captures an osmotic effect induced by
diffusion and determined by the marginal score. Based on this decomposition, we propose Bridge Matching, a flow-based framework for learning decomposed generative dynamics through both marginal and conditional formulations. In generative modeling experiments, we recombine the learned components as \(b_t = u_t + \lambda_d d_t\), showing that the proposed decomposition enables interpretable and controllable sampling by adjusting the osmotic contribution in probability transport.
Read the full abstract ▾
Machine Learning · Cross-listed from cs.MS
cuRegOT: A GPU-Accelerated Solver for Entropic-Regularized Optimal Transport
Yixuan Qiu
Optimal transport (OT) has emerged as a fundamental tool in modern machine learning, yet its computational cost remains a significant bottleneck for large-scale applications. While harnessing the massive parallelism of modern GPU hardware is critical for efficiency, the de facto standard Sinkhorn
algorithm, despite its ease of parallelization, often suffers from slow convergence in challenging problems. More recently, the sparse-plus-low-rank quasi-Newton method offers a balance between convergence rate and per-iteration complexity; however, its efficiency on GPUs is severely hindered by the serial nature of sparse
matrix symbolic analysis and irregular
memory access patterns. To bridge this gap, we present cuRegOT, a high-performance GPU solver tailored for entropic-regularized OT. We introduce a suite of algorithmic and architectural optimizations, including an amortized symbolic analysis strategy to mitigate CPU bottlenecks, an asynchronous Sinkhorn iterates generation mechanism, and a fused kernel for bandwidth-efficient gradient evaluation. These strategies are backed by rigorous theoretical guarantees ensuring algorithmic convergence. Extensive numerical experiments demonstrate that cuRegOT achieves significant speedups over state-of-the-art GPU-based solvers across a variety of
benchmark tasks.
Read the full abstract ▾
Machine Learning
PRIM: Meta-Learned Bayesian Root Cause Analysis
Christopher Lohse, Anish Dhir, Amadou Ba et al. (+3 more)
Root Cause Analysis (RCA) aims to identify the main drivers of anomalies in complex multivariate systems. The task is challenging due to the need for structural causal knowledge for some methods, and the computational cost of inference at test time. This paper introduces PRIM (Prior-fitted Root cause Identification with Meta-learning), a causal meta-learning approach that frames RCA as a
Bayesian inference task over a prior of causal models. By marginalising out structural uncertainty, PRIM detects changes in the data-generating mechanism between normal and anomalous periods, infers distributional differences without explicit statistical testing, and implicitly learns causal structure without model fitting at test time. Following the simulation-based meta-learning paradigm of prior-fitted networks (PFNs), PRIM uses a
transformer neural process (TNP) that jointly attends over normal and anomalous samples and the structural dependencies of nodes, enabling fast zero-shot inference in systems with up to 100 variables. Across synthetic
benchmarks and two realistic
benchmark datasets, PetShop and CausRCA, PRIM is competitive with other graph-free methods.
Read the full abstract ▾
All authors Christopher Lohse, Anish Dhir, Amadou Ba, Bradley Eck, Marco Ruffini, Jonas Wahl
Machine Learning · Cross-listed from cs.AI
Not All Turns Matter: Credit Assignment for Multi-Turn Jailbreaking
Zhida He, Xiaoyu Wen, Han Qi et al. (+7 more)
Deploying LLMs in multi-turn dialogues facilitates jailbreak attacks that distribute harmful intent across seemingly benign turns. Recent training-based multi-turn jailbreak methods learn long-horizon attack strategies from interaction feedback, but often rely on coarse trajectory-level outcome signals that broadcast uniformly to every turn. However, we find that turn-level contributions in multi-turn jailbreaking are non-uniform, phase-dependent, and target-specific. Such coarse outcome supervision induces a credit assignment problem, leading to over-rewarding redundant turns in successful trajectories and under-crediting useful intermediate turns in failed ones. To address this, we propose TRACE, a turn-aware credit assignment framework for reinforcement learning (RL)-based multi-turn jailbreaking. For successful trajectories, TRACE estimates turn-level contributions via leave-one-turn-out semantic masking; for failed ones, TRACE assigns penalties based on
prompt harmfulness and semantic relevance, with an additional local refusal-aware penalty. Furthermore, we reuse the attack-side credit signal for multi-turn defense
alignment. Extensive experiments on open-source and closed-source targets show that TRACE achieves strong overall performance in effectiveness, transferability, and efficiency, yielding about a 25% relative improvement in attack success rate over the strongest RL baseline while also improving the safety-utility balance when reused for defense
alignment.
Read the full abstract ▾
All authors Zhida He, Xiaoyu Wen, Han Qi, Ziyuan Zhou, Peng Yu, Xingcheng Xu, Dongrui Liu, Xia Hu, Chaochao Lu, Qiaosheng Zhang
Machine Learning · Cross-listed from stat.ML
Measuring and Decomposing Mode Separation via the Canonical Diffusion
Shaul Tolkovsky, Ori Meidler and Or Zuk
Mode separation, namely how sharply a distribution fragments into barrier-separated clusters, is a fundamental
geometric property of densities, difficult to quantify in high dimensions. Modes are often the meaningful alternatives in the data, such as the conformations of a molecule or the distinct semantic types among a generative model's outputs. Mode separation is distinct from dispersion, which is what standard label-free tools capture: differential
entropy rises with spread regardless of fragmentation, and PCA orders directions by variance regardless of barriers. Mutual information does capture it, but only given the latent component labels, which are usually unavailable. We measure mode separation through a single
stochastic process intrinsic to the density, namely the unique reversible
diffusion with $f$ as its stationary distribution and a constant scalar
diffusion coefficient. From its autocovariance
matrix we extract two readouts. The first, SSA (Sum of Squared Autocorrelations), is a scalar barrier-sensitive measure. The second, DA (Dominant Autocorrelation directions), gives linear projections ordered by metastability rather than variance. Under an isotropic-Gaussian null, we derive the limiting spectrum of the empirical autocovariance in closed form, generalizing the Marchenko–Pastur law. Its upper edge gives a noise floor that separates metastable directions from sampling noise. Both readouts use only samples and a score function, scaling to high dimensions through pretrained score-based generative models via Tweedie's identity. We apply our framework to three settings: (i) synthetic Gaussian mixtures, where SSA tracks mutual information; (ii) SDXL text-to-image generations, where SSA and DA capture structure that
entropy and PCA miss; and (iii) molecular dynamics of alanine dipeptide, where DA recovers the known slow backbone dihedrals from static samples alone.
Read the full abstract ▾
Machine Learning · Cross-listed from cs.RO
ProcVLM: Learning Procedure-Grounded Progress Rewards for Robotic Manipulation
Youhe Feng, Hansen Shi, Haoyang Li et al. (+7 more)
Long-horizon robotic manipulation requires dense feedback that reflects how a task advances through its procedural stages, not merely whether the final outcome is successful. Existing reward models often rely on trajectory-level success labels or time-based interpolation, which can conflate elapsed time with true task progress and therefore fail to capture unfinished steps, stagnation, and failure states. We present ProcVLM, a progress-aware
vision-
language model that learns procedure-grounded progress as a dense reward signal for manipulation. Rather than deriving progress from terminal outcomes or temporal proxies, ProcVLM grounds progress estimation in procedural structure and intra-stage visual change, and further adopts a
reasoning-before-estimation paradigm that infers the remaining atomic actions before estimating task progress. Specifically, we construct this supervision by synthesizing frame-level subtask-semantic annotations, assigning progress budgets according to subtask structure, and distributing each budget based on intra-subtask visual change. To train ProcVLM at scale, we build a standardized procedural supervision synthesis pipeline and construct ProcCorpus-60M from 30
embodied datasets with 60M annotated frames, from which we derive ProcVQA for procedure-aware pretraining, with progress estimation as the central task alongside action
segmentation and future planning. Experiments on ProcVQA and reward-model
benchmarks show that ProcVLM improves
embodied procedural
reasoning and yields more discriminative trajectory-internal progress estimates than representative baselines, supporting its use as a dense reward model for downstream reward-guided policy optimization. Project page: https://procvlm.github.io/
Read the full abstract ▾
All authors Youhe Feng, Hansen Shi, Haoyang Li, Xinlei Guo, Yang Wang, Chengyang Zhang, Jinkai Zhang, Xiaohan Zhang, Jie Tang, Jing Zhang
Machine Learning
Unlearners Can Lie: Evaluating and Improving Honesty in LLM Unlearning
Renjie Gu, Jiazhen Du, Yihua Zhang et al. (+1 more)
Unlearning in large
language models (LLMs) aims to remove harmful training data while preserving overall utility. However, we find that existing methods often hallucinate, generate abnormal
token sequences, or behave inconsistently, raising safety and trust concerns. According to prior literature on LLM honesty, such behaviors are often associated with dishonesty. This motivates us to investigate the notion of honesty in the context of model unlearning. We propose a formal definition of unlearning honesty, which includes: (1) preserving both utility and honesty on retained knowledge, and (2) ensuring effective forgetting while encouraging the model to acknowledge its limitations and respond consistently to questions related to forgotten knowledge. To systematically evaluate the honesty of unlearning, we introduce a suite of metrics that cover utility, honesty on the retained set, effectiveness of forgetting, rejection rate and refusal stability in Q&A and MCQ settings. Evaluating 9 methods across 3 mainstream families shows that all current methods fail to meet these standards. After experimental and theoretical analyses, we present ReVa, a representation-
alignment procedure that fine-tunes feature-randomized unlearned models to better acknowledge forgotten knowledge. On Q&A tasks from the forget set, ReVa achieves the highest rejection rate after two rounds of interaction, nearly doubling the performance of the second-best method. Remarkably, It also improves honesty on the retained set. We release our data and code at https://github.com/renjiegu.
Read the full abstract ▾
All authors Renjie Gu, Jiazhen Du, Yihua Zhang, Sijia Liu
Machine Learning
Anchoring the Eigengap: Cross-Modal Spectral Stabilization for Sample-Efficient Representation Learning
Nikhil J. Dhinagar, Vidhi Chhatbar, Chirag Jagad et al. (+6 more)
Deep
vision models degrade sharply in low-data regimes, particularly in medical imaging where labeled samples are scarce. We show this arises not merely from overfitting but from a
geometric failure: finite-sample noise corrupts the
embedding covariance, collapsing the eigengap and limiting the number of recoverable signal-bearing modes. We develop a spectral theory of finite-sample representation learning that quantifies the recoverable dimension K(N), the number of eigenmodes that can be stably estimated from N samples. Using perturbation theory and concentration bounds, we show that only modes with
eigenvalues above the noise floor $\|\hat{\Sigma} - \Sigma\|_{\mathrm{op}} \sim \sqrt{D/N}$ are reliable, yielding a truncated Mahalanobis energy that governs classification performance. Under a power-law spectral model, this energy can be approximated by a truncated Riemann
zeta function, linking
eigenvalue decay to data efficiency and AUC. Within this framework,
multimodal learning acts as spectral stabilization:
vision-
language models impose low-rank constraints that suppress noise-dominated directions and preserve the eigengap, increasing K(N) under data scarcity. Across MNIST and multi-disease neuroimaging, we show that
multimodal training maintains more stable modes and improves class separation, even when unimodal models achieve comparable few-shot accuracy. These results identify spectral collapse as a fundamental bottleneck in low-data learning. We use truncated Mahalanobis energy and K(N) to diagnose encoder quality, and introduce
zeta-based spectral filtering as a principled approach to improve data efficiency.
Read the full abstract ▾
All authors Nikhil J. Dhinagar, Vidhi Chhatbar, Chirag Jagad, Pavithra Senthilkumar, Sophia I. Thomopoulos, Mahir H. Khan, Sook-Lei Liew, the ENIGMA-Stroke Recovery Working Group, Paul M. Thompson
Machine Learning · Cross-listed from cs.SD
Omni-DeepSearch: A Benchmark for Audio-Driven Omni-Modal Deep Search
Tao Yu, yiming ding, Shenghua Chai et al. (+16 more)
Current omni-modal
benchmarks mainly evaluate models under settings where multiple modalities are provided simultaneously, while the ability to start from audio alone and actively search for cross-modal evidence remains underexplored. In this paper, we introduce Omni-DeepSearch, a
benchmark for audio-driven omni-modal deep search. Given one or more audio clips and a related question, models must infer useful clues from audio, invoke text, image, and
video search tools, and perform multi-hop
reasoning to produce a short, objective, and verifiable answer. Omni-DeepSearch contains 640 samples across 15 fine-grained categories, covering four
retrieval target modalities and four audio content types. A multi-stage filtering pipeline ensures audio dependence,
retrieval necessity, visual modality necessity, and answer uniqueness. Experiments on recent closed-source and open-source omni-modal models show that this task remains highly challenging: the strongest evaluated model, Gemini-3-Pro, achieves only 43.44% average accuracy. Further analyses illustrate key bottlenecks in audio entity inference, query formulation, tool-use reliability, multi-hop
retrieval, and cross-modal verification. These results highlight audio-driven omni-modal deep search as an important and underexplored direction for future
multimodal agents.
Read the full abstract ▾
All authors Tao Yu, yiming ding, Shenghua Chai, Minghui Zhang, Zhongtian Luo, Xinming Wang, Xinlong Chen, Zhaolu Kang, Junhao Gong, Yuxuan Zhou, Haopeng Jin, Zhiqing Cui, Jiabing Yang, YiFan Zhang, Hongzhu Yi, Zheqi He, Xi Yang, Yan Huang, Liang Wang
Machine Learning · Cross-listed from cs.MA
Beyond the All-in-One Agent: Benchmarking Role-Specialized Multi-Agent Collaboration in Enterprise Workflows
Tao Yu, Hao Wang, Changyu Li et al. (+15 more)
Large
language model (LLM)
agents are increasingly expected to operate in enterprise environments, where work is distributed across specialized roles, permission-controlled systems, and cross-departmental procedures. However, existing enterprise
benchmarks largely evaluate single
agents with broad tool access, while existing multi-
agent benchmarks rarely capture realistic enterprise constraints such as role specialization, access control, stateful business systems, and policy-based approvals. We introduce EntCollabBench, a
benchmark for evaluating enterprise multi-
agent collaboration. EntCollabBench simulates a permission-isolated organization with 11 role-specialized
agents across six departments and contains two evaluation subsets: a Workflow subset, where
agents collaboratively modify enterprise system states, and an Approval subset, where
agents make policy-grounded decisions. Evaluation is based on execution traces, database state verification, and deterministic policy adjudication rather than natural-
language response judging. Experiments with representative LLM
agents show that current models still struggle with end-to-end enterprise collaboration, especially in delegation, context transfer, parameter grounding, workflow closure, and decision commitment. EntCollabBench provides a reproducible testbed for measuring and improving
agent systems intended for realistic organizational environments.
Read the full abstract ▾
All authors Tao Yu, Hao Wang, Changyu Li, Shenghua Chai, Minghui Zhang, Zhongtian Luo, Yuxuan Zhou, Haopeng Jin, Zhaolu Kang, Jiabing Yang, YiFan Zhang, Xinming Wang, Hongzhu Yi, Zheqi He, Jing-Shu Zheng, Xi Yang, Yan Huang, Liang Wang
Machine Learning
FedGMI: Generative Model-Driven Federated Learning for Probabilistic Mixture Inference
Qijun Hou, Yuchen Shi, Pingyi Fan et al. (+1 more)
Federated Learning (FL) facilitates collaborative model training across decentralized clients while preserving data
privacy by avoiding raw data exchange. Despite its potential, FL performance is often compromised by data heterogeneity across clients. To address this, Clustered
Federated Learning (CFL) groups clients with similar data distributions to improve model performance, but constrained by intra-cluster heterogeneity. Conversely, Personalized
Federated Learning (PFL) tailors models to individual clients, but usually neglects the underlying structural similarities among clients. In this work, we investigate a probabilistic mixture (PM) scenario, where each client's local data distribution is modeled as a convex combination of several shared inherent distributions. To effectively model this structure, we propose FedGMI, a framework that utilizes Variational Autoencoders (VAEs) as generative density
estimators to represent these inherent distributions and infer the mixture components of clients' local data distributions. This approach enables structured personalization without sacrificing the benefits of collaborative learning. Extensive experiments demonstrate that FedGMI effectively characterizes and discriminate the inherent distributions, as well as accurately estimates mixture proportions. Furthermore, FedGMI maintains robust performance even under communication cost constraints.
Read the full abstract ▾
All authors Qijun Hou, Yuchen Shi, Pingyi Fan, Khaled B. Letaief
Machine Learning
MDL-GBG: A Non-parametric and Interpretable Granular-Ball Generation Method for Clustering
Zeqiang Xian, Caihui Liu, Yong Zhang et al. (+3 more)
Existing granular-ball generation methods are still mainly driven by handcrafted quality measures and heuristic splitting or stopping criteria, which may weaken the transparency of local generation decisions in clustering. To address this issue, this paper proposes Minimum Description Length based Granular-Ball Generation (MDL-GBG), a non-parametric and interpretable granular-ball generation method for clustering. MDL-GBG reformulates granular-ball generation as a local model selection problem under the Minimum Description Length principle. For each granular ball, three candidate explanations are compared, namely a single-ball model, a two-ball model, and a core-ball-residual model, and the model with the shortest description length is selected. In this way, ball retention, splitting, and residual peeling are unified within a common coding-theoretic framework. A residual reassignment mechanism is further introduced to re-evaluate peeled-off boundary samples after stable granular balls are formed. Experiments on 20 UCI
datasets show that the stable granular balls generated by MDL-GBG provide an effective upstream representation for clustering. In particular, MDL-GBG+AC achieves the highest average ARI, ACC, and NMI values among the compared methods, while the Friedman-Nemenyi analysis further supports its favorable average ranking. These results indicate that MDL-GBG offers a principled and interpretable alternative to heuristic granular-ball generation strategies.
Read the full abstract ▾
All authors Zeqiang Xian, Caihui Liu, Yong Zhang, Wenjing Qiu, Duoqian Miao, Witold Pedrycz
Machine Learning
LAQuant: A Simple Overhead-free Large Reasoning Model Quantization by Layer-wise Lookahead Loss
Euntae Choi, Sumin Song and Sungjoo Yoo
Large
reasoning models (LRMs) reach competition-level math and coding accuracy via long autoregressive decoding, making per-
token decoding cost a primary deployment concern. Weight quantization is the standard tool for acceleration, but representative recipes – including state-of-the-art end-to-end (E2E) QAT – lose accuracy on long-decoding
reasoning benchmarks despite preserving perplexity and short-decode accuracy. Through a systematic gradient-direction analysis, we identify two factors driving this gap: (i) KV-cache fidelity preservation under the QAT loss, which E2E supervision attenuates via the softmax Fisher metric; and (ii) Hessian-subspace
alignment between calibration data and the deployment distribution. We propose LookAhead Quantization (LAQuant), a layer-wise weight-only QAT method that addresses both factors without online-transform overhead by combining
reasoning-domain calibration with a one-layer lookahead loss whose implicit cross-layer co-adaptation preserves the next-layer residual stream. For Qwen3-4B under W3G128 quantization, LAQuant improves AIME25 Pass@1 over ParoQuant by 15.11pp (1.93pp over ParoQuant++ at matched calibration) while achieving a 3.42x decoding speedup over FP16 on RTX A6000, compared with ParoQuant's 3.01x.
Read the full abstract ▾
Machine Learning
Communicating Sound Through Natural Language
Emanuele Rossi and Emanuele Rodolà
Natural
language is widely used to describe,
prompt, and control audio systems, but rarely serves as the representation carrying audio itself. We introduce lexical acoustic coding (LAC), a framework in which pre-trained LLM sender and receiver
agents transmit sound through natural
language. Under fixed system
prompts, the
agents write their own analysis and synthesis code, communicating only through a lexical sentence, shared vocabulary, and optional symbolic music structure. The sender analyzes an input waveform into interpretable, non-learned acoustic descriptors, quantizes each with a feature-specific interval vocabulary, and verbalizes the lexical code as English. The receiver parses the sentence back into lexical-acoustic constraints and renders a waveform through closed-loop refinement. The transmitted text serves as both a rich caption and as the transport representation itself. We frame LAC as a finite-rate lossy quantizer, exposing trade-offs between vocabulary size, rate, and fidelity. Experiments on short sounds and symbolic music transfer show that plain text preserves measurable acoustic structure while remaining interpretable, editable, and native to LLM-mediated communication.
Read the full abstract ▾
Machine Learning
The Wristband Gaussian Loss: Deterministic, Composable Latents via a Sphere-Interval Decomposition
Mikhail Parakhin, André M. Carvalho and Patrick Haluptzok
We present the Wristband Gaussian Loss, a deterministic batch loss for Gaussianizing point
embeddings without sampling, KL terms, or iterative transport. Each $x \in \mathbb{R}^d$ is mapped to a direction $u = x/\|x\|$ and a CDF-transformed radius $t = F_{\chi^2_d}(\|x\|^2)$ on the wristband $S^{d-1} \times [0,1]$. We prove (and machine-verify in Lean~4) that for $d \ge 2$ the pushforward wristband map equals $\sigma_{d-1} \otimes \mathrm{Unif}[0,1]$ iff the source is $\mathcal{N}(0, I_d)$, and that the Neumann-reflected wristband repulsion energy is uniquely minimized at the uniform target. We compute this reflected-kernel objective in two ways: a nearest three-image pairwise truncation at $O(N^2 d)$, and a spectral Neumann path joining angular and radial Mercer modes (spherical-harmonic and cosine) at $O(N d K)$, with empirically matched gradients. A 1D Wasserstein radial term and a moment penalty serve as finite-sample accelerators with the same optimum, and Monte-Carlo null calibration turns the components into a single standardized statistic. We evaluate direct point-cloud Gaussianization with a calibrated barycentric $W_2$ score: a deterministic Gaussian reference batch is built by recursive Hungarian averaging, with each method reported as a $z$-score against same-size Gaussian batches. On the axis-uniform X
benchmark, Wristband is competitive in 2D and gives the best 10D score. On a harder radial–angular-copula impostor whose Gaussian radial and angular marginals are correct but dependent, Wristband gives the best 10D and 128D scores. Coupled with learnable-key Euclidean
attention and exact invertible flows, the resulting Deterministic Gaussian Autoencoder delivers a Gaussian-latent interface for counterfactual sampling with independent factors and a context/residual construction for dependent factors.
Read the full abstract ▾
Machine Learning
The Global Empirical NTK: Self-Referential Bias and Dimensionality of Gradient Descent Learning
James Hazelden, Laura Driscoll, Eli Shlizerman et al. (+1 more)
In training a neural network with gradient descent (GD), each iteration induces a linear operator that governs the first-order updates to a model's internal state variables. We call this operator the Global Empirical Neural Tangent Kernel (GeNTK). In finite-width networks, GeNTK is typically intractable to form, leading prior work to focus on restrictive settings, such as tracking only outputs or infinite-width limits. Here, we bridge this gap by studying the structure of GeNTK across many models. Formulating the model state as the solution to a global implicit constraint, we derive GeNTK as a product of two operators: K, accounting for immediate parameter-to-state interactions, and P, describing internal state-to-state dependencies. For a broad class of "weight-based models", such as RNNs and
transformers, we prove a universal Kronecker-core
theorem showing that K admits an exact, forward-pass-computable form given by the Gram
matrix of weight-site variables. This core gives a measurable state-dependent restriction on Euclidean GD, termed "self-referential bias": current activity filters which adjoint directions can drive a first-order update. For recurrent models (GRUs and RNNs), we examine the spectrum of GeNTK and formalize notions of low-rank, biased learning in terms of reduced spatial or temporal ranks of the full operator. Practically, this explains why some RNN initializations yield better learning, while some plateau, by biasing the GeNTK and thus restricting learning to a small domain of learnable tasks. To demonstrate computational viability, we validate the main
theorems in a self-
attention transformer and a pretrained Pythia
transformer language model with 410 million parameters. Overall, our results expose tractable local rank and mode-selective biases in the finite-width GD geometry and open the door to principled study of the GeNTK, a bridge between theory and practice.
Read the full abstract ▾
All authors James Hazelden, Laura Driscoll, Eli Shlizerman, Eric Shea-Brown
Machine Learning · Cross-listed from cs.GR
MeshFIM: Local Low-Poly Mesh Editing via Fill-in-the-Middle Autoregressive Generation
Dingdong Yang, Jian Liu, Biwen Lei et al. (+6 more)
Autoregressive (AR) models can generate high-quality low-poly meshes from point clouds, but they still operate in an all-or-nothing manner: when a local region is unsatisfactory, the entire mesh must be regenerated, wasting computation and destroying satisfactory mesh structure elsewhere. We introduce MeshFIM, a Fill-in-the-Middle (FIM) framework that regenerates a target region of a low-poly mesh conditioned on the surrounding context. MeshFIM addresses three mesh-specific challenges: enforcing exact attachment along the exposed boundary, preserving topological order in the context, and suppressing overflow beyond the intended region. It does so with five complementary design choices: boundary vertex markers, context positional
embeddings, expanded context width, context augmentation, and a low-poly geometry encoder whose gated subtraction mechanism focuses generation on the missing region by leveraging the difference between the reference surface and the existing mesh. Detailed ablation studies are presented to show the effectiveness of every introduced component. Based on MeshFIM, we demonstrate two applications: interactive brush-based editing and automatic defect repair on low-poly mesh (see Figure 1). Last but not least, experiments show that MeshFIM outperforms a range of baselines in mesh refinement, mesh repair and whole mesh generation plus stitch-back scheme.
Read the full abstract ▾
All authors Dingdong Yang, Jian Liu, Biwen Lei, Haohan Weng, Zhuo Chen, Song Guo, Hao Richard Zhang, Ali Mahdavi Amiri, Chunchao Guo
Machine Learning
Causal Dimensionality of Transformer Representations: Measurement, Scaling, and Layer Structure
Nilesh Sarkar and Dawar Jyoti Deka
Sparse autoencoders (SAEs) decompose
transformer residual streams into interpretable feature dictionaries, yet the relationship between SAE width and causal influence on model output has not been systematically characterised. We introduce causal dimensionality kappa(L, M, T), defined as the effective rank of the expected Jacobian outer product at layer L, and show it can be estimated via the SAE width sweep paired with attribution patching. Across seven SAE widths from 16,384 to 1,048,576 features on Gemma-2-2B layer 12, representational capacity grows 15.6x while causal capacity grows only 4.35x: a robust separation we term the representational-causal wedge. A saturating fit yields kappa-hat approximately 1,990 with kappa-hat / d_model = 0.86 and participation-ratio lower bound kappa_PR approximately 280. Crucially, kappa is invariant to model scaling: Gemma-2-9B and Gemma-2-2B yield identical N_causal = 328 at the same SAE width despite a 3.46x parameter increase (the count is forced to 2% of SAE width by calibration; the substantive empirical claim is shape invariance of the AtP score distribution under matched seq=512 conditions). Across eight network depths kappa is constant while the absolute attribution threshold drops 20x from layer 1 to layer 23. Five controls (architecture invariance, threshold robustness,
geometric privilege, synthetic ground-truth recovery, and a four-
cell encoder/decoder ablation) pin down what kappa measures and what it does not. Our findings establish kappa as a measurable, model-intrinsic property of
transformer layers: sub-linearly recoverable by SAE width, invariant to model scaling, and structured across network depth.
Read the full abstract ▾
Machine Learning
SlimQwen: Exploring the Pruning and Distillation in Large MoE Model Pre-training
Shengkun Tang, Zekun Wang, Bo Zheng et al. (+7 more)
Structured
pruning and knowledge
distillation (KD) are typical techniques for compressing large
language models, but it remains unclear how they should be applied at pretraining scale, especially to recent mixture-of-experts (MoE) models. In this work, we systematically study MoE compression in large-scale pretraining, focusing on three key questions: whether
pruning provides a better initialization than training from scratch, how expert compression choices affect the final model after continued training, and which training strategy is most effective. We have the following findings: First, across depth, width, and expert compression,
pruning a pretrained MoE consistently outperforms training the target architecture from scratch under the same training budget. Second, different one-shot expert compression methods converge to similar final performance after large-scale continual pretraining. Motivated by this, we introduce a simple partial-preservation expert merging strategy that improves downstream performance across most
benchmarks. Third, combining KD with the
language modeling loss outperforms KD alone, particularly on knowledge-intensive tasks. We further propose multi-
token prediction (MTP)
distillation, which yields consistent gains. Finally, given the same training
tokens, progressive
pruning schedules outperform one-shot compression, suggesting that gradual architecture transitions lead to better optimization trajectories. Putting it all together, we compress Qwen3-Next-80A3B to a 23A2B model that retains competitive performance. These results offer practical guidance for efficient MoE compression at scale.
Read the full abstract ▾
All authors Shengkun Tang, Zekun Wang, Bo Zheng, Liangyu Wang, Rui Men, Siqi Zhang, Xiulong Yuan, Zihan Qiu, Zhiqiang Shen, Dayiheng Liu
Machine Learning
ErrOPD: When Does On-Policy Distillation Transfer Teacher Errors?
Xin Li, Hao Jiang, Annan Wang et al. (+2 more)
A distilled student can copy its teacher's mistakes along with its skills. On-policy
distillation (OPD), used in recent post-training recipes, has the teacher grade every
token of the student's own outputs. Which teacher errors does it pass on? We build ErrOPD, a testbed whose teachers are fine-tuned to close a JSON list one item early, an error their students almost never make. The answer depends on the KL direction: across four model families, forward-KL OPD transmits the error within the training budget (+13-23 points), while reverse-KL OPD at our reference learning rate adds under 0.05 points. A duplicated ID and, where the decision arises, two SQL errors show the same split. The reverse-KL update scales with the student's own probability of a
token, and an intervention tests this: giving reverse KL forward KL's gradient magnitude at the error decision alone makes it transmit on two model pairs. Reverse-KL OPD can delay such errors rather than block them: some runs adopt the error with longer training, and Qwen3 students that already make it, even rarely, adopt it within the budget. Because greedy evaluation on Qwen3 misses the inherited error, distilled students should also be audited on sampled outputs.
Read the full abstract ▾
All authors Xin Li, Hao Jiang, Annan Wang, Yichi Zhang, Chau Yuen
Machine Learning
FILO: A Unified Fisher-Geometric Framework for Low-Rank Adaptation
Ziyun Liu, Fengmiao Bian and Jian-Feng Cai
Low-Rank Adaptation (LoRA) reparameterizes a weight
matrix $W$ as a product of two low-rank factors, but the Jacobian $J_G$ of the generator mapping the factors to the weight
matrix is rank-deficient, so the factor-space preconditioner $J_G^* F_t J_G$ induced by any $W$-space preconditioner $F_t$ is singular, and consequently the standard chain rule cannot be uniquely inverted to map a preconditioned $W$-space direction back to a factor-space update. Therefore, we introduce FILO, a unified Fisher-
geometric framework for low-rank adaptation that enables factor updates without directly inverting this singular operator and casts existing LoRA optimizers by their factor-space inversion strategy and $W$-space preconditioner $F_t$. Existing methods occupy four families along these axes: factor-space adaptive updates, block-diagonal surrogates for $J_G^* J_G$, Frobenius-residual pseudoinverse methods, and Riemannian
manifold methods. Within this design space, a gradient-statistics-aware $F_t$ paired with a closed-form factor-space solve at $O((m+n)r)$
memory remains underexplored. We instantiate FILO as FILO-Adafactor by adopting the Adafactor diagonal Kronecker preconditioner $H_t$ on $W$ and selecting from the resulting factor-space solution family the element minimizing an $H_t$-weighted imbalance between the two factor contributions. By construction, the induced weight update is the closest LoRA approximation to the preconditioned $W$-space direction under the $H_t$-weighted norm. Across GPT-2 on E2E, Mistral-7B and Qwen2-7B on GLUE, ARC, and GSM8K, and
diffusion-model personalization, FILO-Adafactor and its momentum variant are competitive with or improve over a representative set of LoRA optimizers while keeping peak GPU
memory at the LoRA optimizer level.
Read the full abstract ▾
Machine Learning
Generative Actor-Critic with Soft Bridge Policies
Ke He, Le He, Shunpu Tang et al. (+2 more)
Expressive generative policies such as
diffusion and flow models are appealing for maximum-
entropy (MaxEnt) online reinforcement learning because of their ability to model
multimodal and highly non-Gaussian action distributions. However, training effective soft generative policies faces two obstacles that often arise together. First, marginal action densities are often unavailable, so existing methods typically rely on
entropy bounds, heuristic proxies or approximations. Second, iterative shared-parameter samplers raise inference cost and require backpropagation through time over repeated network evaluations, increasing
memory cost and potentially destabilizing policy optimization. These obstacles motivate us to seek a generative actor-critic method that exposes a tractable soft objective while generating each action in just one network evaluation. To this end, we propose soft generative actor-critic (SoftGAC), whose actor is a single
stochastic feedforward network with internal Gaussian bridge layers. This explicit bridge structure allows us to lift the soft objective to path space, yielding an analytical path-KL regularizer against a high-
entropy reference process for both actor improvement and the soft Bellman bootstrap, without estimating the endpoint
entropy. Crucially, the actor generates each action in only one network evaluation under a total parameter budget comparable to strong baselines, substantially reducing inference cost relative to the existing iterative
diffusion and flow policies. It also avoids long unrolled backpropagation through time. Across challenging continuous-control
benchmarks, our method achieves competitive or superior returns compared with strong
diffusion and flow policies, demonstrating a clear advantage in the compute-return tradeoff.
Read the full abstract ▾
All authors Ke He, Le He, Shunpu Tang, Yafei Wang, Lisheng Fan
Machine Learning · Cross-listed from cs.RO
Latent Geometry Beyond Search: Amortizing Planning in World Models
Hoang Nguyen, Xiaohao Xu and Xiaonan Huang
Modern
vision-based world models can represent observations as compact yet expressive latent
manifolds, but fast goal-oriented planning in these spaces remains challenging. This raises a central question: when does a learned representation simplify control, rather than merely enabling prediction? We study this question in a pretrained LeWorldModel, whose latent geometry is regularized for smoothness and uniformity. Our key insight is that, under such geometry, planning can be amortized into a latent inverse-dynamics mapping instead of requiring online search. We therefore replace iterative planning with a lightweight Goal-Conditioned Inverse Dynamics Model (GC-IDM) that maps the current latent state, goal latent state, and remaining horizon directly to the next action. Empirically, across four
benchmark environments spanning
navigation, contact-rich manipulation, and continuous control, our controller matches or exceeds CEM in seven of eight environment-protocol settings while reducing per-decision cost by 100-130x. A broader sweep over test-time planners (CEM, MPPI, iCEM, and gradient-based methods) shows that this result is not specific to a particular optimizer. These findings suggest that much of the structure recovered by test-time planning is already locally encoded in the latent representation. More broadly, our results indicate that sufficiently structured latent spaces can shift part of the planning burden from online optimization to learned inference. Our code is publicly available at https://github.com/hdnndh/Latent-Geometry-Beyond-Search-Amortizing-Planning-in-World-Models .
Read the full abstract ▾
Machine Learning · Cross-listed from cs.PF
Choosing a JPEG Decoder for PyTorch DataLoaders: Workload-Specific Throughput on Four CPUs
Vladimir Iglovikov and Dmitry Kosarevsky
A JPEG decoder
benchmark can combine worker counts, CPUs, and
datasets in one large result
matrix. We simplify that comparison by fixing a PyTorch DataLoader at eight workers and asking one question: how much faster is each decoder than Pillow on the same CPU and JPEG workload? We
benchmark 12 Python decoders on four 16-vCPU Google Cloud platforms and two workloads from the Forchheim Image Database. The first contains 324 large camera originals. The second contains 1,668 originals and copies processed by social media services. Every decoder-CPU-workload setting has five isolated repetitions. JPEG bytes reside in
memory, and each timed decode returns a complete three-channel RGB array. The workload changes the practical result. On camera originals, the
geometric mean across CPUs ranges from 1.00 to 1.09 times Pillow. On the mixed workload, simplejpeg, imagecodecs, and turbojpeg reach 1.39 - 1.40 Pillow. The exact leader still changes by CPU. We therefore recommend comparing decoders on representative JPEGs, while using the fixed eight-worker results as a compact starting point. Full worker curves and absolute throughput tables are in the appendix.
Read the full abstract ▾
Machine Learning
Classification-Head Bias in Class-Level Machine Unlearning: Diagnosis, Mitigation, and Evaluation
Weidong Zheng, Kongyang Chen, Yuanwei Guo et al. (+1 more)
Class-level machine unlearning aims to remove the influence of specified classes while preserving model utility on retained classes. Existing methods are commonly evaluated by retain-set accuracy, forget-set accuracy, and unlearning time, but these metrics provide limited insight into how forgetting is achieved internally. In this paper, we reveal a bias-dominated shortcut in class-level unlearning: the prediction of forgotten classes can be suppressed by decreasing the corresponding bias terms in the final classification head. We first analyze the gradient dynamics of classification-head biases under softmax cross-
entropy training, explaining why retain-set-only optimization tends to reduce the biases of absent classes. Based on this observation, we introduce BiasShift as a diagnostic baseline, showing that simple bias manipulation can satisfy conventional unlearning metrics while leaving abnormal bias patterns that reveal forgotten labels. To mitigate excessive forgotten-class bias suppression, we propose two bias-aware mechanisms, namely Two-Stage Bias Gradient Reversal Mechanism (TS-BGRM) and Lower-Bound Hinge Regularization (LB-HR). We further introduce three bias-oriented metrics, including Bias Stability Coefficient (BSC), Median Bias Gap (MBG), and Minimal Bias Score (MBS), to quantify bias dependence and potential leakage. Experiments on CIFAR-10, CIFAR-100, and Tiny-ImageNet demonstrate that the proposed methods maintain competitive unlearning performance while producing more stable bias distributions. We have released our code at {https://github.com/zwd2024/Beyond-the-Shadow-of-Bias-From-Classification-Head-Bias-to-Parameter-Redistribution}.
Read the full abstract ▾
All authors Weidong Zheng, Kongyang Chen, Yuanwei Guo, Yatie Xiao
Machine Learning · Cross-listed from cs.CV
Control Your View: High-Resolution Global Semantic Manipulation in Learned Image Compression
Jiaming Liang, Chi-Man Pun, Weisi Lin et al. (+1 more)
Learned image compression (LIC) integrates deep neural networks (DNNs) to map high-dimensional images into compact latent representations, reducing redundancy and achieving superior rate-distortion (RD) performance in benign settings. Unfortunately, due to inherent vulnerabilities in DNNs, LIC systems are susceptible to
adversarial perturbations that lead to downstream deterioration, compression rate degradation, untargeted distortion, and both local semantic manipulation (LSM) and low-resolution ($3\times28\times28$) global semantic manipulation (GSM). However, high-resolution GSM remains unexplored due to its intractability. Notably, the existing project gradient descent (PGD) method achieves near-perfect white-box attacks for classification,
segmentation, and other tasks, yet fails to generalize to high-resolution GSM. Our theoretical and empirical analyses reveal that well-performing GSM drives
adversarial examples from the Identity Region to the Amplification Region through the Lazying-Oscillating-Refining stages. General $\ell_{\infty}$-bounded attacks fail on high-resolution GSM because their step-size schedules cannot accommodate both the Oscillating and Refining stages. Based on this, we propose the Periodic
Geometric Decay schedule that enables $\ell_{\infty}$-bounded high-resolution GSM. To verify our approach, we integrate it with PGD, yielding a minimal variant, PGD$^{2}$-GSM. Extensive experiments on the Kodak $(3\times768\times512)$ demonstrate that our PGD$^{2}$-GSM is the first to stably achieve high-resolution GSM, thereby exposing a novel threat to LIC systems. Code is available at https://github.com/chinaliangjiaming/PGD2-GSM.
Read the full abstract ▾
All authors Jiaming Liang, Chi-Man Pun, Weisi Lin, Greta Seng Peng Mok
Machine Learning · Cross-listed from cs.AI
Bias by Necessity: Impossibility Theorems for Sequential Processing with Convergent AI and Human Validation
Jikun Wu, Dongxin Guo and Siu-Ming Yiu
Are certain cognitive biases mathematically inevitable consequences of sequential information processing? We prove that primacy effects, anchoring, and order-dependence are architecturally necessary in autoregressive
language models due to causal masking constraints. Our three impossibility
theorems establish: (1) primacy bias arises from asymmetric
attention accumulation; (2) anchoring emerges from sequential conditioning with provable information bounds; and (3) exact debiasing by permutation marginalization requires factorial-time computation, with Monte Carlo approximation feasible at constant per-tolerance overhead. We validate these bounds across 12 frontier LLMs ($R^2 = 0.89$; $\Delta$BIC $= 16.6$ vs. next-best alternative). We then derive quantitative predictions from the framework and test them in two pre-registered human experiments ($N = 464$ analyzed). Study 1 confirms anchor position modulates anchoring magnitude ($d = 0.52$, BF$_{10} = 847$). Study 2 shows working
memory load amplifies primacy bias ($d = 0.41$, BF$_{10} = 156$), with WM capacity predicting bias reduction ($r = -.38$). These convergent findings reframe cognitive biases as resource-rational responses to sequential processing.
Read the full abstract ▾