Machine Learning
Path Integration and Object-Location Binding Emerge in an Action-Conditioned Predictive Sequence Network
Linda Ariel Ventura, Victoria Bosch, Tim C Kietzmann et al. (+1 more)
Adaptive cognition requires structured internal models of objects and their relations. Predictive neural networks are often proposed to learn such world models, but how these are instantiated and how they support prediction remain unclear. We investigate this in a minimal in-silico setting. A recurrent neural network samples
tokens sequentially from 2D continuous
token scenes and is trained to predict the upcoming
token from the current input and a saccade-like displacement. On novel scenes, prediction accuracy improves across the sequence, indicating in-context learning. Decoding analyses reveal path integration and dynamic binding of
token identity to position. Interventional analyses show that new bindings can be learned late in sequence and that out-of-distribution bindings can be learned as well. Together, these findings show how structured representations relying on flexible binding emerge to support prediction, offering a mechanistic account of sequential world modeling relevant to cognitive science.
Read the full abstract ▾
All authors Linda Ariel Ventura, Victoria Bosch, Tim C Kietzmann, Sushrut Thorat
Machine Learning
DeepDFA: Injecting Temporal Logic in Deep Learning for Sequential Subsymbolic Applications
Elena Umili, Francesco Argenziano and Roberto Capobianco
Integrating logical knowledge into deep neural network training is still a hard challenge, especially for sequential or temporally extended domains involving subsymbolic observations. To address this problem, we propose DeepDFA, a neurosymbolic framework that integrates high-level temporal logic - expressed as Deterministic Finite Automata (DFA) or Moore Machines - into neural architectures. DeepDFA models temporal rules as continuous, differentiable layers, enabling symbolic knowledge injection into subsymbolic domains. We demonstrate how DeepDFA can be used in two key settings: (i) static image sequence classification, and (ii) policy learning in interactive non-Markovian environments. Across extensive experiments, DeepDFA outperforms traditional deep learning models (e.g., LSTMs, GRUs,
Transformers) and novel neuro-symbolic systems, achieving state-of-the-art results in temporal knowledge integration. These results highlight the potential of DeepDFA to bridge subsymbolic learning and symbolic
reasoning in sequential tasks.
Read the full abstract ▾
Machine Learning · Cross-listed from cs.CL
Self-Verification Dilemma: Experience-Driven Suppression of Overused Checking in LLM Reasoning
Quanyu Long, Kai Jie Jiang, Jianda Chen et al. (+3 more)
Large
Reasoning Models (LRMs) achieve strong performance by generating long
reasoning traces with reflection. Through a large-scale empirical analysis, we find that a substantial fraction of reflective steps consist of self-verification (recheck) that repeatedly confirm intermediate results. These rechecks occur frequently across models and
benchmarks, yet the vast majority are confirmatory rather than corrective, rarely identifying errors and altering
reasoning outcomes. This reveals a mismatch between how often self-verification is activated and how often it is actually useful. Motivated by this, we propose a novel, experience-driven test-time framework that reduces the overused verification. Our method detects the activation of recheck behavior, consults an offline experience pool of past verification outcomes, and estimates whether a recheck is likely unnecessary via efficient
retrieval. When historical experience suggests unnecessary, a suppression signal redirects the model to proceed. Across multiple model and
benchmarks, our approach reduces
token usage up to 20.3% while maintaining the accuracy, and in some
datasets even yields accuracy improvements.
Read the full abstract ▾
All authors Quanyu Long, Kai Jie Jiang, Jianda Chen, Xu Guo, Leilei Gan, Wenya Wang
Machine Learning
ScDiVa: Masked Discrete Diffusion for Joint Modeling of Single-Cell Identity and Expression
Mingxuan Wang, Cheng Chen, Gaoyang Jiang et al. (+4 more)
Single-
cell RNA-seq profiles are high-dimensional, sparse, and unordered, causing autoregressive generation to impose an artificial ordering bias and suffer from error accumulation. To address this, we propose scDiVa, a masked discrete
diffusion foundation model that aligns generation with the dropout-like corruption process by defining a continuous-time forward masking mechanism in
token space. ScDiVa features a bidirectional denoiser that jointly models discrete gene identities and continuous values, utilizing
entropy-normalized serialization and a latent anchor
token to maximize information efficiency and preserve global
cell identity. The model is trained via depth-invariant time sampling and a dual denoising objective to simulate varying sparsity levels while ensuring precise recovery of both identity and magnitude. Pre-trained on 59 million
cells, scDiVa achieves strong transfer performance across major
benchmarks, including batch integration,
cell type annotation, and perturbation response prediction. These results suggest that masked discrete
diffusion serves as a biologically coherent and effective alternative to autoregression.
Read the full abstract ▾
All authors Mingxuan Wang, Cheng Chen, Gaoyang Jiang, Zijia Ren, Chuangxin Zhao, Lu Shi, Yanbiao Ma
Machine Learning
Scaling Continual Learning to 300+ Tasks with Bi-Level Routing Mixture-of-Experts
Meng Lou, Yunxiang Fu and Yizhou Yu
Continual learning, especially class-incremental learning (CIL), on the basis of a pre-trained model (PTM) has garnered substantial research interest in recent years. However, how to effectively learn both discriminative and comprehensive feature representations while maintaining stability and plasticity over very long task sequences remains an open problem. We propose CaRE, a scalable {C}ontinual Le{a}rner with efficient Bi-Level {R}outing Mixture-of-{E}xperts (BR-MoE). The core idea of BR-MoE is a bi-level routing mechanism: a router selection stage that dynamically activates relevant task-specific routers, followed by an expert routing phase that dynamically activates and aggregates experts, aiming to inject discriminative and comprehensive representations into every intermediate network layer. On the other hand, we introduce a challenging
dataset, OmniBenchmark-1K, for CIL performance evaluation on very long task sequences with hundreds of tasks. Extensive experiments show that CaRE demonstrates leading performance across a variety of
datasets and task settings, including commonly used CIL
datasets with classical CIL settings (e.g., 5-20 tasks). To the best of our knowledge, CaRE is the first continual learner that scales to very long task sequences (ranging from 100 to over 300 non-overlapping tasks), while outperforming all baselines by a large margin on such task sequences. We hope that this work will inspire further research into continual learning over extremely long task sequences. Code and
dataset are publicly released at https://github.com/LMMMEng/CaRE.
Read the full abstract ▾
Machine Learning · Cross-listed from cs.AI
IntentRL: Training Proactive User-intent Agents for Open-ended Deep Research via Reinforcement Learning
Haohao Luo, Zexi Li, Yuexiang Xie et al. (+3 more)
Deep Research (DR)
agents extend Large
Language Models (LLMs) beyond parametric knowledge by autonomously retrieving and synthesizing evidence from large web corpora into long-form reports, enabling a long-horizon agentic paradigm. However, unlike real-time conversational assistants, DR is computationally expensive and time-consuming, creating an autonomy-interaction dilemma: high autonomy on ambiguous user queries often leads to prolonged execution with unsatisfactory outcomes. To address this, we propose IntentRL, a framework that trains proactive
agents to clarify latent user intents before starting long-horizon research. To overcome the scarcity of open-ended research data, we introduce a scalable pipeline that expands a few seed samples into high-quality dialogue turns via a shallow-to-deep intent refinement graph. We further adopt a two-stage reinforcement learning (RL) strategy: Stage I applies RL on offline dialogues to efficiently learn general user-interaction behavior, while Stage II uses the trained
agent and a user simulator for online rollouts to strengthen adaptation to diverse user feedback. Extensive experiments show that IntentRL significantly improves both intent hit rate and downstream task performance, outperforming the built-in clarify modules of closed-source DR
agents and proactive LLM baselines.
Read the full abstract ▾
All authors Haohao Luo, Zexi Li, Yuexiang Xie, Wenhao Zhang, Yaliang Li, Ying Shen
Machine Learning
Soft-Radial Projection for Constrained End-to-End Learning
Philipp J. Schneider and Daniel Kuhn
Constrained end-to-end learning trains neural networks whose outputs must satisfy feasibility constraints, such as resource, budget, or operational limits. While projection and optimization layers guarantee constraint satisfaction, boundary-based projections can introduce unfavorable optimization geometry: regions of the unconstrained output space are mapped to the same active face of the feasible set. This makes the resulting map locally rank-deficient, suppressing gradients in constrained directions and potentially impairing optimization - a bottleneck known as gradient saturation. We propose Soft-Radial Projection (SRP), a differentiable reparameterization layer that maps unconstrained network outputs into the relative interior of a convex feasible set by smoothly rescaling rays from a strictly feasible anchor point. Unlike hard radial or orthogonal projections, the resulting map is one-to-one and has a full-rank Jacobian almost everywhere, while guaranteeing strict feasibility. We prove that constrained networks equipped with SRP retain universal approximation guarantees. For polytopes and balls, the layer admits a closed-form forward pass, while for general convex sets, the ray-boundary intersection reduces to a one-dimensional search. Experiments on decision-focused learning
benchmarks demonstrate strong solution quality at low computational cost, with particularly favorable results in portfolio optimization and substantially lower inference latency than differentiable optimization layers in the evaluated settings.
Read the full abstract ▾
Machine Learning
Causal Inference on Networks under Misspecified Exposure Mappings: A Partial Identification Framework
Maresa Schröder, Miruna Oprescu, Stefan Feuerriegel et al. (+1 more)
Estimating treatment effects in networks is challenging, as each potential outcome depends on the treatments of all other nodes in the network. To overcome this difficulty, existing methods typically impose an exposure mapping that compresses the treatment assignments in the network into a low-dimensional summary. However, if this mapping is misspecified, standard
estimators for direct and spillover effects can be severely biased. We propose a novel partial identification framework for causal inference on networks to assess the robustness of treatment effects under misspecifications of the exposure mapping. Specifically, we derive sharp upper and lower bounds on direct and spillover effects under such misspecifications. As such, our framework presents a novel application of causal sensitivity analysis to exposure mappings. We instantiate our framework for three canonical exposure settings widely used in practice: (i) weighted means of the neighborhood treatments, (ii) threshold-based exposure mappings, and (iii) truncated neighborhood interference in the presence of higher-order spillovers. Furthermore, we develop orthogonal
estimators for these bounds and prove that the resulting bound estimates are valid, sharp, and efficient. Our experiments show the bounds remain informative and provide reliable conclusions under misspecification of exposure mappings.
Read the full abstract ▾
All authors Maresa Schröder, Miruna Oprescu, Stefan Feuerriegel, Nathan Kallus
Machine Learning
Beyond Variance: Prompt-Efficient RLVR via Rare-Event Amplification and Bidirectional Pairing
Yujuan Pang, Jiaxin Li, Xin Sheng et al. (+2 more)
Reinforcement learning with verifiable rewards (RLVR) is effective for training large
language models on deterministic outcome
reasoning tasks. Prior work shows RLVR works with few
prompts, but
prompt selection is often based only on training-accuracy variance, leading to unstable optimization directions and weaker transfer. We revisit
prompt selection from a mechanism-level view and argue that an effective minibatch should provide both (i) a reliable positive anchor and (ii) explicit negative learning signals from rare failures. Based on this principle, we propose positive–negative pairing: at each update, we sample a hard-but-solvable $q^{+}$ and an easy-but-brittle
prompt $q^{-}$(high success rate but not perfect), characterized by low and high empirical success rates under multiple rollouts. We further introduce Weighted GRPO, which reweights binary outcomes at the pair level and uses group-normalized advantages to amplify rare successes on $q^{+}$ into sharp positive guidance while turning rare failures on $q^{-}$ into strong negative penalties. This bidirectional signal provides informative learning feedback for both successes and failures, improving sample efficiency without suppressing exploration. On Qwen2.5-Math-7B, a single paired minibatch per update consistently outperforms a GRPO baseline that selects two
prompts via commonly used variance-based selection heuristics: AIME~2025 Pass@8 improves from 16.8 to 22.2, and AMC23 Pass@64 from 94.0 to 97.0, while remaining competitive with large-scale RLVR trained from a pool of 1209 training
prompts. Similar gains are observed on Qwen2.5-Math-7B-Instruct.
Read the full abstract ▾
All authors Yujuan Pang, Jiaxin Li, Xin Sheng, Ran Peng, Yong Ma
Machine Learning · Cross-listed from stat.ML
Score-based diffusion models for severely ill-posed problems in diffuse optical tomography
Fabian Schneider, Meghdoot Mozumder, Konstantin Tamarov et al. (+4 more)
Score-based
diffusion models are a recently developed framework for posterior sampling in
Bayesian inverse problems, enabling high-quality reconstructions in inverse problems by leveraging expressive prior distributions learned from empirical data. Despite their strong empirical performance and growing interest within the machine learning community, their behaviour in realistic, severely ill-posed inverse problems with experimental measurement data remains under-explored. Diffuse optical
tomography (DOT) is an inverse boundary value problem that uses boundary measurements of near-infrared light to recover spatially varying absorption and scattering parameters in biological tissue. The problem is highly ill-posed and particularly sensitive to both measurement noise and modelling errors. We introduce a regularization strategy by constructing a mixed score consisting of a learned component and a model-based component. We show that the resulting mixed score approximates the score of a corresponding mixture distribution locally and in the small
diffusion-time regime, providing a theoretical justification for the approach. We compare four approaches for difference imaging in DOT: a classical model-based method, an approximate score-based
diffusion method (DPS), an exact posterior sampling method (UCoS) and a novel, regularized version of UCoS. We show that both the model-based approach and approximate
diffusion-based sampling degrade significantly in the presence of limited-view geometry and real experimental data, whereas UCoS yields more accurate reconstructions.
Read the full abstract ▾
All authors Fabian Schneider, Meghdoot Mozumder, Konstantin Tamarov, Leila Taghizadeh, Tanja Tarvainen, Tapio Helin, Duc-Lam Duong
Machine Learning · Cross-listed from cs.AI
CRL-VLA: Continual Vision-Language-Action Learning
Qixin Zeng, Shuo Zhang, Hongyin Zhang et al. (+6 more)
Lifelong learning is critical for
embodied agents in open-world environments, where reinforcement learning fine-tuning has emerged as an important paradigm to enable
Vision-
Language-Action (VLA) models to master dexterous manipulation through environmental interaction. Thus, Continual Reinforcement Learning (CRL) is a promising pathway for deploying VLA models in lifelong robotic scenarios, yet balancing stability (retaining old skills) and plasticity (learning new ones) remains a formidable challenge for existing methods. We introduce CRL-VLA, a framework for continual post-training of VLA models with rigorous theoretical bounds. We derive a unified performance bound linking the stability-plasticity trade-off to goal-conditioned advantage magnitude, scaled by policy divergence. CRL-VLA resolves this dilemma via asymmetric regulation: constraining advantage magnitudes on prior tasks while enabling controlled growth on new tasks. This is realized through a simple but effective dual-critic architecture with novel Goal-Conditioned Value Formulation (GCVF), where a frozen critic anchors semantic consistency and a trainable
estimator drives adaptation. Experiments on the LIBERO
benchmark demonstrate that CRL-VLA effectively harmonizes these conflicting objectives, outperforming baselines in both anti-forgetting and forward adaptation.
Read the full abstract ▾
All authors Qixin Zeng, Shuo Zhang, Hongyin Zhang, Renjie Wang, Han Zhao, Libang Zhao, Runze Li, Donglin Wang, Chao Huang
Machine Learning · Cross-listed from cond-mat.mtrl-sci
Acceleration of Atomistic NEGF: Algorithms, Parallelization, and Machine Learning
Mathieu Luisier, Nicolas Vetsch, Alexander Maeder et al. (+8 more)
The Non-
equilibrium Green's function (NEGF) formalism is a particularly powerful method to simulate the
quantum transport properties of nanoscale devices such as transistors, photo-diodes, or
memory cells, in the ballistic limit of transport or in the presence of various scattering sources such as electronphonon, electron-
photon, or even electron-electron interactions. The inclusion of all these mechanisms has been first demonstrated in small systems, composed of a few atoms, before being scaled up to larger structures made of thousands of atoms. Also, the accuracy of the models has kept improving, from empirical to fully ab-initio ones, e.g., density functional theory (DFT). This paper summarizes key (algorithmic) achievements that have allowed us to bring DFT+NEGF simulations closer to the dimensions and functionality of realistic systems. The possibility of leveraging graph neural networks and machine learning to speed up ab-initio device simulations is discussed as well.
Read the full abstract ▾
All authors Mathieu Luisier, Nicolas Vetsch, Alexander Maeder, Vincent Maillou, Anders Winka, Leonard Deuschle, Chen Hao Xia, Manasa Kaniselvan, Marko Mladenovic, Jiang Cao, Alexandros Nikolaos Ziogas
Machine Learning · Cross-listed from cs.AI
DiscoverLLM: From Executing Intents to Discovering Them
Tae Soo Kim, Yoonjoo Lee, Jaesang Yu et al. (+2 more)
To handle ambiguous and open-ended requests, Large
Language Models (LLMs) are increasingly trained to interact with users to surface intents they have not yet expressed (e.g., ask clarification questions). However, users are often ambiguous because they have not yet formed their intents: they must observe and explore outcomes to discover what they want. Simply asking "what kind of tone do you want?" fails when users themselves do not know. We introduce DiscoverLLM, a novel and generalizable framework that trains LLMs to help users form and discover their intents. Central to our approach is a novel user simulator that models cognitive state with a hierarchy of intents that progressively concretize as the model surfaces relevant options – where the degree of concretization serves as a reward signal that models can be trained to optimize. Resulting models learn to collaborate with users by adaptively diverging (i.e., explore options) when intents are unclear, and converging (i.e., refine and implement) when intents concretize. Across proposed interactive
benchmarks in creative writing, technical writing, and SVG drawing, DiscoverLLM achieves over 10% higher task performance while reducing conversation length by up to 40%. In a user study with 75 human participants, DiscoverLLM improved conversation satisfaction and efficiency compared to baselines.
Read the full abstract ▾
All authors Tae Soo Kim, Yoonjoo Lee, Jaesang Yu, John Joon Young Chung, Juho Kim
Machine Learning · Cross-listed from cs.SD
CoCoEmo: Composable and Controllable Human-Like Emotional TTS via Activation Steering
Siyi Wang, Shihong Tan, Siyi Liu et al. (+4 more)
Emotional expression in human
speech is nuanced and compositional, often involving multiple, sometimes conflicting, affective cues that may diverge from linguistic content. In contrast, most expressive text-to-
speech systems enforce a single utterance-level
emotion, collapsing affective diversity and suppressing mixed or text-
emotion-misaligned expression. While activation steering via latent direction vectors offers a promising solution, it remains unclear whether
emotion representations are linearly steerable in TTS, where steering should be applied within hybrid TTS architectures, and how such complex
emotion behaviors should be evaluated. This paper presents the first systematic analysis of activation steering for emotional control in hybrid TTS models, introducing a quantitative, controllable steering framework, and multi-rater evaluation protocols that enable composable mixed-
emotion synthesis and reliable text-
emotion mismatch synthesis. Our results demonstrate, for the first time, that emotional prosody and expressive variability are primarily synthesized by the TTS
language module instead of the flow-matching module, and also provide a lightweight steering approach for generating natural, human-like emotional
speech.
Read the full abstract ▾
All authors Siyi Wang, Shihong Tan, Siyi Liu, Hong Jia, Gongping Huang, James Bailey, Ting Dang
Machine Learning
Most Convolutional Networks Suffer from Small Adversarial Perturbations
Amit Daniely and Idan Mehalel
The existence of
adversarial examples is relatively understood for random fully connected neural networks, but much less so for convolutional neural networks (CNNs). The recent work [Daniely, 2025] establishes that
adversarial examples can be found in CNNs, in some non-optimal distance from the input. We extend over this work and prove that
adversarial examples in random CNNs with input dimension $d$ can be found already in $\ell_2$-distance of order $\lVert x \rVert /\sqrt{d}$ from the input $x$, which is essentially the nearest possible. We also show that such
adversarial small perturbations can be found using a single step of gradient descent. To derive our results we use
Fourier decomposition to efficiently bound the singular values of a random linear convolutional operator, which is the main ingredient of a CNN layer. This bound might be of independent interest.
Read the full abstract ▾
Machine Learning · Cross-listed from quant-ph
Enhancing Quantum Diffusion Models for Complex Image Generation
Jeongbin Jo, Santanam Wishal, Shah Md Khalil Ullah et al. (+2 more)
Quantum generative models offer a novel approach to exploring high-dimensional
Hilbert spaces but face significant challenges in scalability and expressibility when applied to multi-modal distributions. In this study, we explore a Hybrid
Quantum-Classical U-Net architecture integrated with Adaptive Non-Local Observables (ANO) as a potential solution to these hurdles. By compressing classical data into a dense
quantum latent space and utilizing trainable observables, our model aims to extract non-local features that complement classical processing. We also investigate the role of Skip Connections in preserving semantic information during the reverse
diffusion process. Experimental results on the full MNIST
dataset (digits 0-9) demonstrate that the proposed architecture is capable of generating structurally coherent and recognizable images for all digit classes. While hardware constraints still impose limitations on resolution, our findings suggest that hybrid architectures with adaptive measurements provide a feasible pathway for mitigating mode collapse and enhancing generative capabilities in the NISQ era.
Read the full abstract ▾
All authors Jeongbin Jo, Santanam Wishal, Shah Md Khalil Ullah, Shan Zeng, Dikshant Dulal
Machine Learning · Cross-listed from cs.AI
Risk Awareness Injection: Calibrating Vision-Language Models for Safety without Compromising Utility
Mengxuan Wang, Yuxin Chen, Gang Xu et al. (+3 more)
Vision language models (VLMs) extend the
reasoning capabilities of large
language models (LLMs) to cross-modal settings, yet remain highly vulnerable to
multimodal jailbreak attacks. Existing defenses predominantly rely on safety fine-tuning or aggressive
token manipulations, incurring substantial training costs or significantly degrading utility. Recent research shows that LLMs inherently recognize unsafe content in text, and the incorporation of visual inputs in VLMs frequently dilutes risk-related signals. Motivated by this, we propose Risk Awareness Injection (RAI), a lightweight and training-free framework for safety calibration that restores LLM-like risk recognition by amplifying unsafe signals in VLMs. Specifically, RAI constructs an Unsafe Prototype Subspace from
language embeddings and performs targeted modulation on selected high-risk visual
tokens, explicitly activating safety-critical signals within the cross-modal feature space. This modulation restores the model's LLM-like ability to detect unsafe content from visual inputs, while preserving the semantic integrity of original
tokens for cross-modal
reasoning. Extensive experiments across multiple jailbreak and utility
benchmarks demonstrate that RAI substantially reduces attack success rate without compromising task performance.
Read the full abstract ▾
All authors Mengxuan Wang, Yuxin Chen, Gang Xu, Tao He, Hongjie Jiang, Ming Li
Machine Learning
The Label Horizon Paradox: Rethinking Supervision Targets in Financial Forecasting
Chen-Hui Song, Shuoling Liu and Liyuan Chen
While deep learning has revolutionized financial
forecasting through sophisticated architectures, the design of the supervision signal itself is rarely scrutinized. We challenge the canonical assumption that training labels must strictly mirror inference targets, uncovering the Label Horizon Paradox: the optimal supervision signal often deviates from the prediction goal, shifting across intermediate horizons governed by
market dynamics. We theoretically ground this phenomenon in a dynamic signal-noise trade-off, demonstrating that generalization hinges on the competition between marginal signal realization and noise accumulation. To operationalize this insight, we propose a bi-level optimization framework that autonomously identifies the optimal proxy label within a single training run. Extensive experiments on large-scale financial
datasets demonstrate consistent improvements over conventional baselines, thereby opening new avenues for label-centric research in financial
forecasting.
Read the full abstract ▾
Machine Learning · Cross-listed from stat.ML
Improving the Linearized Laplace Approximation via Quadratic Approximations
Pedro Jiménez, Luis A. Ortega, Pablo Morales-Álvarez et al. (+1 more)
Deep neural networks (DNNs) often produce overconfident out-of-distribution predictions, motivating
Bayesian uncertainty quantification. The Linearized Laplace Approximation (LLA) achieves this by linearizing the DNN and applying Laplace inference to the resulting model. Importantly, the linear model is also used for prediction. We argue this linearization in the posterior may degrade fidelity to the true Laplace approximation. To alleviate this problem, without increasing significantly the computational cost, we propose the Quadratic Laplace Approximation (QLA). QLA approximates each second order factor in the approximate Laplace log-posterior using a rank-one factor obtained via efficient power iterations. QLA is expected to yield a posterior precision closer to that of the full Laplace without forming the full Hessian, which is typically intractable. For prediction, QLA also uses the linearized model. Empirically, QLA yields modest yet consistent uncertainty estimation improvements over LLA on five regression
datasets.
Read the full abstract ▾
All authors Pedro Jiménez, Luis A. Ortega, Pablo Morales-Álvarez, Daniel Hernández-Lobato
Machine Learning
On the Entropy Dynamics in Reinforcement Fine-Tuning of Large Language Models
Shumin Wang, Yuexiang Xie, Wenhao Zhang et al. (+4 more)
Entropy serves as a critical metric for measuring the diversity of outputs generated by large
language models (LLMs), providing valuable insights into their exploration capabilities. While recent studies increasingly focus on monitoring and adjusting
entropy to better balance exploration and exploitation in reinforcement fine-tuning (RFT), a principled understanding of
entropy dynamics during this process is yet to be thoroughly investigated. In this paper, we establish a theoretical framework for analyzing the
entropy dynamics during the RFT process, which begins with a discriminant expression that quantifies
entropy change under a single logit update. This foundation enables the derivation of a first-order expression for
entropy change, which can be further extended to the update formula of Group Relative Policy Optimization (GRPO). The corollaries and insights drawn from the theoretical analysis inspire the design of
entropy control methods, and also offer a unified lens for interpreting various
entropy-based methods in existing studies. We provide empirical evidence to support the main conclusions of our analysis and demonstrate the effectiveness of the derived
entropy-discriminator clipping methods. This study yields novel insights into RFT training dynamics, providing theoretical support and practical strategies for optimizing the exploration-exploitation balance during LLM fine-tuning.
Read the full abstract ▾
All authors Shumin Wang, Yuexiang Xie, Wenhao Zhang, Yuchang Sun, Yanxi Chen, Yaliang Li, Yanyong Zhang
Machine Learning · Cross-listed from cs.CV
From Vicious to Virtuous Cycles: Synergistic Representation Learning for Unsupervised Video Object-Centric Learning
Hyun Seok Seong, WonJun Moon and Jae-Pil Heo
Unsupervised object-centric learning models, particularly slot-based architectures, have shown great promise in decomposing complex scenes. However, their reliance on reconstruction-based training creates a fundamental conflict between the sharp, high-frequency
attention maps of the encoder and the spatially consistent but blurry reconstruction maps of the decoder. We identify that this discrepancy gives rise to a vicious cycle: the noisy feature map from the encoder forces the decoder to average over possibilities and produce even blurrier outputs, while the gradient computed from blurry reconstruction maps lacks high-frequency details necessary to supervise encoder features. To break this cycle, we introduce Synergistic Representation Learning (SRL) that establishes a virtuous cycle where the encoder and decoder mutually refine one another. SRL leverages the encoder's sharpness to deblur the semantic boundary within the decoder output, while exploiting the decoder's spatial consistency to denoise the encoder's features. This mutual refinement process is stabilized by a warm-up phase with a slot regularization objective that initially allocates distinct entities per slot. By bridging the representational gap between the encoder and decoder, SRL achieves state-of-the-art results on
video object-centric learning
benchmarks. Codes are available at https://github.com/hynnsk/SRL.
Read the full abstract ▾
Machine Learning
Chain-of-Goals Hierarchical Policy for Long-Horizon Offline Goal-Conditioned RL
Jinwoo Choi, Sang-Hyun Lee and Seung-Woo Seo
Offline goal-conditioned reinforcement learning remains challenging for long-horizon tasks. While hierarchical approaches mitigate this issue by decomposing tasks, most existing methods rely on separate high- and low-level networks and generate only a single intermediate subgoal, leaving several structural limitations in long-horizon decision-making. To address this limitation, we draw inspiration from chain-of-thought
reasoning and propose the Chain-of-Goals Hierarchical Policy (CoGHP), a novel framework that reformulates hierarchical decision-making as autoregressive sequence modeling within a unified architecture. Given a state and a final goal, CoGHP autoregressively generates a sequence of latent subgoals followed by the primitive action, where each latent subgoal acts as a
reasoning step that conditions subsequent predictions. To implement this efficiently, we introduce an MLP-Mixer backbone, which supports cross-
token communication and captures structural relationships among state, goal, latent subgoals, and action. Across challenging
navigation and manipulation
benchmarks, CoGHP consistently outperforms strong offline baselines, demonstrating improved performance on long-horizon tasks. Project page: https://wlsdn9350.github.io/projects/coghp/
Read the full abstract ▾
Machine Learning
An Approximate Ascent Approach To Prove Convergence of PPO
Leif Doering, Daniel Schmidt, Moritz Melcher et al. (+4 more)
Proximal Policy Optimization (PPO) is among the most widely used deep reinforcement learning
algorithms, yet its theoretical foundations remain incomplete. Most importantly, convergence and understanding of fundamental PPO advantages remain widely open. Under standard theory assumptions we show how PPO's policy update scheme (performing multiple epochs of minibatch updates on multi-use rollouts with a surrogate gradient) can be interpreted as approximated policy gradient ascent. We show how to control the bias accumulated by the surrogate gradients and use techniques from random reshuffling to prove a convergence
theorem for PPO that sheds light on PPO's success. Additionally, we identify a previously overlooked issue in truncated Generalized Advantage Estimation commonly used in PPO. The
geometric weighting scheme induces infinite mass collapse onto the longest $k$-step advantage
estimator at episode boundaries. Empirical evaluations show that a simple weight correction can yield substantial improvements in environments with strong terminal signal, such as Lunar Lander.
Read the full abstract ▾
All authors Leif Doering, Daniel Schmidt, Moritz Melcher, Sebastian Kassing, Benedikt Wille, Tilman Aach, Simon Weissmann
Machine Learning
Dynamic Topology Optimization for Non-IID Data in Decentralized Learning
Bart Cox, Antreas Ioannou and Jérémie Decouchant
Decentralized learning (DL) enables a set of nodes to train a model collaboratively without central coordination, offering benefits for
privacy and scalability. However, DL struggles to train a high accuracy model when the data distribution is non-independent and identically distributed (non-IID) and when the communication
topology is static. To address these issues, we propose Morph, a
topology optimization
algorithm for DL. In Morph, nodes adaptively choose peers for model exchange based on maximum model dissimilarity. Morph maintains a fixed in-degree while dynamically reshaping the communication graph through gossip-based peer discovery and diversity-driven neighbor selection, thereby improving robustness to data heterogeneity. Experiments on CIFAR-10 and FEMNIST with up to 100 nodes show that Morph consistently outperforms static and epidemic baselines, while closely tracking the fully connected upper bound. On CIFAR-10, Morph achieves a relative improvement of 1.12x in test accuracy compared to the state-of-the-art baselines. On FEMNIST, Morph achieves an accuracy that is 1.08x higher than Epidemic Learning. Similar trends hold for 50 node deployments, where Morph narrows the gap to the fully connected upper bound within 0.5 percentage points on CIFAR-10. These results demonstrate that Morph achieves higher final accuracy, faster convergence, and more stable learning as quantified by lower inter-node variance, while requiring fewer communication rounds than baselines and no global knowledge.
Read the full abstract ▾
Machine Learning
Rethinking Benign Relearning: Syntax as the Hidden Driver of Unlearning Failures
Sangyeon Yoon, Hyesoo Hong, Wonje Jeung et al. (+1 more)
Machine unlearning aims to remove specific content from trained models while preserving overall performance. However, the phenomenon of benign relearning, in which forgotten information reemerges even from benign fine-tuning data, reveals that existing unlearning methods remain fundamentally fragile. A common explanation attributes this effect to topical relevance, but we find this account insufficient. Through systematic analysis, we demonstrate that syntactic similarity, rather than topicality, is the primary driver: across
benchmarks, syntactically similar data consistently trigger recovery even without topical overlap, due to their
alignment in representations and gradients with the forgotten content. Motivated by this insight, we introduce syntactic diversification, which paraphrases the original forget queries into heterogeneous structures prior to unlearning. This approach effectively suppresses benign relearning, accelerates forgetting, and substantially alleviates the trade-off between unlearning efficacy and model utility.
Read the full abstract ▾
All authors Sangyeon Yoon, Hyesoo Hong, Wonje Jeung, Albert No
Machine Learning · Cross-listed from cs.CV
GryphOne: Symbol-Aware Masked Diffusion for Structural Refinement in Offline Handwritten Mathematical Expression Recognition
Takaya Kawakatsu and Ryo Ishiyama
Handwritten mathematical expression recognition (HMER) requires
reasoning over diverse symbols and structures, yet autoregressive models struggle with exposure bias and syntax inconsistency. We present GryphOne, a discrete
diffusion framework which reformulates HMER as iterative symbolic refinement instead of sequential generation. GryphOne progressively refines symbols and relations, removing autoregression and improving consistency. Symbol-aware tokenization and random-masking mutual learning further enhance robustness to handwriting diversity. On the MathWriting
benchmark, GryphOne achieves 5.51% CER and 59.9% EM (ExpRate), outperforming all reimplemented models in the matched setting as well as the commercial HMER system. Held-out evaluation on CROHME 2014-2023 further shows strong cross-
dataset generalization.
Read the full abstract ▾
Machine Learning
MeKi: Memory-based Expert Knowledge Injection for Efficient LLM Scaling
Ning Ding, Fangcheng Liu, Kyungrae Kim et al. (+4 more)
Scaling Large
Language Models (LLMs) typically relies on increasing the number of parameters or test-time computations to boost performance. However, these strategies are impractical for edge device deployment due to limited RAM and NPU resources. Despite hardware constraints, deploying performant LLM on edge devices such as smartphone remains crucial for user experience. To address this, we propose MeKi (
Memory-based Expert Knowledge Injection), a novel system that scales LLM capacity via storage space rather than FLOPs. MeKi equips each
Transformer layer with
token-level
memory experts that injects pre-stored semantic knowledge into the generation process. To bridge the gap between training capacity and inference efficiency, we employ a re-parameterization strategy to fold parameter
matrices used during training into a compact static lookup table. By offloading the knowledge to ROM, MeKi decouples model capacity from computational cost, introducing zero inference latency overhead. Extensive experiments demonstrate that MeKi significantly outperforms dense LLM baselines with identical inference speed, validating the effectiveness of
memory-based scaling paradigm for on-device LLMs. Project homepage is at https://github.com/ningding-o/MeKi.
Read the full abstract ▾
All authors Ning Ding, Fangcheng Liu, Kyungrae Kim, Linji Hao, Kyeng-Hun Lee, Hyeonmok Ko, Yehui Tang
Machine Learning · Cross-listed from cs.AI
GFlowPO: Generative Flow Network as a Language Model Prompt Optimizer
Junmo Cho, Suhan Kim, Sangjune An et al. (+5 more)
Finding effective
prompts for
language models (LMs) is critical yet notoriously difficult: the
prompt space is combinatorially large, rewards are sparse due to expensive target-LM evaluation. Yet, existing RL-based
prompt optimizers often rely on on-policy updates and a meta-
prompt sampled from a fixed distribution, leading to poor sample efficiency. We propose GFlowPO, a probabilistic
prompt optimization framework that casts
prompt search as a posterior inference problem over latent
prompts regularized by a meta-prompted reference-LM prior. In the first step, we fine-tune a lightweight
prompt-LM with an off-policy Generative Flow Network (GFlowNet) objective, using a replay-based training policy that reuses past
prompt evaluations to enable sample-efficient exploration. In the second step, we introduce Dynamic
Memory Update (DMU), a training-free mechanism that updates the meta-
prompt by injecting both (i) diverse
prompts from a replay buffer and (ii) top-performing
prompts from a small priority queue, thereby progressively concentrating the search process on high-reward regions. Across few-shot text classification, instruction induction
benchmarks, and question answering tasks, GFlowPO consistently outperforms recent discrete
prompt optimization baselines.
Read the full abstract ▾
All authors Junmo Cho, Suhan Kim, Sangjune An, Minsu Kim, Dong Bok Lee, Heejun Lee, Sung Ju Hwang, Hae Beom Lee
Machine Learning
Achieving Linear Speedup for Composite Federated Learning
Kun Huang, Shi Pu and Karl Henrik Johansson
This paper proposes FedNMap, a normal map-based method for composite
federated learning, where the objective consists of a smooth loss and a possibly nonsmooth regularizer. FedNMap leverages a normal map-based update scheme to handle the nonsmooth term and incorporates a local correction strategy to mitigate the impact of data heterogeneity across clients. Under standard assumptions, including smooth local losses, weak convexity of the regularizer, and bounded
stochastic gradient variance, FedNMap achieves linear speedup with respect to both the number of clients and the number of local updates for nonconvex losses, both with and without the Polyak-Łojasiewicz condition. To the best of our knowledge, this is the first
algorithm establishing linear speedup for nonconvex composite
federated learning. Numerical experiments corroborate our theoretical findings and demonstrate the linear speedup of FedNMap.
Read the full abstract ▾
Machine Learning · Cross-listed from cs.SD
PACE: Pretrained Audio Continual Learning
Chang Li, Kanglei Zhou and Liyuan Wang
Audio is a fundamental modality for analyzing
speech, music, and environmental sounds. Although pretrained audio models have significantly advanced audio understanding, they remain fragile in real-world settings where data distributions shift over time. In this work, we present the first systematic
benchmark for audio continual learning (CL) with pretrained models (PTMs), together with a comprehensive analysis of its unique challenges. Unlike in
vision, where parameter-efficient fine-tuning (PEFT) has proven effective for CL, directly transferring such strategies to audio leads to poor performance. This stems from a fundamental property of audio backbones: they focus on low-level spectral details rather than structured semantics, causing severe upstream-downstream misalignment. Through extensive empirical study, we identify analytic classifiers with first-session adaptation (FSA) as a promising direction, but also reveal two major limitations: representation saturation in coarse-grained scenarios and representation drift in fine-grained scenarios. To address these challenges, we propose PACE, a novel method that enhances FSA via a regularized analytic classifier and enables multi-session adaptation through adaptive subspace-orthogonal PEFT for improved semantic
alignment. In addition, we introduce spectrogram-based boundary-aware perturbations to mitigate representation overlap and improve stability. Experiments on six diverse audio CL
benchmarks demonstrate that PACE substantially outperforms state-of-the-art baselines, marking an important step toward robust and scalable audio continual learning with PTMs.
Read the full abstract ▾