Machine Learning
Craig Atkinson
Quantized
language models face a fundamental dilemma: low sampling temperatures yield repetitive, mode-collapsed outputs, while high temperatures (T > 2.0) cause trajectory divergence and semantic incoherence. We present HELIX, a
geometric framework that decouples output
entropy from
hallucination by tethering hidden-state trajectories to a pre-computed truthfulness
manifold. HELIX computes a Unified Truth Score (UTS) combining
token-level semantic
entropy with Mahalanobis distance from the
manifold. When UTS indicates trajectory divergence, graduated steering vectors redirect activations toward structurally coherent regions while affecting only 0.2-2.5% of
tokens. On 4-bit quantized Granite 4.0 H Small (32B/9B active, hybrid Mamba-
Transformer): GSM8K maintains 88.84% accuracy at T = 3.0 (2.81pp degradation from T = 0.5); MMLU maintains 72.49% across 14,042 questions (1.24pp degradation). This demonstrates that high-temperature
hallucination is primarily trajectory divergence rather than semantic collapse. Notably, steering the sparse
Transformer attention layers (~10% of layers) is sufficient to correct drift in the Mamba-2 state-space formulation.
Geometric tethering reveals a previously-masked High-
Entropy Creative Reservoir. At T > 2.0, steered outputs exhibit 5-20% idea duplication versus 70-80% at conservative settings. Cross-architecture validation (Qwen3-30B-A3B MOE) confirms this phenomenon is architecture-independent, with 46.7% higher unique concept generation. HELIX acts as a syntax tether, enabling exploration of semantic diversity without violating the logical backbone required for valid output. This enables Multi-Temperature Synthesis, generating 200% more unique concepts than single-temperature inference.
Read the full abstract ▾
Machine Learning · Cross-listed from cs.GR
DesignAsCode: Bridging Structural Editability and Visual Fidelity in Graphic Design Generation
Ziyuan Liu, Shizhao Sun, Danqing Huang et al. (+5 more)
Graphic design generation demands a delicate balance between high visual fidelity and fine-grained structural editability. However, existing approaches typically bifurcate into either non-editable raster image synthesis or abstract layout generation devoid of visual content. Recent combinations of these two approaches attempt to bridge this gap but often suffer from rigid composition schemas and unresolvable visual dissonances (e.g., text-background conflicts) due to their inexpressive representation and open-loop nature. To address these challenges, we propose DesignAsCode, a novel framework that reimagines graphic design as a programmatic synthesis task using HTML/CSS. Specifically, we introduce a Plan-Implement-Reflect pipeline, incorporating a Semantic Planner to construct dynamic, variable-depth element hierarchies and a Visual-Aware Reflection mechanism that optimizes the code to rectify rendering artifacts. Extensive experiments demonstrate that DesignAsCode significantly outperforms baselines in both structural validity and aesthetic quality. Furthermore, our code-native representation unlocks advanced capabilities, including automatic layout retargeting, complex document generation (e.g., resumes), and CSS-based animation. Our project page is available at https://liuziyuan1109.github.io/design-as-code/.
Read the full abstract ▾
All authors Ziyuan Liu, Shizhao Sun, Danqing Huang, Yingdong Shi, Meisheng Zhang, Ji Li, Jingsong Yu, Jiang Bian
Machine Learning
Robust Pre-Training of Medical Vision-and-Language Models with Domain-Invariant Multi-Modal Masked Reconstruction
Melika Filvantorkaman and Mohsen Piri
Medical
vision-
language models show strong potential for joint
reasoning over medical images and clinical text, but their performance often degrades under domain shift caused by variations in imaging devices, acquisition protocols, and reporting styles. Existing multi-modal pre-training methods largely overlook robustness, treating it as a downstream adaptation problem. In this work, we propose Robust Multi-Modal Masked Reconstruction (Robust-MMR), a self-supervised pre-training framework that explicitly incorporates robustness objectives into masked
vision-
language learning. Robust-MMR integrates asymmetric perturbation-aware masking, domain-consistency regularization, and modality-resilience constraints to encourage domain-invariant representations. We evaluate Robust-MMR on multiple medical
vision-
language benchmarks, including medical visual question answering (VQA-RAD, SLAKE, VQA-2019), cross-domain image-text classification (MELINDA), and robust image-caption
retrieval (ROCO). Robust-MMR achieves 78.9% cross-domain accuracy on VQA-RAD, outperforming the strongest baseline by 3.8 percentage points, and reaches 74.6% and 77.0% accuracy on SLAKE and VQA-2019, respectively. Under perturbed evaluation, Robust-MMR improves VQA-RAD accuracy from 69.1% to 75.6%. For image-text classification, cross-domain MELINDA accuracy increases from 70.3% to 75.2%, while
retrieval experiments show a reduction in mean rank degradation from over 16 to 4.1 under perturbation. Qualitative results further demonstrate improved clinical
reasoning for disease detection and structural abnormality assessment. These findings show that explicitly modeling robustness during pre-training leads to more reliable and transferable medical
vision-
language representations for real-world deployment.
Read the full abstract ▾
Machine Learning · Cross-listed from eess.SP
NeuroSleep: Neuromorphic Event-Driven Single-Channel EEG Sleep Staging for Edge-Efficient Sensing
Boyu Li, Xingchun Zhu and Yonghui Wu
Objective. Reliable, continuous neural sensing on wearable edge platforms is fundamental to long-term health monitoring; however, for electroencephalography (
EEG)-based sleep monitoring, dense high-frequency processing is often computationally prohibitive under tight energy budgets. Approach. To address this bottleneck, this paper proposes NeuroSleep, an integrated event-driven sensing and inference system for energy-efficient sleep staging. NeuroSleep first converts raw
EEG into complementary multi-scale bipolar event streams using Residual Adaptive Multi-Scale Delta Modulation (R-AMSDM), enabling an explicit fidelity-sparsity trade-off at the sensing front end. Furthermore, NeuroSleep adopts a hierarchical inference architecture that comprises an Event-based Adaptive Multi-scale Response (EAMR) module for local feature extraction, a Local Temporal-
Attention Module (LTAM) for context aggregation, and an Epoch-Leaky Integrate-and-Fire (ELIF) module to capture long-term state persistence. Main results. Experimental results using subject-independent 5-fold cross-validation on the Sleep-EDF Expanded sleep-cassette (SC) subset with single-channel
EEG demonstrate that NeuroSleep achieves a mean accuracy of 74.2% with only 0.932 M parameters while reducing sparsity-adjusted effective operations by approximately 53.6% relative to dense processing. Compared to the representative dense
Transformer baseline, NeuroSleep improves accuracy by 7.5% with a 45.8% reduction in computational load. Significance. By coupling neuromorphic event encoding with state-aware context modeling, NeuroSleep offers a deployment-oriented framework for single-channel sleep staging that reduces redundant high-rate processing and improves energy scalability for wearable and edge platforms.
Read the full abstract ▾
Machine Learning · Cross-listed from cs.CV
Towards Understanding Multimodal Fine-Tuning: Spatial Features
Lachin Naghashyar, Hunar Batra, Ashkan Khakzar et al. (+4 more)
Contemporary
Vision-
Language Models (VLMs) achieve strong performance on a wide range of tasks by pairing a
vision encoder with a pre-trained
language model, fine-tuned for visual-text inputs. Yet despite these gains, it remains unclear how
language backbone representations adapt during
multimodal training and when
vision-specific capabilities emerge. In this work, we present the first mechanistic analysis of VLM adaptation. Using stage-wise model diffing, a technique that isolates representational changes introduced during
multimodal fine-tuning, we reveal how a
language model learns to "see". We first identify
vision-preferring features that emerge or reorient during fine-tuning. We then show that a selective subset of these features reliably encodes spatial relations, revealed through controlled shifts to spatial
prompts. Finally, we trace the causal activation of these features to a small group of
attention heads. Our findings show that stage-wise model diffing reveals when and where spatially grounded
multimodal features arise. It also provides a clearer view of modality fusion by showing how visual grounding reshapes features that were previously text-only. This methodology enhances the interpretability of
multimodal training and provides a foundation for understanding and refining how pretrained
language models acquire
vision-grounded capabilities.
Read the full abstract ▾
All authors Lachin Naghashyar, Hunar Batra, Ashkan Khakzar, Philip Torr, Ronald Clark, Christian Schroeder de Witt, Constantin Venhoff
Machine Learning
Hybrid Feedback-Guided Optimal Learning for Wireless Interactive Panoramic Scene Delivery
Xiaoyi Wu, Juaren Steiger, Bin Li et al. (+1 more)
Immersive applications such as virtual and augmented reality impose stringent requirements on frame rate, latency, and synchronization between physical and virtual environments. To meet these requirements, an edge server must render panoramic content, predict user head motion, and transmit a portion of the scene that is large enough to cover the user viewport while remaining within wireless bandwidth constraints. Each portion produces two feedback signals: prediction feedback, indicating whether the selected portion covers the actual viewport, and transmission feedback, indicating whether the corresponding packets are successfully delivered. Prior work models this problem as a multi-armed bandit with two-level bandit feedback, but fails to exploit the fact that prediction feedback can be retrospectively computed for all candidate portions once the user head pose is observed. As a result, prediction feedback constitutes full-information feedback rather than bandit feedback. Motivated by this observation, we introduce a two-level hybrid feedback model that combines full-information and bandit feedback, and formulate the portion selection problem as an online learning task under this setting. We derive an instance-dependent regret lower bound for the hybrid feedback model and propose AdaPort, a hybrid learning
algorithm that leverages both feedback types to improve learning efficiency. We further establish an instance-dependent regret upper bound that matches the lower bound asymptotically, and demonstrate through measurements on an end-to-end testbed that AdaPort outperforms state-of-the-art learning-based baselines as well as the heuristic minimum scene delivery scheme.
Read the full abstract ▾
All authors Xiaoyi Wu, Juaren Steiger, Bin Li, R. Srikant
Machine Learning · Cross-listed from cs.AI
BRIDGE: Predicting Human Task Completion Time From Model Performance
Fengyuan Liu, Jay Gala, Nilaksh et al. (+3 more)
Evaluating the real-world capabilities of AI systems requires grounding
benchmark performance in human-interpretable measures of task difficulty. Existing approaches that rely on direct human task completion time annotations are costly, noisy, and difficult to scale across
benchmarks. In this work, we propose BRIDGE, a unified psychometric framework that learns a latent difficulty scale from model responses and anchors it to human task completion time. Using a two-parameter logistic Item Response Theory model, we jointly estimate latent task difficulty and model capability from model performance data across multiple
benchmarks. We demonstrate that latent task difficulty varies linearly with the logarithm of human completion time, allowing human task completion time to be inferred for new
benchmarks from model performance alone. Leveraging this
alignment, we forecast frontier model capabilities in terms of human task length and independently reproduce METR's exponential scaling results, with the 50% solvable task horizon doubling approximately every 6 months.
Read the full abstract ▾
All authors Fengyuan Liu, Jay Gala, Nilaksh, Dzmitry Bahdanau, Siva Reddy, Hugo Larochelle
Machine Learning
XShare: Collaborative in-Batch Expert Sharing for Faster MoE Inference
Daniil Vankov, Nikita Ivkin, Kyle Ulrich et al. (+3 more)
Mixture-of-Experts (MoE) architectures are increasingly used to efficiently scale large
language models. However, in production inference, request batching and speculative decoding significantly amplify expert activation, eroding these efficiency benefits. We address this issue by modeling batch-aware expert selection as a modular optimization problem and designing efficient greedy
algorithms for different deployment settings. The proposed method, namely XShare, requires no retraining and dynamically adapts to each batch by maximizing the total gating score of selected experts. It reduces expert activation by up to 30% under standard batching, cuts peak GPU load by up to 3x in expert-parallel deployments, and achieves up to 14% throughput gains in speculative decoding via hierarchical, correlation-aware expert selection even if requests in a batch drawn from heterogeneous
datasets.
Read the full abstract ▾
All authors Daniil Vankov, Nikita Ivkin, Kyle Ulrich, Xiang Song, Ashish Khetan, George Karypis
Machine Learning
tLoRA: Efficient Multi-LoRA Training with Elastic Shared Super-Models
Kevin Li, Dibyadeep Saha, Avni Kanodia et al. (+1 more)
As Low-Rank Adaptation (LoRA) becomes the standard approach for efficiently fine-tuning large
language models (LLMs), shared clusters increasingly execute many concurrent LoRA training jobs over the same frozen backbone. While recent advances enable batching (co-locating) multiple adapters during serving, efficient training-time co-location of heterogeneous LoRA adapters presents unique challenges. Jobs often differ in adapter rank, batch size, and resource allocation, and naïve batching can introduce synchronization stalls, communication overheads, and per-job slowdowns that are worse than executing independently. We introduce tLoRA, a framework that enables efficient batch training of multiple LoRA jobs. tLoRA fuses adapters that share the same base model into an elastic shared super-model, exploiting existing distributed training frameworks to derive parallelism plans that share resources effectively. At the kernel level, tLoRA employs a fused LoRA kernel that adaptively reconstructs low-rank computation tiles and schedules rank-aware nano-batches to maximize overlap between computation and communication across adapters. At the scheduling layer, tLoRA incorporates an online, residual-capacity-aware scheduler that adaptively groups jobs to maximize collective throughput. Evaluations using real-world cluster traces demonstrate that tLoRA improves training throughput by 1.2–1.8x, job training completion time by 2.3–5.4x, and GPU utilization by 37%.
Read the full abstract ▾
All authors Kevin Li, Dibyadeep Saha, Avni Kanodia, Fan Lai
Machine Learning · Cross-listed from cs.CV
3D Transport-based Morphometry (3D-TBM) for medical image analysis
Hongyu Kan, Kristofor Pas, Ivan Medri et al. (+5 more)
Transport-Based Morphometry (TBM) has emerged as a new framework for 3D medical image analysis. By
embedding images into a transport domain via invertible transformations, TBM facilitates effective classification, regression, and other tasks using transport-domain features. Crucially, the inverse mapping enables the projection of analytic results back into the original image space, allowing researchers to directly interpret clinical features associated with model outputs in a spatially meaningful way. To facilitate broader adoption of TBM in clinical imaging research, we present 3D-TBM, a tool designed for morphological analysis of 3D medical images. The framework includes data preprocessing, computation of optimal transport
embeddings, and analytical methods such as visualization of main transport directions, together with techniques for discerning discriminating directions and related analysis methods. We also provide comprehensive documentation and practical tutorials to support researchers interested in applying 3D-TBM in their own medical imaging studies. The source code is publicly available through PyTransKit.
Read the full abstract ▾
All authors Hongyu Kan, Kristofor Pas, Ivan Medri, Naqib Sad Pathan, Natasha Ironside, Shinjini Kundu, Jingjia He, Gustavo Kunde Rohde
Machine Learning
Robust Ultra-High-Dimensional Variable Selection With Correlated Structure Using Group Testing
Wanru Guo, Juan Xie, Binbin Wang et al. (+4 more)
Background: High-dimensional genomic data exhibit strong group correlation structures that challenge conventional feature selection methods, which often assume feature independence or rely on pre-defined pathways and are sensitive to outliers and model misspecification. Methods: We propose the Dorfman screening framework, a multi-stage procedure that forms data-driven variable groups via hierarchical clustering, performs group and within-group hypothesis testing, and refines selection using elastic net or adaptive elastic net. Robust variants incorporate OGK-based covariance estimation, rank-based correlation, and Huber-weighted regression to handle contaminated and non-normal data. Results: In simulations, Dorfman-Sparse-Adaptive-EN performed best under normal conditions, while Robust-OGK-Dorfman-Adaptive-EN showed clear advantages under data contamination, outperforming classical Dorfman and competing methods. Applied to NSCLC gene expression data for trametinib response, robust Dorfman methods achieved the lowest prediction errors and enriched recovery of clinically relevant genes. Conclusions: The Dorfman framework provides an efficient and robust approach to genomic feature selection. Robust-OGK-Dorfman-Adaptive-EN offers strong performance under both ideal and contaminated conditions and scales to ultra-high-dimensional settings, making it well suited for modern genomic biomarker discovery.
Read the full abstract ▾
All authors Wanru Guo, Juan Xie, Binbin Wang, Weicong Chen, Xiaoyi Lu, Vipin Chaudhary, Curtis Tatsuoka
Machine Learning
Graph homophily booster: Reimagining the role of discrete features in heterophilic graph learning
Ruizhong Qiu, Ting-Wei Li, Gaotang Li et al. (+1 more)
Graph neural networks (GNNs) have emerged as a powerful tool for modeling graph-structured data. However, existing GNNs often struggle with heterophilic graphs, where connected nodes tend to have dissimilar features or labels. While numerous methods have been proposed to address this challenge, they primarily focus on architectural designs without directly targeting the root cause of the heterophily problem. These approaches still perform even worse than the simplest MLPs on challenging heterophilic
datasets. For instance, our experiments show that 21 latest GNNs still fall behind the MLP on the Actor
dataset. This critical challenge calls for an innovative approach to addressing graph heterophily beyond architectural designs. To bridge this gap, we propose and study a new and unexplored paradigm: directly increasing the graph homophily via a carefully designed graph transformation. In this work, we present a simple yet effective framework called GRAPHITE to address graph heterophily. To the best of our knowledge, this work is the first method that explicitly transforms the graph to directly improve the graph homophily. Stemmed from the exact definition of homophily, our proposed GRAPHITE creates feature nodes to facilitate homophilic message passing between nodes that share similar features. Furthermore, we both theoretically and empirically show that our proposed GRAPHITE significantly increases the homophily of originally heterophilic graphs, with only a slight increase in the graph size. Extensive experiments on challenging
datasets demonstrate that our proposed GRAPHITE significantly outperforms state-of-the-art methods on heterophilic graphs while achieving comparable accuracy with state-of-the-art methods on homophilic graphs.
Read the full abstract ▾
All authors Ruizhong Qiu, Ting-Wei Li, Gaotang Li, Hanghang Tong
Machine Learning · Cross-listed from cs.CR
Beyond Crash: Hijacking Your Autonomous Vehicle for Fun and Profit
Qi Sun, Ahmed Abdo, Luis Burbano et al. (+4 more)
Autonomous Vehicles (AVs), especially
vision-based AVs, are rapidly being deployed without human operators. As AVs operate in safety-critical environments, understanding their robustness in an
adversarial environment is an important research problem. Prior physical
adversarial attacks on
vision-based autonomous vehicles predominantly target immediate safety failures (e.g., a crash, a traffic-rule violation, or a transient lane departure) by inducing a short-lived perception or control error. This paper shows a qualitatively different risk: a long-horizon route integrity compromise, where an attacker gradually steers a victim AV away from its intended route and into an attacker-chosen destination while the victim continues to drive “normally.” This will not pose a danger to the victim vehicle itself, but also to potential passengers sitting inside the vehicle, who may not notice the route changes. In this paper, we design and implement the first
adversarial framework, called JackZebra, which performs route-level hijacking of a
vision-based end-to-end driving stack using a physically plausible attacker vehicle with a reconfigurable display and a camera sensor mounted on the rear. The central challenge is temporal persistence:
adversarial influence must remain effective in changing viewpoints, lighting, weather, traffic, and the victim's continual replanning – without triggering conspicuous failures. Our key insight is to treat route hijacking as a closed-loop control problem and to convert
adversarial patches into steering primitives that can be selected online via an interactive adjustment loop based on observed victim behavior using the rear camera. Our evaluations in both simulated and real-world scenarios show that JackZebra can successfully hijack victim vehicles to deviate from original routes and stop at places designated by the adversary with a high success rate.
Read the full abstract ▾
All authors Qi Sun, Ahmed Abdo, Luis Burbano, Ziyang Li, Yaxing Yao, Alvaro Cardenas, Yinzhi Cao
Machine Learning · Cross-listed from cs.AI
Is there "Secret Sauce” in Large Language Model Development?
Matthias Mertens, Natalia Fischl-Lanzoni and Neil Thompson
Do leading LLM developers possess a proprietary “secret sauce”, or is LLM performance driven by scaling up compute? Using training and
benchmark data for 809 models released between 2022 and 2025, we estimate scaling-law regressions with release-date and developer fixed effects. We find clear evidence of developer-specific efficiency advantages, but their importance depends on where models lie in the performance distribution. At the frontier, 80-90% of performance differences are explained by higher training compute, implying that scale–not proprietary technology–drives frontier advances. Away from the frontier, however, proprietary techniques and shared algorithmic progress substantially reduce the compute required to reach fixed capability thresholds. Some companies can systematically produce smaller models more efficiently. Strikingly, we also find substantial variation of model efficiency within companies; a firm can train two models with more than 40x compute efficiency difference. We also discuss the implications for AI leadership and capability
diffusion.
Read the full abstract ▾
Machine Learning
ArcMark: Distortion-Free Multi-Byte LLM Watermark via Optimal Transport
Atefeh Gilani, Sajani Vithana, Carol Xuan Long et al. (+3 more)
Watermarking is an important tool for promoting the responsible use of large
language models (LLMs). Existing watermarks insert a signal into generated
tokens that either flags LLM-generated text (zero-bit watermarking) or encodes more complex messages (multi-bit watermarking). Though a number of recent approaches insert multiple bits into text without perturbing average next-
token predictions, they largely extend design principles from the zero-bit setting, such as encoding a single bit per
token. In contrast, a watermarker capable of
embedding multiple bytes into the text would dramatically increase the potential applications, by
embedding information such as the ID of the user who submitted the
prompt, the precise model version that was used, or even the
prompt itself. We address this problem by introducing ArcMark: a new watermark construction based on coding and information-theoretic principles that is capable of reliably
embedding multiple bytes of information into just a few hundred
tokens, without any distortion of the underlying LLM next-
token distribution. We derive ArcMark by formulating the distortion-free watermarking problem as a channel coding problem, and deriving an information-theoretic channel capacity that establishes the fundamental limit of
embedding information in LLM output in a distortion-free manner. This capacity formulation informs the design of ArcMark. In practice, ArcMark outperforms competing multi-bit distortion-free watermarks in terms of reconstruction accuracy, including in the face of attacks that alter a subset of the LLM text. ArcMark output is also shown to be indistinguishable from unwatermarked text in terms of perplexity, and in downstream task quality.
Read the full abstract ▾
All authors Atefeh Gilani, Sajani Vithana, Carol Xuan Long, Oliver Kosut, Lalitha Sankar, Flavio P. Calmon
Machine Learning · Cross-listed from eess.IV
Extracting Root-Causal Brain Activity Driving Psychopathology from Resting State fMRI
Eric V. Strobl
Neuroimaging studies of psychiatric disorders often correlate imaging patterns with diagnostic labels or composite symptom scores, yielding diffuse associations that obscure underlying mechanisms. We instead seek to identify root-causal maps – localized BOLD disturbances that initiate pathological cascades – and to link them selectively to symptom dimensions. We introduce a bilevel structural causal model that connects between-subject symptom structure to within-subject resting-state fMRI via independent latent sources with localized direct effects. Based on this model, we develop SOURCE (Symptom-Oriented Uncovering of Root-Causal Elements), a procedure that links interpretable symptom axes to a parsimonious set of localized drivers. Experiments show that SOURCE recovers localized maps consistent with root-causal BOLD drivers and increases interpretability and anatomical specificity relative to existing comparators.
Read the full abstract ▾
Machine Learning
Cerebellar-Inspired Residual Control for Fault Recovery: From Inference-Time Adaptation to Structural Consolidation
Nethmi Jayasinghe, Diana Gontero, Spencer T. Brown et al. (+3 more)
Robotic policies deployed in real-world environments often encounter post-training faults, where retraining, exploration, or system identification are impractical. We introduce an inference-time, cerebellar-inspired residual control framework that augments a frozen reinforcement learning policy with online corrective actions, enabling fault recovery without modifying base policy parameters. The framework instantiates core cerebellar principles, including high-dimensional pattern separation via fixed feature expansion, parallel microzone-style residual pathways, and local error-driven plasticity with excitatory and inhibitory eligibility traces operating at distinct time scales. These mechanisms enable fast, localized correction under post-training disturbances while avoiding destabilizing global policy updates. A conservative, performance-driven meta-adaptation regulates residual authority and plasticity, preserving nominal behavior and suppressing unnecessary intervention. Experiments on MuJoCo
benchmarks under actuator, dynamic, and environmental perturbations show improvements of up to $+66\%$ on HalfCheetah-v5 and $+53\%$ on
Humanoid-v5 under moderate faults, with graceful degradation under severe shifts and complementary robustness from consolidating persistent residual corrections into policy parameters.
Read the full abstract ▾
All authors Nethmi Jayasinghe, Diana Gontero, Spencer T. Brown, Vinod K. Sangwan, Mark C. Hersam, Amit Ranjan Trivedi
Machine Learning
Fault-Tolerant Evaluation for Sample-Efficient Model Performance Estimators
Zihan Zhu, Yanqiu Wu and Qiongkai Xu
In the era of Model-as-a-Service, organizations increasingly rely on third-party AI models for rapid deployment. However, the dynamic nature of emerging AI applications, the continual introduction of new
datasets, and the growing number of models claiming superior performance make efficient and reliable validation of model services increasingly challenging. This motivates the development of sample-efficient performance
estimators, which aim to estimate model performance by strategically selecting instances for labeling, thereby reducing annotation cost. Yet existing evaluation approaches often fail in low-variance settings: RMSE conflates bias and variance, masking persistent bias when variance is small, while p-value based tests become hypersensitive, rejecting adequate
estimators for negligible deviations. To address this, we propose a fault-tolerant evaluation framework that integrates bias and variance considerations within an adjustable tolerance level ${\varepsilon}$, enabling the evaluation of performance
estimators within practically acceptable error margins. We theoretically show that proper calibration of ${\varepsilon}$ ensures reliable evaluation across different variance regimes, and we further propose an
algorithm that automatically optimizes and selects ${\varepsilon}$. Experiments on real-world
datasets demonstrate that our framework provides comprehensive and actionable insights into
estimator behavior.
Read the full abstract ▾
Machine Learning
Vegas: Self-Speculative Decoding with Verification-Guided Sparse Attention
Yikang Yue, Yuqi Xue and Jian Huang
Long-context large
language model (LLM) inference has become the norm for today's AI applications. However, it is severely bottlenecked by the increasing
memory demands of its KV cache. Previous works have shown that self-speculative decoding with sparse
attention, where
tokens are drafted using a subset of the KV cache and verified in parallel against the full KV cache, speeds up inference in a lossless manner. However, they rely on a standalone KV selection
algorithm to select the KV entries used for drafting and overlook the fact that the criticality of each KV entry is inherently computed during verification. In this paper, we propose Vegas, a self-speculative decoding method with verification-guided sparse
attention. Vegas identifies critical KV cache entries as a byproduct of verification and computes
attention only over these entries when drafting subsequent
tokens. This not only improves the draft
token acceptance rate but also incurs low KV selection overhead, thereby improving decoding throughput. Vegas achieves a 1.25$\times$-2.81$\times$ speedup in decoding throughput over default vLLM and a 1.15$\times$-1.29$\times$ speedup over state-of-the-art sparse
attention-based self-speculative decoding methods. Our code is available at https://github.com/platformxlab/vegas.
Read the full abstract ▾
Machine Learning
The Median is Easier than it Looks: Approximation with a Constant-Depth, Linear-Width ReLU Network
Abhigyan Dutta, Itay Safran and Paul Valiant
We study the approximation of the median of $d$ inputs using ReLU neural networks. We present depth-width tradeoffs under several settings, culminating in a constant-depth, linear-width construction that achieves exponentially small approximation error with respect to the uniform distribution over the unit hypercube. By further establishing a general reduction from the maximum to the median, our results break a barrier suggested by prior work on the maximum function, which indicated that linear width should require depth growing at least as $\log\log d$ to achieve comparable accuracy. Our construction relies on a multi-stage procedure that iteratively eliminates non-central elements while preserving a candidate set around the median. We overcome obstacles that do not arise for the maximum to yield approximation results that are strictly stronger than those previously known for the maximum itself.
Read the full abstract ▾
Machine Learning
Collaborative and Efficient Fine-tuning: Leveraging Task Similarity
Gagik Magakyan, Amirhossein Reisizadeh, Chanwoo Park et al. (+2 more)
Adaptability has been regarded as a central feature in the foundation models, enabling them to effectively acclimate to unseen downstream tasks. Parameter-efficient fine-tuning methods such as celebrated LoRA facilitate efficient adaptation of large foundation models using labeled, high-quality and generally scarce task data. To mitigate data scarcity in fine-tuning of foundation models, we propose to leverage task similarity across multiple downstream users. Intuitively, users with similar tasks must be able to assist each other in boosting the effective fine-tuning data size. We propose Collaborative Low-Rank Adaptation, or CoLoRA, which exploits task similarity to collaboratively and efficiently fine-tune personalized foundation models. The main idea in CoLoRA is to train one shared adapter capturing underlying task similarities across all tasks, and personalized adapters tailored to user-specific tasks. We theoretically study CoLoRA on heterogeneous linear regression and provide provable guarantees for ground truth recovery. We also conduct several natural
language experiments with varying task similarity, which further demonstrate that when trained together with similar tasks, individual performances are significantly boosted.
Read the full abstract ▾
All authors Gagik Magakyan, Amirhossein Reisizadeh, Chanwoo Park, Pablo A. Parrilo, Asuman Ozdaglar
Machine Learning
Beyond Solving: Prescriptive Probing for Neural Routing Solvers
Reuben Narad, Léonard Boussioux and Michael Wagner
Neural combinatorial optimization (NCO) trains fast heuristics for routing problems, but planners often need more than a single solve: they ask which stop to drop, which transition to preserve, or which subset of stops to remove if a route is infeasible. Answering such counterfactual questions by re-solving each candidate is expensive even when the action set is small. We introduce Prescriptive Probing: using the frozen representations of a trained NCO model to rank candidate interventions for what-if decision support. While recent interpretability work descriptively probes what NCO solvers encode, we ask whether those same representations can prescriptively guide intervention choices on real-road routing problems. Where exhaustive labels are available, we train supervised probes from offline re-solve labels; where they are not (e.g., combinatorial action spaces), we train a lightweight reinforcement learning intervention head. Across real-road
benchmarks on the Asymmetric Traveling Salesperson Problem and Capacitated Vehicle Routing Problem, probes built on frozen model representations achieve the strongest performance on a majority of intervention tasks and improve over local-repair heuristics on edge-forbiddance and on key node- and multi-node-removal metrics. We further investigate probeability as a property of the routing model itself, varying model quality, the mix of supervised learning and reinforcement learning, and the architecture family. Overall, our results suggest a new use case for NCO solvers: representations learned for route construction can transfer to decision support.
Read the full abstract ▾
Machine Learning
Adaptive Retrieval helps Reasoning in LLMs – but mostly if it's not used
Srijan Shakya, Anamaria-Roberta Hartl, Sepp Hochreiter et al. (+1 more)
Large
Language Models (LLMs) often falter in complex
reasoning tasks due to their static, parametric knowledge, leading to
hallucinations and poor performance in specialized domains like mathematics. This work explores a fundamental principle for enhancing generative models: treating
retrieval as a form of dynamic in-context learning. We test an adaptive
retrieval-augmented architecture where an LLM
agent actively decides when to query an external knowledge base during its
reasoning process. We compare this adaptive strategy against a standard Chain-of-Thought (CoT) baseline and a static
retrieval approach on the GSM8K and MATH-500
benchmarks. Although our experiments show that static
retrieval is inferior to CoT, the adaptive
retrieval shows interesting behavior: While traces including retrieved results show slightly worse performance compared to CoT, traces that do not include
retrieval actually perform better compared to CoT. This suggests that: (a)
retrieval only rarely helps
reasoning (we show a few counterexamples, e.g. using useful
theorems) and (b) actively not using
retrieval is indicative of good model performance. Furthermore, we find that the model scales its
retrieval frequency with the difficulty of the problem, reinforcing that the decision to retrieve is a crucial metacognitive signal. The
agent's ability to self-assess its knowledge and selectively engage with external information represents a key principle for building more robust and reliable generative models.
Read the full abstract ▾
All authors Srijan Shakya, Anamaria-Roberta Hartl, Sepp Hochreiter, Korbinian Pöppel
Machine Learning
DSL: Understanding and Improving Softmax Recommender Systems with Competition-Aware Scaling
Bucher Sahyouni, Matthew Vowels, Liqun Chen et al. (+1 more)
Softmax Loss (SL) is being increasingly adopted for recommender systems (RS) as it has demonstrated better performance, robustness and
fairness. Yet in implicit-feedback, a single global temperature and equal treatment of uniformly sampled negatives can lead to brittle training, because sampled sets may contain varying degrees of relevant or informative competitors. The optimal loss sharpness for a user-item pair with a particular set of negatives, can be suboptimal or destabilising for another with different negatives. We introduce Dual-scale Softmax Loss (DSL), which infers effective sharpness from the sampled competition itself. DSL adds two complementary branches to the log-sum-exp backbone. Firstly it reweights negatives within each training instance using hardness and item–item similarity, secondly it adapts a per-example temperature from the competition intensity over a constructed competitor slate. Together, these components preserve the geometry of SL while reshaping the competition distribution across negatives and across examples. Over several representative
benchmarks and backbones, DSL yields substantial gains over strong baselines, with improvements over SL exceeding $10%$ in several settings and averaging $6.22%$ across
datasets, metrics, and backbones. Under out-of-distribution (OOD) popularity shift, the gains are larger, with an average of $9.31%$ improvement over SL. We further provide a theoretical, distributionally robust optimisation (DRO) analysis, which demonstrates how DSL reshapes the robust payoff and the KL deviation for ambiguous instances. This helps explain the empirically observed improvements in accuracy and robustness.
Read the full abstract ▾
All authors Bucher Sahyouni, Matthew Vowels, Liqun Chen, Simon Hadfield
Machine Learning
Online Learning for Uninformed Markov Games: Empirical Nash-Value Regret and Non-Stationarity Adaptation
Junyan Liu, Haipeng Luo, Zihan Zhang et al. (+1 more)
We study online learning in two-player uninformed Markov games, where the opponent's actions and policies are unobserved. In this setting, Tian et al. (2021) show that achieving no-external-regret is impossible without incurring an exponential dependence on the episode length $H$. They then turn to the weaker notion of Nash-value regret and propose a V-learning
algorithm with regret $O(K^{2/3})$ after $K$ episodes. However, their
algorithm and guarantee do not adapt to the difficulty of the problem: even in the case where the opponent follows a fixed policy and thus $O(\sqrt{K})$ external regret is well-known to be achievable, their result is still the worse rate $O(K^{2/3})$ on a weaker metric. In this work, we fully address both limitations. First, we introduce empirical Nash-value regret, a new regret notion that is strictly stronger than Nash-value regret and naturally reduces to external regret when the opponent follows a fixed policy. Moreover, under this new metric, we propose a parameter-free
algorithm that achieves an $O(\min \{\sqrt{K} + (CK)^{1/3},\sqrt{LK}\})$ regret bound, where $C$ quantifies the variance of the opponent's policies and $L$ denotes the number of policy switches (both at most $O(K)$). Therefore, our results not only recover the two extremes – $O(\sqrt{K})$ external regret when the opponent is fixed and $O(K^{2/3})$ Nash-value regret in the worst case – but also smoothly interpolate between these extremes by automatically adapting to the opponent's non-stationarity. We achieve so by first providing a new analysis of the epoch-based V-learning
algorithm by Mao et al. (2022), establishing an $O(\eta C + \sqrt{K/\eta})$ regret bound, where $\eta$ is the epoch incremental factor. Next, we show how to adaptively restart this
algorithm with an appropriate $\eta$ in response to the potential non-stationarity of the opponent, eventually achieving our final results.
Read the full abstract ▾
All authors Junyan Liu, Haipeng Luo, Zihan Zhang, Lillian J. Ratliff
Machine Learning
Exactly Computing do-Shapley Values
R. Teal Witter, Álvaro Parafita, Tomas Garriga et al. (+4 more)
Structural Causal Models (SCM) are a powerful framework for describing complicated dynamics across the natural sciences. A particularly elegant way of interpreting SCMs is do-Shapley, a game-theoretic method of quantifying the average effect of $d$ variables across exponentially many interventions. Like Shapley values, computing do-Shapley values generally requires evaluating exponentially many terms. The foundation of our work is a reformulation of do-Shapley values in terms of the irreducible sets of the underlying SCM. Leveraging this insight, we can exactly compute do-Shapley values in time linear in the number of irreducible sets $r$, which itself can range from $d$ to $2^d$ depending on the graph structure of the SCM. Since $r$ is unknown a priori, we complement the exact
algorithm with an
estimator that, like general Shapley value
estimators, can be run with any query budget. As the query budget approaches $r$, our
estimators can produce more accurate estimates than prior methods by several orders of magnitude, and, when the budget reaches $r$, return the Shapley values up to machine precision. Beyond computational speed, we also reduce the identification burden: we prove that non-parametric identifiability of do-Shapley values requires only the identification of interventional effects for the $d$ singleton coalitions, rather than all classes.
Read the full abstract ▾
All authors R. Teal Witter, Álvaro Parafita, Tomas Garriga, Maximilian Muschalik, Fabian Fumagalli, Axel Brando, Lucas Rosenblatt
Machine Learning
Risk-Sensitive Exponential Actor Critic
Alonso Granados and Jason Pacheco
Model-free deep reinforcement learning (RL)
algorithms have achieved tremendous success on a range of challenging tasks. However, safety concerns remain when these methods are deployed on real-world applications, necessitating risk-aware
agents. A common utility for learning such risk-aware
agents is the entropic risk measure, but current policy gradient methods optimizing this measure must perform high-variance and numerically unstable updates. As a result, existing risk-sensitive model-free approaches are limited to simple tasks and tabular settings. In this paper, we provide a comprehensive theoretical justification for policy gradient methods on the entropic risk measure, including on- and off-policy gradient
theorems for the
stochastic and deterministic policy settings. Motivated by theory, we propose risk-sensitive exponential actor-critic (rsEAC), an off-policy model-free approach that incorporates novel procedures to avoid the explicit representation of exponential value functions and their gradients, and optimizes its policy w.r.t the entropic risk measure. We show that rsEAC produces more numerically stable updates compared to existing approaches and reliably learns risk-sensitive policies in challenging risky variants of continuous tasks in MuJoCo.
Read the full abstract ▾
Machine Learning · Cross-listed from cs.SE
Automated Modernization of Machine Learning Engineering Notebooks for Reproducibility
Bihui Jin, Kaiyuan Wang and Pengyu Nie
Interactive computational notebooks (e.g., Jupyter notebooks) are widely used in machine learning engineering (MLE) to program and share end-to-end pipelines, from data preparation to model training and evaluation. However, environmental erosion-the rapid
evolution of hardware and software ecosystems for machine learning-has rendered many published MLE notebooks non-reproducible in contemporary environments, hindering code reuse and scientific progress. To quantify this gap, we study 12,106 notebooks selected from 75 popular Kaggle competitions: only 26% remain reproducible today. Crucially, we find that environment backporting, i.e., downgrading dependencies to match the submission time, does not improve reproducibility (decreased to 12%) but rather introduces additional failure modes. To address environmental erosion, we design and implement MLEModernizer, an LLM-driven agentic framework that treats the contemporary environment as a fixed constraint and modernizes notebook code to restore reproducibility. MLEModernizer iteratively executes notebooks, collects execution feedback, and applies three types of targeted fixes: error-repair, runtime-reduction, and score-calibration. Evaluated on 8,210 notebooks that are non-reproducible under the baseline environment, MLEModernizer makes 3,292 (40.1%, GPT-5.2) and 3,683 (44.9%, GPT-OSS-120b) notebooks reproducible. MLEModernizer presents a best-effort automated recovery and modernization technique that can improve reproducibility for a subset of notebooks. Practitioners can leverage MLEModernizer to validate, reuse, and maintain MLE artifacts as the hardware and software ecosystems continue to evolve.
Read the full abstract ▾
Machine Learning
Systematic Performance Assessment of Deep Material Networks for Multiscale Material Modeling
Xiaolong He, Haoyan Wei, Wei Hu et al. (+2 more)
Deep Material Networks (DMNs) are structure-preserving, mechanistic machine learning models that embed micromechanical principles into their architectures, enabling strong extrapolation capabilities and significant potential to accelerate multiscale modeling of complex microstructures. A key advantage of these models is that they can be trained exclusively on linear elastic data and then generalized to nonlinear inelastic regimes during online prediction. Despite their growing adoption, systematic evaluations of their performance across the full offline-online pipeline remain limited. This work presents a comprehensive comparative assessment of DMNs with respect to prediction accuracy, computational efficiency, and training robustness. We investigate the effects of offline training choices, including initialization, batch size, training data size, and activation regularization on online generalization performance and uncertainty. The results demonstrate that both prediction error and variance decrease with increasing training data size, while initialization and batch size can significantly influence model performance. Moreover, activation regularization is shown to play a critical role in controlling network complexity and therefore generalization performance. Compared with the original DMN, the rotation-free Interaction-based Material Network (IMN) formulation achieves a 3.4x - 4.7x speed-up in offline training, while maintaining comparable online prediction accuracy and computational efficiency. These findings clarify key trade-offs between model expressivity and efficiency in structure-preserving material networks and provide practical guidance for their deployment in multiscale material modeling.
Read the full abstract ▾
All authors Xiaolong He, Haoyan Wei, Wei Hu, Henan Mao, C.T. Wu
Machine Learning
Latent Target Score Matching, with an application to Simulation-Based Inference
Joohwan Ko and Tomas Geffner
Denoising score matching (DSM) for training
diffusion models may suffer from high variance at low noise levels. Target Score Matching (TSM) mitigates this when clean data scores are available, providing a low-variance objective. In many applications clean scores are inaccessible due to the presence of latent variables, leaving only joint signals exposed. We propose Latent Target Score Matching (LTSM), an extension of TSM to leverage joint scores for low-variance supervision of the marginal score. While LTSM is effective at low noise levels, a mixture with DSM ensures robustness across noise scales. Across simulation-based inference tasks, LTSM consistently improves variance, score accuracy, and sample quality.
Read the full abstract ▾