Distributed Computing
BOOST: Concurrent Access to Host Memory and HBM to Accelerate LLM Inference
Anish Saxena, Jae Hyung Ju, Hritvik Taneja et al. (+4 more)
GPU
memory bandwidth and capacity limit throughput in large
language model (LLM) inference. The GPU
memory system consists of a primary tier of high-bandwidth
memory (HBM) and a secondary tier of host
memory connected via CPU-to-GPU interconnect. Current serving systems treat the tiers hierarchically: they serve exclusively from HBM when data fits, and otherwise prefetch data from host
memory to HBM before use. In both cases, the host
memory bandwidth is never well utilized. Prefetching expands capacity by utilizing host
memory, but consumes HBM bandwidth for writes, reducing the bandwidth available for demand loads. We observe that fully utilizing both host and HBM bandwidth requires each wave of GPU threadblocks to access both tiers concurrently and in proportion to their bandwidth ratio. Existing bandwidth-proportional placement strategies fail to provide concurrency because they are not aware of GPU waves, and the large 2MB GPU page size. This paper presents BOOST, the first runtime system that provides concurrent and proportional access to both GPU
memory tiers, extracting the combined bandwidth of host
memory and HBM for LLM inference without kernel changes. The key insight in BOOST is to use kernel access patterns to make page allocation and runtime data management wave-aware. For static model weights, BOOST applies modulo-based page placement that eliminates access-ratio variance; for dynamically provisioned
attention key-value (KV) pairs, it makes the free KV page pool wave-aware. We integrate BOOST into vLLM and evaluate on a Grace Hopper system. At iso-batch size, BOOST improves Time-per-Output-
Token (TPOT) by 4.3% over HBM-only serving, whereas prefetching degrades TPOT by 6%. In high-throughput serving, BOOST improves throughput by 31% on average, outperforming prefetching by 15%.
Read the full abstract ▾
All authors Anish Saxena, Jae Hyung Ju, Hritvik Taneja, Po-An Tsai, Aamer Jaleel, Christos Kozyrakis, Moinuddin Qureshi
Distributed Computing
mKernel: Fast Multi-GPU, Multi-Node Fused Kernels
Ziming Mao, Yihan Zhang, Shawn Wei Chew et al. (+5 more)
Communication has become a bottleneck in distributed training and inference of large models. Overlapping communication with computation at the granularity of kernels, on separate streams, reduces only part of this communication cost. Fused kernels often have better performance by transmitting each output tile as soon as it is produced, but existing fused kernels are largely confined to a single NVLink domain. We present mKernel, a library of multi-GPU, multi-node fused kernels that overlap computation, intra-node NVLink communication, and inter-node RDMA at tile granularity. mKernel partitions the streaming multiprocessors (SMs) of a persistent kernel into compute and communication roles, and an on-GPU controller tunes the SM partition adaptively at run time, since the best SM partition varies with the kernel and the input shape. It structures data movement hierarchically so that data traversing the inter-node network is minimized. Finally, it drives the network from the GPU through a lightweight command queue and host proxy implemented directly on RDMA verbs, which allows the same kernels to run on any network backend (e.g. InfiniBand and on AWS EFA); we observe, surprisingly, that GPUDirect Async (IBGDA) yields little additional benefit over host-assisted GPU-initiated communication. We implement five kernels spanning tensor, sequence, and expert parallelism. On two 16-GPU H200 clusters, mKernel achieves speedups of up to 1.72x on GEMM+AllReduce and $1.88\times$ on Ring
Attention.
Read the full abstract ▾
All authors Ziming Mao, Yihan Zhang, Shawn Wei Chew, Shuang Ma, Costin Raiciu, Yang Zhou, Scott Shenker, Ion Stoica
Distributed Computing
Object Model Analysis of a Supercomputer with Digital Twin
Shilpika Shilpika, George K. Thiruvathukal, Venkatram Vishwanath et al. (+1 more)
Operators and developers need a mental model of both the structure and the live behavior of a large supercomputer, but its physical layout, logical organization, and streams of per-node telemetry are difficult to relate to one another, making it hard to trace a metric or event back to a specific hardware component. We present DAT, an interactive three-dimensional digital analytics twin of a compute cluster built in a real-time game engine, Unreal Engine. DAT expands a compact, parametric description of a supercomputer, a reusable Digital Twin Prototype (DTP), into a navigable Digital Twin Instance (DTI) that mirrors its physical containment hierarchy of racks, chassis, blades, and network links, encoding each node's role and health in its appearance, while a lightweight event-driven simulator animates job and hardware activity over a virtual clock. Our current implementation adds a two-path node-selection mechanism, unifying direct 3D pointing with command-shell queries, that opens an in-world visual-analytics panel beside any selected component showing summary statistics and live, time-varying metrics. We describe this architecture, report qualitative behavior from the working prototype, and outline the path toward driving the panels with recorded telemetry and in-situ anomaly detection.
Read the full abstract ▾
All authors Shilpika Shilpika, George K. Thiruvathukal, Venkatram Vishwanath, Michael E. Papka
Distributed Computing
The Essence of Git: Concepts for P2P Replication
Erick Lavoie
Git is most commonly known as a decentralized version control system. Underneath the surface, the design of Git exhibits an elegant object model and incremental replication protocols that have been used to build a variety of applications that go much beyond version control. In this paper, we survey those applications and show how application concepts have been mapped to Git concepts to show the latter's generality. We also show the connection between the Git commit history and the theory of distributed
algorithms and convergent replicated data structures, suggesting that it approaches their essence. We finally compare to current peer-to-peer frameworks to establish the level of abstraction of the Git object model and peek at its potential universality. Throughout, we frame our discussion using the conceptual design approach proposed by Daniel Jackson.
Read the full abstract ▾
Distributed Computing
PEAT: Pseudo-Error Assessment for GPU Kernel Validation in DNN Training
Xuan Truong Nguyen, Hong Quan Tran, Tuan Duc Chu et al. (+1 more)
Deep neural networks (DNNs) are widely adopted in various fields, driving an emerging trend in developing software stacks associated with DNN training systems. For example, many codes have been ported across different frameworks or developed to leverage the computing power of GPUs or domain-specific accelerators. However, validating a kernel implementation in DNN training is time-consuming and generally requires massive storage. Specifically, this poses a fundamental question: how to characterize the behavior of a new implementation when it is integrated into a DNN training flow. Unfortunately, this problem is not well investigated in the literature, to the best of our knowledge. To address this shortcoming, we present PEAT - a lightweight inspection framework for \underline{P}seudo-\underline{E}rror \underline{A}ssessment associated with GPU kernel validation in DNN \underline{T}raining. Firstly, inspired by conventional fault injection (FI), PEAT's Profiler invokes an operation-wise kernel in a training flow to collect a DNN model's states (e.g., checkpoints and activations). More importantly, the Profiler introduces two simple yet effective techniques, playback FI and frequency-based runtime FI, leveraging persistent kernel calling during the training process. Secondly, PEAT's Analyzer characterizes profiled errors, revealing some signatures from the error distribution of a kernel compared to the golden one. Lastly, PEAT's Detector provides some guidelines as a sufficient condition, which enables associating several well-known error models with signature patterns. We demonstrate the applicability of our approach by presenting the results and analysis using GPUs from the two most popular vendors, NVIDIA V100 and AMD MI250, on various AI models, from
vision tasks to
language models, for both pretraining and finetuning scenarios.
Read the full abstract ▾
All authors Xuan Truong Nguyen, Hong Quan Tran, Tuan Duc Chu, Thanh Tuan Dao
Distributed Computing
Dynamic HBM Repartitioning for Multi-Turn MoE Serving
Jinpyo Kim, Mingi Kwon, Younghoon Min et al. (+2 more)
Long-running multi-turn requests accumulate reusable key-value (KV) state. Once this state exceeds a fixed GPU KV-cache allocation, serving systems evict reusable prefixes, repeat prefill work, and may preempt requests. This pressure is particularly acute for Mixture-of-Experts (MoE) models: their expert weights occupy most GPU high-bandwidth
memory (HBM), even though each
token activates only a sparse subset of experts. A static boundary between weights and the KV cache prevents serving systems from using expert
memory to preserve reusable state as conversations grow. We present VAMP, an MoE serving framework that changes this HBM boundary at runtime. When a KV allocation cannot be satisfied, VAMP compares the estimated future work of three alternatives: staging expert weights from host
memory, evicting cached prefixes that may require re-prefill, or preempting and rescheduling requests. It then converts a bounded expert-weight region into KV-cache capacity when that action has the lowest estimated penalty. CUDA Virtual
Memory Management page remapping performs this conversion without copying resident KV data. We implement VAMP in the vLLM serving engine and evaluate Qwen3-Next-80B on recorded and controlled multi-turn workloads. Across five replays of a recorded 2,103-turn SWE-bench
agent workload, VAMP with a 15% maximum expert-offloading ratio reduces time-to-first-
token (TTFT) p90 from 26.1 s to 1.10 s (23.6 times) and increases request throughput by 20.7% relative to unmodified vLLM, while increasing time per output
token (TPOT) by 31.1%.
Read the full abstract ▾
All authors Jinpyo Kim, Mingi Kwon, Younghoon Min, Jongryool Kim, Jishen Zhao
Distributed Computing · Cross-listed from cs.LG
A Measured Communication-Quality Frontier for Federated LoRA Fine-Tuning with Adaptive Phase-Switching
Jerry Adams Franklin
Federated fine-tuning of large
language models with low-rank adaptation (LoRA) reduces the number of trainable parameters, but communication remains the dominant cost, and protocols are usually compared by parameter-count ratios rather than by measured bytes. This paper measures per-round upload and download bytes for five
federated LoRA protocols, three of them from prior work, and places them on a single communication-quality frontier scored by held-out instruction-following loss. The frontier has a knee. ReverseAdaptive, which learns both LoRA factors before freezing one once the relative improvement in training loss falls below a dimensionless threshold, sits at that knee: it cuts measured round-trip communication by 40.5% relative to FLoRA at a held-out loss cost of 0.0063, and beats FFA-LoRA, which freezes that factor at initialization, by 0.0182 in held-out loss, more than twenty times the largest per-method seed standard deviation. The same threshold carries to LLaMA-3.2-3B without retuning, where it saves 30.0%.
Read the full abstract ▾
Distributed Computing · Cross-listed from cs.DB
COMPASS: Steering Distributed Vector Search with Scientific Knowledge Graphs
Song Young Oh, Amal Gueroudji, Seth Ockerman et al. (+5 more)
Vector databases use hashing to partition data across "shards," logical units for distributed execution. This placement, however, destroys semantic locality, forcing each query into scatter-gather limited by the slowest shard. Vector-space clustering can help, but scientific evidence is often connected by factual relations that do not align with
embedding distance. We present COMPASS, a framework that uses a knowledge graph (KG) to determine data placement and query-time shard selection. COMPASS detects communities, splits oversized communities, inserts
embeddings by subject entity, and routes queries to a small set of shards. Across four biomedical KGs, our method searches only 13-18% of the corpus while preserving broadcast recall and recovering up to 2.6x more multi-hop evidence than an
embedding-based baseline. On 15 HPC nodes, COMPASS sustains 7.9x higher throughput with lower tail latency than hash-based broadcast. These results show that KG structure provides a compact complement to
embedding geometry for scalable vector search.
Read the full abstract ▾
All authors Song Young Oh, Amal Gueroudji, Seth Ockerman, Rob Latham, Orcun Yildiz, Ian Foster, Kyle Chard, Robert Ross
Distributed Computing · Cross-listed from cs.CE
Extreme-Scale Linear-Scaling Kohn-Sham DFT at 100 Million Atoms: Bridging Quantum Simulations and Experiments
Qimen Xu, Yu Zhang, Dixing Ni et al. (+11 more)
Kohn-Sham density functional theory (DFT) remains the workhorse of ab initio materials simulation, yet cubic computational and quadratic
memory scaling have confined calculations to a few hundred to thousands of atoms, spanning only nanometers, far below experimentally relevant length scales. We introduce XLSDFT, a linear-scaling DFT framework based on divide-and-conquer decomposition of the one-particle density
matrix and Chebyshev-filtered subspace iteration, achieving linear computational and
memory scaling while retaining DFT accuracy. Deployed on the LineShine exascale supercomputer, XLSDFT reduces computational complexity by orders of magnitude, enabling unprecedented DFT scale: a 200-million-atom silicon
crystal, twentyfold beyond the prior record. Our implementation achieves 96.6% weak-scaling efficiency and sustained 157.9 Pflop/s (FP64) for a 100-million-atom scaling study. We further simulate an 11-million-atom all-solid-state battery interface of unprecedented complexity, 1,000 times beyond prior DFT for such systems, revealing how lithium metal reacts with the solid electrolyte at atomic resolution, in quantitative agreement with spectroscopy experiments.
Read the full abstract ▾
All authors Qimen Xu, Yu Zhang, Dixing Ni, Lei Gao, Guangnan Feng, Qinrui Zheng, Jianting Liu, Haitian Lu, Zhaopeng Jia, Wei Xue, Shriram Chandran, Torsten Hoefler, Haohuan Fu, Yutong Lu
Distributed Computing · Cross-listed from cs.CE
NFT-Based Reward Mechanisms: Sybil Farming, Vesting, and Stochastic Verification
Marco Alberto Javarone, Stefanos Leonardos and Carmine Ventre
We study NFT-based reward mechanisms in which a user can create multiple identities and submit fraudulent claims that mature a reward subject to vesting. We assume that the issuer stochastically verifies claims during the vesting period and that identities can be linked into clusters so that the detection of one identity submitting a fraudulent claim causes the whole cluster to be forfeited through a penalty. A farmer's payoff is then non-linear in the number of identities: rewards increase linearly, while the probability of avoiding detection decreases geometrically. We characterise the optimal farming scale in the continuous relaxation of the problem. This allows us to derive a sufficient condition for deterrence and then consider the issuer's choice of reward vesting and claim verification. Vesting reduces the probability that a fraudulent claim is paid but also affects genuine participants, while verification is costly for the issuer. We characterise the sufficient deterrence frontier for the issuer in terms of auditing cost, vesting schedule, and penalty value. When small amounts of audit capacity can be added at negligible marginal cost, vesting alone is not optimal. We also distinguish the role of penalties from that of cluster-level auditing. Finally, we discuss the implications for NFT reward programmes on high-throughput ledgers, such as Hedera.
Read the full abstract ▾
Distributed Computing
A Dynamic Vertical Scaling Strategy for Distributed Stream Processing Applications in Edge Computing
Guilherme Hiago Costa dos Santos, Carlos Henrique Kayser and Tiago Coelho Ferreto
Distributed Stream Processing applications at the edge must reconcile low latency and high throughput with limited and heterogeneous resources. This paper presents a dynamic vertical scaling strategy based on Proximal Policy Optimization, formulated as a Partially Observable Markov Decision Process. The policy jointly adjusts task allocations and prioritizes compliance with a p95 end-to-end latency Service Level Objective. In EdgeStreamPy simulation experiments with two application profiles, two workloads, and ten paired placements per combination, the selected policies preserved throughput, obtained mean violation rates from 0.03% to 0.30%, below VRebalance in every scenario, and used less CPU in three of four combinations. The comparison covers complete controller configurations with different decision frequencies.
Read the full abstract ▾
Distributed Computing · Cross-listed from cs.PF
Dissecting GPU Utilization for LLM Inference on Nvidia Hopper
Mohammad Siavashi, Gerald Q. Maguire Jr., Dejan Kostic et al. (+1 more)
A single SM utilization percentage can make an LLM inference workload look compute-saturated while hiding how much useful work is being done. The problem is not that the counter is wrong, but that it collapses several different mechanisms into one number. This is most severe during decode, where each request contributes only one new
token and dense projection GEMMs become small-row
matrix multiplications. On Hopper, the bfloat16 GMMA path executes these operations in fixed 64-row
matrix fragments, so small-batch decode can fill only a small fraction of each fragment with real
token rows. In this paper, we profile vLLM with FlashAttention-3 and cuBLASLt on an H100 NVL across cold prefill, warm prefill, and decode, sweeping sequence length and batch size. We replace the usual single utilization number with eight counter-validated views derived from raw Nsight Compute reports, each pinned to an NCU counter or explicit formula. Together, these views map utilization gaps to concrete mechanisms - fragment fill, occupancy limits, stall signatures, wave quantization, and kernel selection - across four production models and six per-layer kernel roles.
Read the full abstract ▾
All authors Mohammad Siavashi, Gerald Q. Maguire Jr., Dejan Kostic, Marco Chiesa
Distributed Computing · Cross-listed from cs.LG
Hidden in Rounds: Predicting the Time Cost of 802.11 Contention in Federated Learning
Satwat Bashir and Tasos Dagiuklas
Federated learning over IEEE~802.11 shares the wireless channel among clients that send model updates. We use ns-3 to measure the frame-delivery ratio and saturation throughput for different client densities and offered loads. A separate FedAvg trainer uses the frame-delivery ratio as a first-order proxy for the update-admission probability and uses an equation to estimate communication time. The method does not simulate the delivery of a complete model update or measure end-to-end training time. Across 720 evaluated runs with two
datasets, two data partitions, six client densities, six offered loads, and five seeds, all runs reached their predefined target accuracy within the round budget. Rounds-to-target changed little with offered load, while communication time-to-target increased by about two orders of magnitude across the client-density range. A Bianchi-anchored
estimator produced a mean absolute percentage error from $2.3\%$ to $10.2\%$ on held-out configurations. This error is measured against communication time constructed from the same round-duration equation, not against independently measured completion time. We also compare uniform participation with persistent heterogeneous participation. The study does not detect a statistically distinguishable excluded-class accuracy gap over five seeds, but the confidence intervals are wide. The results apply only to the evaluated configurations and do not provide a general convergence or
fairness guarantee.
Read the full abstract ▾
Distributed Computing · Cross-listed from physics.ao-ph
4D Parallelism Unlocks Exascale Bayesian Neural Networks for High-Fidelity Atmospheric Modeling
Deifilia Kieckhefen, Juan Pedro Gutiérrez Hermosillo Muriedas, Lars Helge Heyen et al. (+12 more)
We present BEAST, the first-ever
Bayesian Swin
Transformer for atmospheric forecasting on 0.25$^\circ$ global resolution able to accurately quantify both aleatoric and epistemic uncertainty. To overcome the associated computational bottlenecks, we devise an orthogonal 4D-parallelization scheme that introduces a unique domain-tensor-parallelism strategy and a novel uncertainty parallel method, enabling us to fully leverage GPU capacity and efficiently scale model training. For a 2.4-billion-parameter model, we achieve a peak performance of 3.96 EFLOP/s on 20,480 NVIDIA GH200 GPUs on the JUPITER supercomputer. We train BEAST as a 700-million-parameter model with 96 random weight samples on 384 nodes on 40 years of data for nearly one million gradient updates. This model achieves predictive skill scores competitive with state-of-the-art probabilistic atmospheric AI models and numerical models, and can predict extreme events with exceptional skill, while generating large ensembles 3 to 4 times faster than the current-best AI model. Our contribution unlocks the potential of high-fidelity uncertainty quantification in atmospheric AI models, heralding a new era for AI-based models in
climate and Earth system sciences.
Read the full abstract ▾
All authors Deifilia Kieckhefen, Juan Pedro Gutiérrez Hermosillo Muriedas, Lars Helge Heyen, Mathis Bode, Iida Hakulinen, Andreas Herten, Chelsea Maria John, Thorsten Kurth, Anni Moisala, Asena Karolin Özdemir, Kaleb Phipps, Oskar Taubert, Arvid Weyrauch, Markus Götz, Charlotte Debus
Distributed Computing · Cross-listed from cs.NI
Hybrid Monitoring for Early Fault Detection in Cloud-Native 5G Systems
Anton Andersson, Sai Akshara Naineni, Mats Jansborg et al. (+2 more)
This paper presents the design implementation and evaluation of NetMon a hybrid network monitoring system designed for Kubernetes-based 5G packet core deployments specifically evaluated on Ericssons Access and Mobility Management Function AMF clusters NetMon combines eBPF-based passive kernel-level traffic observation with active TCP probing and centralized correlation to detect and localize network degradation within seconds The evaluation results demonstrate that the system detects faults as subtle as 10ms of added latency or 5 packet loss correctly attributes them to the affected infrastructure component and maintains this capability under application loads up to 50 simulated UE load The total resource overhead of 34 millicores CPU and 45 MiB
memory per pod suggests that the approach is promising for further validation without impacting the monitored workload The hybrid approach addresses a gap in existing monitoring tools standard health checks cannot detect partial degradation scrape-based systems introduce detection delays measured in tens of seconds and purely passive tools cannot verify idle network paths By combining these complementary techniques and centralizing the analysis the system provides the early detection and fault localization capabilities required for maintaining service quality in cloud-native 5G infrastructure.
Read the full abstract ▾
All authors Anton Andersson, Sai Akshara Naineni, Mats Jansborg, Yixing Zhang, Romaric Duvignau
Distributed Computing · Cross-listed from cs.CR
A Feature-Rich Embedded NIDS with eBPF/XDP: Detector and Architecture Trade-offs
Shiqi Wu, Oleksii Koshovyi, Georgios Pseiridis Pseiras et al. (+2 more)
Distributed Denial-of-Service (DDoS) attacks remain a serious threat to transport networks, with recent attack volumes exceeding 30 Tbps, and the telecommunications industry being the main target. Recent work has yet to study the impact of the hosting software architecture on network monitoring solutions, or to assess recent
algorithms for improving attack detection. This paper presents a Network Intrusion Detection System (NIDS) for DDoS detection in transport networks, developed in collaboration with Ericsson. Building on a statistical baseline, we improve detection effectiveness with an Isolation Forest trained on a wider set of flow features, extracted by GoFlowMeter, our open-source Go implementation of CICFlowMeter, and we integrate eBPF/XDP so that the NIDS filters real traffic at the kernel level. We further compare three deployments, monolithic, Kafka-based, and gRPC-based microservices, on a Raspberry Pi 5 testbed replaying the CIC-DDoS2019
dataset as real network traffic. Detection quality is governed mainly by the choice of detector rather than by the transport: the Isolation Forest raises recall and F1 score (0.965 live in the monolithic variant) over the baseline by flagging low-volume attack windows that the baseline misses. The transport is not neutral, however: gRPC reaches almost the same accuracy as the monolithic variant while adding less than 2 milliseconds of transport time per window, whereas the asynchronous Kafka pipeline trails by roughly nine percentage points and adds about 27 milliseconds. These findings clarify the trade-off between detection quality and architectural overhead when deploying a NIDS on resource-constrained hardware.
Read the full abstract ▾
All authors Shiqi Wu, Oleksii Koshovyi, Georgios Pseiridis Pseiras, Victor Morel, Romaric Duvignau
Distributed Computing
GreenDirector: carbon- and water-aware workload placement for sustainable computing
Jaime Iglesias Blanco, Ignacio Heredia, María Castrillo et al. (+3 more)
The rapid growth of data center electricity demand, accelerated by AI, makes carbon-only accounting an incomplete measure of computing's environmental impact: low-carbon electricity mixes are often water-intensive, and the resulting harm depends on local, seasonal scarcity rather than on the volume of water consumed. We propose the Environmental Score (ES), a unified, dimensionless index in $[0, 100]$ that jointly captures the carbon footprint and the spatial-temporal, scarcity-weighted water impact of the electricity a workload consumes. It combines real-time, cross-border electricity flow tracing with monthly AWARE2.0 water-scarcity characterization factors, weighting global greenhouse-gas emissions together with the local, seasonal severity of water stress. Building on it, we define the Green Score (GS), a scheduling metric proportional to the useful computational work delivered per unit of real environmental impact, which also accounts for data center power and hardware efficiency. We add both metrics as a green-affinity feature to the GreenDirector schedulers of two production
federated infrastructures, the AI4EOSC scientific cloud and the DIRAC workload management system. In AI4EOSC, a cluster-filling experiment over four pan-European providers shows that greener sites are filled first without degrading scheduling latency or end-user experience. In DIRAC, a trace-driven simulation of 133,631 jobs and a preliminary production deployment for the KM3NeT community reduce carbon emissions and improve carbon efficiency by about 40%, while making the carbon-water trade-off explicit when the lowest-carbon site also carries higher water stress. The results show that hydrological stress can be dynamically weighted into workload placement in live, multi-tenant systems
Read the full abstract ▾
All authors Jaime Iglesias Blanco, Ignacio Heredia, María Castrillo, Andrei Tsaregorodtsev, Mazen Ezzeddine, Álvaro López García
Distributed Computing · Cross-listed from cs.CR
NovaFabric: Tamper-Evident, Replayable Evidence for Autonomous AI Agent Runs
Mohsen Seyedkazemi Ardebili
When an autonomous AI
agent does something consequential, what can be proven about what it did?
Agent-observability platforms capture traces, but a trace is mutable: alterable undetected, with no recipe for re-executing it, silent on whether captured secrets were removed. Regulation (EU AI Act, ISO 42001, NIST AI RMF) presumes records an independent party can check. We present NovaFabric, producing audit-grade execution evidence: provider-neutral, tamper-evident, replayable, shareable. It records an
agent run, without modifying
agent logic, into a portable Run Capsule (fifteen-entity schema), sealed with a holistic DSSE signature, RFC 3161 timestamp, Merkle log and redaction attestation. Sealed runs are re-executable under a four-mode replay protocol and exportable as an Evidence Bundle for third-party verification with stock tooling (specified, not evaluated). The contribution is integration, not new cryptography: OpenTelemetry, DSSE/in-toto and W3C PROV. We evaluate eight research questions at measured scope. Mocked replay serves every model response from the capsule (no live model call, 10/10) but is offline w.r.t. models, not the network; only 2/10 tool-using workloads completed; the gap is missing tool-response substitution. Tampering is rejected across three tested classes. Declared-stream completeness is 0.652 (95% CI +/-0.064, ten scenarios). A repaired rule pack redacts 14/14 credential types, preserving 9/9 decoys; diff localises 140/140 mutations. Blast-radius queries: 45.5ms p99 over 10M edges (3.3x faster than a columnar baseline), 167.9ms over 100M (1 client, n=30). A 314-machine, ten-region run finds capsule REST ingest lossless but capped at 61.6 req/s (p99 26.8s) by per-worker serialisation. Six defects found in NovaFabric and its evaluation corpus: four fixed, one withdrawn, one open. Verification is conditional on a stated trusted computing base.
Read the full abstract ▾
Distributed Computing
RoofLang: Enabling AI-Driven Architecting of LLM Inference Systems
Ziyue Yang, Yuting Jiang, Lei Qu et al. (+1 more)
AI is beginning to make substantive contributions to LLM inference optimization. Existing AI optimizations are predominantly profiling-based. Profiling-bound feedback confines the search to the capabilities and performance of an existing software stack, preventing a fundamentally better architecture of LLM inference systems from being identified. To enable the AI-driven LLM inference system architecting loop, we argue that a general workload representation, a verifiable mutation space, and an implementation-independent evaluator are required. We present the RoofLang domain-specific
language (DSL) that provides these features. In our evaluation, RoofLang reveals that DeepSeek V4-series models could achieve 3.5-39.5$\times$ higher peak decode throughput than other representative models. This gap is disproportionate to their total parameter counts and arises largely from compact KV-cache designs that support larger batches and reduce
memory traffic. A persistent optimizer
agent further discovered several new architectures that improved both throughput and interactivity of DeepSeek V4 Pro on NVIDIA B300 by 6.23-50.1%.
Read the full abstract ▾
All authors Ziyue Yang, Yuting Jiang, Lei Qu, Peng Cheng
Distributed Computing
HeatCache: Thermal-aware Energy-efficient LLM Inference Scheduling for Chassis-level Liquid Cooling in Sustainable Edge Server Rooms
Rui Lu, Huanghuang Liang, Kaiqi Guan et al. (+1 more)
LLM inference is increasingly deployed at institution-scale edges to meet service requirements. However, multi-GPU inference consumes a large amount of electricity and produces substantial heat. To improve sustainability, operators and regulations often demand raising the ambient setpoint to reduce cooling electricity. This can increase thermal throttling and hardware aging, leading to Service-Level Objective violations. In this paper, we present HeatCache, a thermal-aware, energy-efficient LLM inference scheduler for commercial chassis-level AIO
liquid-cooled GPUs at sustainable ambient temperatures. HeatCache treats AIO loops as a temporary heat buffer, measured by heat budget and schedules requests to minimize energy subject to thermal safety and SLO constraints, based on an electrical-informed heat-demand estimation from HeatiTS. We implement HeatCache atop vLLM and show that it reduces computing energy by up to 18.0%, decreases thermal-throttle exposure by 81.7%, and maintains SLO violation rates below 0.9% even up to $48~^{\circ}\mathrm{C}$.
Read the full abstract ▾
All authors Rui Lu, Huanghuang Liang, Kaiqi Guan, Dan Wang
Distributed Computing
HoliBench: A Cross-Platform Benchmarking and Deployment Toolkit for Foundation Models in CPS-IoT Applications
Inesh Chakrabarti, Zejun Xiong, Pragya Sharma et al. (+1 more)
Foundation models, including large
language models,
vision-
language models, and time-series foundation models, are increasingly deployed on embedded and edge platforms for CPS and IoT applications, where energy, latency, and
memory are as critical as task accuracy. Existing benchmarking tools evaluate model capability in isolation, reporting accuracy assuming sufficient compute, while hardware profiling tools remain platform-specific and mutually incompatible. As a result, users lack a unified workflow for making deployment decisions across heterogeneous devices. We present HoliBench, a modular benchmarking and deployment toolkit that jointly characterizes accuracy, latency, and energy across platforms from single-board computers to GPU servers. Its platform abstraction layer calibrates cross-device measurement, and the toolkit supports multiple model modalities, inference engines, concurrencies, and existing evaluation harnesses. An interactive interface exposes constraint-aware configuration selection over a design space that is profiled once and reused across studies. Using HoliBench, we characterize 20 models across 7 device types, 3 quantization levels, 8 inference backends, and over 30 tasks, surfacing tradeoffs that existing tools miss: quantization reduces latency only on hardware with low-precision support, accuracy gains show diminishing returns relative to energy, and for autoregressive workloads, average inference power is approximately constant across output lengths. We further find that single-model profiles compose under sequential co-resident execution. In a multi-model CPS deployment, standalone profiles predict combined-pipeline latency and power within 1.2% and 2.5%, enabling deployment exploration without exhaustively profiling every pipeline configuration. We release HoliBench as open-source infrastructure for deployment-aware evaluation of foundation models.
Read the full abstract ▾
All authors Inesh Chakrabarti, Zejun Xiong, Pragya Sharma, Mani Srivastava
Distributed Computing · Cross-listed from cs.AI
OneLA: Scaling Linear-Attention Decoding to Large Beams in Generative Recommendation
Xiangrui Yang, Cheng Peng, Yunfeng Zhao et al. (+9 more)
Generative recommendation (GR) relies on large-beam decoding to generate hundreds of candidate items, creating a new scaling challenge for recurrent linear
attention. Existing linear
attention serving systems either materialize a full recurrent state for every beam or repeatedly replay shared history, incurring substantial
memory and traffic overhead. To address this, we present OneLA, a linear-
attention decoding framework that exploits the shared
prompt and short divergent suffixes of GR workloads. Specifically, OneLA represents all beam states using a single shared
prompt-derived state and compact, append-only records of their divergent transitions. Using this representation, OneLA computes only the state information required at each decoding step, without reconstructing a full recurrent state for every beam. Furthermore, OneLA uses a lightweight ancestry index to track the transition records that make up each beam's history, allowing beams to be updated without moving or copying existing records. A fused GPU kernel further reuses the shared state across beams. Our analysis shows that OneLA achieves 1.54-2.46x end-to-end decode speedups while substantially reducing recurrent-state
memory use and data movement.
Read the full abstract ▾
All authors Xiangrui Yang, Cheng Peng, Yunfeng Zhao, Liang Zeng, Ao Hu, Jiawei Yang, Shengzhe Wang, Jingshan Lv, Xiao Liang, Chen Yang, Jiaqiang Liu, Yiming Qiu
Distributed Computing
ForgeMegakernel: A General Framework for Efficient Auto-Regressive Model Decode Megakernels
Leshan Li, Zhui Zhu, Xianglong Deng et al. (+6 more)
Auto-regressive model decode is bandwidth-bound, since every weight and key/value-cache byte crosses high-bandwidth
memory once per
token. A megakernel is an ideal solution, but existing automatic megakernel generation approaches cannot achieve both generalization across models and correctness guarantees. We present ForgeMegakernel, which generates a per-model high-performance decode megakernel using coding
agents. ForgeMegakernel pairs a universal knowledge base of ten progressive milestones with an independent mid-state test oracle. The milestones provide the megakernel's structural properties: a fine-grained instruction stream for each SM, dependency counters replacing the global synchronization, and a shared-
memory buffer pool for workload balance across SMs and greater parallelism. The test oracle derives the mid-states of the megakernel and checks the performance, error and precision during the generation process, guaranteeing a correct and trustworthy forged megakernel. We evaluated ForgeMegakernel on 14 representative decoding operations across eight model families spanning 0.6B-13B parameters. The generated megakernels achieved 50.5-85.9% MBU and
geometric mean speedups of 1.21x over SGLang 0.5.18 and 1.54x over a megakernel compiler under identical configurations. Inside SGLang, evaluated on GSM8K with ragged
prompts, all 14 megakernels decoded faster than the SGLang engine at comparable answer accuracy.
Read the full abstract ▾
All authors Leshan Li, Zhui Zhu, Xianglong Deng, Yaojian Chen, Qingfeng He, Yuxuan Li, Rong Zhao, Xu Han, Zhiyuan Liu
Distributed Computing
Unleashing the Power of Equality Saturation for Tensor Program Superoptimization
Qi Zhan, Xing Hu, Xin Xia et al. (+1 more)
Efficient GPU implementations of tensor programs often require joint optimization of high-level algebraic formulations and low-level execution strategies. However, the resulting search space grows rapidly as transformations combine across operators, making joint optimization difficult to scale. We present EqiForge, a tensor program superoptimizer based on equality saturation. Its unified IR represents high-level tensor expressions and tiled computations in a single expression
language. By composing equality rules, EqiForge derives fused implementations such as FlashAttention-style kernels directly from tensor expressions. Early compaction prunes redundant partial programs before completion, while subgraph composition extends the search to larger graphs. Across tensor-program
benchmarks, EqiForge achieves a
geometric mean speedup of 1.32x and a maximum of 2.74x over the fastest available baseline per configuration. Its
attention kernels outperform FlashAttention by up to 1.87x in decode and approach its performance in prefill. EqiForge also discovers new implementations that outperform torch.compile on various
Transformer layers, including QK-normalized MLA (3.16x) and mHC (5.84x).
Read the full abstract ▾
All authors Qi Zhan, Xing Hu, Xin Xia, Shanping Li
Distributed Computing
Argus: Orchestrating Cross-Layer GPU Performance Measurements around Semantic Regions
Jianzhu Yao, Yue Guan, Srivatsan Ramesh et al. (+11 more)
GPU developers and automated optimizers need performance evidence for semantic code regions–such as neural-network operator implementations and pipeline stages–but this evidence is fragmented across profiling tools. Answering a region-level question can require manually constructing probes and program variants, isolating interfering measurements, and mapping evidence to regions and execution contexts. We present Argus, a region-centric measurement planner and runtime that automates this workflow. Clients identify regions with boundary markers and select signals and execution scopes. Argus preserves region identity across compilation, execution, and measurement variants, constructs interference-aware multi-run plans, and orchestrates transformations and profiling across backends. It joins compiler-, hardware-, and system-level evidence using region identity and dynamic execution context, producing reports that record measurement origins and attribution ambiguity. We evaluate Argus across agentic kernel optimization, persistent megakernel optimization, and cross-level PGO. Across 44 persistent-GEMM and
attention configurations, Argus improves 39/44 cases and raises AlphaEvolve's
geometric-mean speedup from 5.4% to 8.9%. On a persistent TinyLlama-1.1B decode megakernel, an optimization
agent reaches 1.65 ms/
token with Argus versus 4.92 ms/
token without it, producing a kernel $2.1\times$ faster than PyTorch with CUDA Graphs. Finally, Argus-guided cross-level PGO improves compute–communication overlap, increasing throughput by 7% on average across five multi-GPU settings.
Read the full abstract ▾
All authors Jianzhu Yao, Yue Guan, Srivatsan Ramesh, Yuanwei Fang, Jian Jiao, Boda Li, Yueming Hao, Xinwei Qiang, Pramod Viswanath, Yufei Ding, Bill Yoshimi, Alexey Loginov, Shane Nay, Adnan Aziz