Information Theory · Cross-listed from stat.ML
The optimal information complexity of VC learning
Steve Hanneke and Juexiao Wang
Steinke and Zakynthinou(2020) introduces the Conditional Mutual Information (CMI) framework of analyzing the information complexity of learning
algorithms based on
algorithm-dependent information-theoretic quantities. We study one of these quantities, the evaluated Conditional Mutual Information (eCMI). It has been an interesting question whether the optimal PAC guarantee for VC classes can be recovered from the
algorithm-dependent analyses via CMI. And we show that it is possible to recover this guarantee by constructing a learning
algorithm whose eCMI is of order O(d) in the realizable case, where d is the VC-dimension of the concept class. Specially, our
algorithm is a randomized Majority-of-5 base learners with optimal in-expectation generalization guarantee.
Read the full abstract ▾
Information Theory
Performance Analysis and Sensing-Aware Resource Allocation for Full-Duplex Massive MIMO ISCC Networks
Vo Phi Son, Van-Dinh Nguyen, Nhan Thanh Nguyen et al. (+2 more)
Emerging edge applications must sense their environment and process data within tight latency budgets. Full-duplex (FD) integrated sensing, communication, and computing (ISCC) supports these functions concurrently, but radar probing interferes with uplink reception while user transmissions degrade target estimation. These coupled effects call for joint sensing and offloading decisions. We investigate a FD massive multiple-input multiple-output (mMIMO) ISCC network in which users partially offload tasks to an edge server while an access point probes a target. For zero-forcing uplink reception with imperfect channel estimates, we derive a tractable rate bound and target-angle Cramer-Rao lower bounds (CRLBs) under the stated interference and beam-
alignment assumptions. We then jointly optimize offloading fractions, user and radar transmit powers, and edge computing resources to minimize average end-to-end task latency subject to sensing-accuracy, communication-rate, energy, deadline, and resource constraints. A successive convex approximation
algorithm is developed to efficiently address the resulting nonconvex problem. The analysis shows that larger arrays improve sensing and uplink performance, but residual radar interference creates a finite rate ceiling when user powers scale down with array size. Simulation results show joint allocation reduces radar transmit power by approximately 86% relative to equal communication-user power at a -40 dB CRLB threshold and lowers latency compared with fixed-allocation and half-duplex schemes.
Read the full abstract ▾
All authors Vo Phi Son, Van-Dinh Nguyen, Nhan Thanh Nguyen, Tuan-Vu Truong, Symeon Shatzinotas
Information Theory · Cross-listed from eess.SY
Mission-critical spectrum sharing with decentralized Multi-Agent Reinforcement Learning
Dimitrios Pylorof, Imtiaz Nasim, Humberto E. Garcia et al. (+3 more)
Motivated by emerging mission-critical applications and an increasingly congested spectrum, we develop a decentralized multi-
agent reinforcement learning (MARL) model for dynamic spectrum access. The model enables secondary users to learn effective transmission strategies across shared frequency bands while minimizing collisions with high-priority primary users and among themselves. We design the
agent-level learners following a Markov potential game approach, connecting independent local updates to system-level improvement. We instantiate this design using lightweight linear actor-critic learners suitable for resource-constrained edge devices, rather than computationally intensive centralized or deep multi-
agent architectures. Across spectrum environments with different incumbent activities, the learned policies adapt their transmission policy and waiting behavior to preserve throughput while greatly reducing transmission collisions relative to random and forecast-aware heuristic baselines. The results establish the value of decentralized MARL and shows up to 96.8% reduction in overall collisions.
Read the full abstract ▾
All authors Dimitrios Pylorof, Imtiaz Nasim, Humberto E. Garcia, Vivek Agarwal, Jasni A. Mannil, Mingyue Ji
Information Theory · Cross-listed from cs.LG
An Accuracy-Information Tradeoff for Loss-Difference Conditional Mutual Information
Hazar Yueksel
Loss-difference conditional mutual information (ld-CMI) uses the smallest of the standard observations in the supersample hierarchy of generalization bounds: it measures what a learner's loss differences reveal about which candidate of each pair it was trained on. Accuracy is known to force information into the model; data processing does not carry such lower bounds to losses. We show, by bounding three moments of the loss differences, that accuracy also forces ld-CMI. For linear predictors with a smooth convex loss of nonzero slope at zero, such as the logistic loss, plus a regularizer whose
curvature and growth are both of power $r\ge2$, on product distributions over a scaled sign cube in dimension at least linear in $n$, every proper learner with expected excess risk at most $\varepsilon$ on these distributions at the optimal sample size $n\asymp\varepsilon^{-2+2/r}$ has worst-case ld-CMI of order $n$ bits, and $\Theta(n/(1+(\tau/\varepsilon)^2))$ bits under Gaussian noise of standard deviation $\tau$ on the loss differences. The same holds without a regularizer, at $n\asymp\varepsilon^{-2}$. Consequently, range-scaled ld-CMI bounds cannot vanish on these distributions, although every proper learner's generalization gap is $O(n^{-1/2})$. We also show that model-level information does not determine noisy loss-difference information, and that the growth, slope and dimension conditions are needed, the last up to a logarithm.
Read the full abstract ▾
Information Theory
On Optimal Encodings and Systematicity in Function-Correcting Codes
Charul Rajput, Kanchana Lokshmii Jagatti and B. Sundar Rajan
Function-correcting codes (FCCs) protect the value of a function $f$ of the message against $t$ errors. In the original formulation of Lenz et al. (2023), the encoding is systematic, and the lower bound of $2t$ on the redundancy relies on this form. Recently, D. Ho (arXiv, 2026) showed, for linear functions and linear encodings, that systematicity can cost redundancy. We begin with the OR function to show that the cost is not confined to the linear setting: a non-systematic encoding attains redundancy $1$ while every systematic encoding needs $2t$. For a linear function $f$ and a fixed linear code $C$, let $d_f$ denote the minimum distance between codewords of messages with different function values. Different generator
matrices of $C$ assign different codewords to the messages and can give different values of $d_f$. We study which generator
matrix of $C$ gives the largest $d_f$. We give an
algorithm that constructs an optimal generator
matrix and determines the optimal value as the weight of a codeword in a greedy basis of $C$. We then characterize, in terms of information sets, when a generator
matrix in systematic form attains this optimum, and give a necessary condition that is checked on the low-weight codewords of $C$ alone. For two-valued functions, we drop both linearity and systematicity. Using the minimality of initial segments of the simplicial order with respect to Hamming neighbourhoods, we show that a non-systematic $(f,t)$-FCC of length $n$ exists if and only if a condition depending on $f$ only through the size of its smaller preimage holds. For $t=1$, redundancy $1$ is sufficient for every nonconstant two-valued function on $\mathbb{F}_2^k$ with $k\ge 10$. For general $t$, redundancy $1$ suffices for all two-valued functions once $k$ is large enough, and for each $s<2t$ we give an upper bound on the threshold in $k$ beyond which redundancy $s$ suffices.
Read the full abstract ▾
Information Theory · Cross-listed from quant-ph
Efficiently computable bounds on the energy-constrained quantum reading capacity
Vishal Singh, Zixin Huang and Mark M. Wilde
In
quantum reading, classical messages are encoded in sequences of
quantum channels and recovered by probing the channels and processing their outputs. We study the reading capacity of a finite family of finite-dimensional channels under an average constraint on the probing energy, allowing arbitrary adaptive operations between channel uses. Using an input-dependent chain rule for the Belavkin-Staszewski relative
entropy, we derive a converse bound expressed as an optimization involving channel Choi operators and the operator relative
entropy. This formulation admits semidefinite approximations and reduces, without an energy constraint, to a channel information radius. To obtain achievable rates, we construct bilinear semidefinite lower approximations to a standard non-adaptive reading bound. Alternating optimization produces feasible probe states and encoding distributions, whose Holevo information gives a directly evaluable achievable rate. For jointly classical-
quantum channels, we derive a separate converse based on the Umegaki relative
entropy. Without an energy constraint, this converse matches a non-adaptive achievable rate, recovering a recent result of Pascual Abraldes and Winter: non-adaptive protocols suffice to achieve the unconstrained reading capacity of jointly classical-
quantum channels.
Read the full abstract ▾
Information Theory
A Unified Framework for Characterizing General MIMO Channels
Zeyan Zhuang, Anzheng Tang, Xin Zhang et al. (+2 more)
Modern
MIMO systems are evolving toward higher-dimensional and more flexible architectures, such as distributed and holographic
MIMO, offering substantial benefits while giving rise to increasingly complex channel correlation structures. This growing architectural diversity makes it difficult to characterize the fundamental limits of different
MIMO systems on a case-by-case basis, motivating a unified analytical framework applicable across a broad range of architectures. This paper addresses this research gap by investigating a generally correlated Rayleigh channel whose vectorized form follows a general Gaussian distribution with an arbitrary covariance
matrix. For that purpose, we first prove that the spectral norm of the channel
matrix is bounded in the asymptotic regime where the numbers of transmit and receive
antennas grow proportionally. We then derive a deterministic approximation for the ergodic mutual information, with an explicit convergence rate governed by the structure of the channel correlation. The approximation is characterized by a pair of
matrix-valued self-consistent equations, for which we establish the existence and uniqueness of the solution and propose an iterative numerical
algorithm. Furthermore, we develop a framework based on positive linear maps to analyze the stability of these equations, which can facilitate the spectral analysis of broader classes of random
matrices. The proposed characterization unifies many existing results for structured
MIMO channels as special cases while remaining applicable to a broad class of general
MIMO architectures that are difficult to evaluate using existing analytical frameworks. To demonstrate its utility, we apply the developed theory to two representative systems, namely downlink distributed
MIMO and uplink heterogeneous
MIMO. Numerical results confirm the accuracy of the derived deterministic approximations.
Read the full abstract ▾
All authors Zeyan Zhuang, Anzheng Tang, Xin Zhang, Dongfang Xu, Shenghui Song
Information Theory
Task-Sufficient Contraction: Source Selection for Machine Information Interfaces
Joss Armstrong
A declared task can sometimes certify a reduced source before a downstream encoder, codebook, rate, distortion target, or optimizer is chosen. This paper studies when one such reduction preserves the complete downstream problem family, a property termed Task-Sufficient Contraction. The reduced source is fixed by the task before the later operating point is selected. An exact contraction allows the later problem to be solved on that source with the same result as if the full source had been retained. For a machine with a fixed set of possible actions and a fixed loss, the paper identifies a consumer-specific source by merging states only when every available action has the same regret in both. For finite action sets, replacing the richer source by this reduced source preserves the complete one-step rate-regret curve, even though the reduction is fixed before the distortion target is chosen. A second result gives an exact characterization for quadratic loss on affine feasible-action sets: the canonical reduced source is the projection onto the directions in which feasible actions can differ. Under a fixed energy budget, this becomes centered load, while retaining only the optimal water-filled action is too coarse. Earlier Information Bottleneck, semantic rate-distortion, and goal-oriented quantization results are then used to distinguish exact, architecture-conditioned, approximate, failed, and corrected contractions. The framework suggests a way for heterogeneous machines to exchange what a receiving task needs without first aligning their full internal representations.
Read the full abstract ▾
Information Theory
Slow Beats Fast at the Kesten-Stigum Threshold: Minimax, Fisher-Information and Belief-Propagation Characterizations of the Information-Computation Gap in Sparse Stochastic Block Models
Soroor Ghandali
We study community recovery in the sparse symmetric
stochastic block model with $q$ communities, average degree $d$ and signal strength $\lambda$ through statistical decision theory and Fisher information, and obtain three characterizations of the Kesten-Stigum threshold $d\lambda^2=1$ and of the information-computation gap below it. First, on each community-size profile the minimax risk of any class of rules closed under averaging and vertex relabeling equals its Bayes risk under the uniform prior; the posterior mean is the unique Bayes rule and is admissible, and the Bayes risk of degree-$D$
polynomial rules is the trivial risk times $1-\mathrm{Corr}_D^2$. Combined with known low-degree and information-theoretic results, this gives the gap as a worst-case statement: for $q\ge 5$ there is a window below the threshold in which no low-degree rule beats the trivial risk asymptotically, while an exponential-time rule does on a set of labelings of probability $1-o(1)$. Second, the Fisher information about $\lambda$ carried by cycle counts is a series with terms of order $k(d\lambda^2)^k$, convergent exactly when $d\lambda^2<1$; below the threshold the relative error of every unbiased cycle-based
estimator of $\lambda^k$ stays above an explicit constant, and every cycle-count test has success probability bounded below one. Third, the derivative of belief propagation at its uninformative fixed point multiplies a random perturbation by $|\lambda|\sqrt{d}$ per iteration, and one EM step taken there leaves $\lambda$ unchanged. A signal-to-noise computation recovers the condition $d\lambda^{1/\chi}>1$ of Chin et al. for $q=n^\chi$ communities and identifies personalized PageRank as a walk count with suboptimal weights. Experiments on networks with up to $3\times 10^5$ vertices confirm the threshold for $q=2$, the hard window for $q=5$, and the many-community scaling.
Read the full abstract ▾
Information Theory
Algorithms for Sampling Self-Orthogonal and Totally Self-Orthogonal Codes in Odd Characteristic
Martin R. Albrecht, Benjamin Benčina and Russell W. F. Lai
We give an
algorithm that samples uniformly random linear codes of any hull dimension and type over finite fields of odd characteristic, implying the first
algorithm for sampling uniformly random self-orthogonal codes of any rate, including self-dual codes. Our
algorithm is a re-visitation of the
algorithm given by Albrecht, Benčina and Lai (EC'25), using the mass formulae proven by Li, Shi and Ling (IEEE Trans. Inf. Theory 71(1)). This allows us to instantiate code-based cryptographic schemes that rely on the hardness of Permutation Code Equivalence (PCE) on `random' self-dual codes for security and that were previously unable to sample them. Building on the observation by Bardet, Otmani and Saeed-Taha (ISIT'19) that Euclidean orthogonality is insufficient when considering PCE over finite extension fields due to non-trivial Galois automorphisms, we study the behaviour of what we call total orthogonality, that is orthogonality with respect to all induced Galois geometries simultaneously. We characterise total orthogonality of vectors and codes, and give an
algorithm that samples linear codes with a total hull of a prescribed dimension; a subcode that acts as the hull in all Galois geometries of the ambient space. The
algorithm incurs a rate decrease by a factor equal to the extension degree, however, we argue why this may be necessary in the context of PCE and explore how it limits the practicality of our
algorithm. We consider the notion of Galois type of a linear code when Galois hulls are symmetric and show that all linear codes have constant Hermitian type.
Read the full abstract ▾
Information Theory · Cross-listed from cs.CC
Reed-Solomon Codes at Capacity: Algorithmic List-Decoding and Proximity Gaps
Prahladh Harsha, Mrinal Kumar and Ramprasad Saptharishi
Understanding the limits of list-decodability of Reed-Solomon codes has been one of the most important open problems in algebraic coding theory. Recently, Brakensiek, Chen, Putterman, Zhang, and Zheng, in a remarkable breakthrough, showed that Reed-Solomon (RS) codes over fields of large characteristic are algorithmically list-decodable all the way up to capacity. Building on this result, Jeronimo subsequently extended these techniques to solve the proximity-gaps question for RS codes. In this article, we give a unified and transparent exposition of these results.
Read the full abstract ▾
Information Theory
Graph-Based Linear Codes Associated with the Finite Ring $\mathbb{F}_p[x]/\langle x^4 \rangle$
Apurba Sarkar, Kalyan Hansda and Makhan Maji
We investigate the linear codes derived from the zero-divisor graph $\Gamma(R)$ of the finite local ring $R = \mathbb{F}_p[x]/\langle x^4 \rangle$ for an odd
prime $p$. Using an equitable partition of $Z^*(R)$, we determine the parameters of the binary incidence code of $\Gamma(R)$ and construct induced bipartite subgraph codes achieving dual minimum distance $4$. Over $\mathbb{F}_p$, we parameterize the adjacency and
Laplacian codes, derive the closed-form weight distribution of the
Laplacian code via the MacWilliams identity, and establish its LCD property. Additionally, we determine the automorphism groups of the graph and its
matrix codes. Finally, from extremal subgraphs of $\Gamma(R)$, we construct two families of Griesmer-optimal linear codes over $\mathbb{F}_p$, comprising a constant-weight minimal code and an optimal two-weight code.
Read the full abstract ▾
Information Theory
HintKD: Hint Knowledge Distillation for Bandwidth-Constrained Cloud-Edge Inference
Wanling Luo, Jiayi Zhang, Bokai Xu et al. (+1 more)
Collaborative inference between the edge cloud and user equipment (UE) is a promising paradigm for deploying large deep neural networks (DNNs) in 6G networks. However, existing cloud-edge inference and
distillation schemes often require the real-time transmission of high-dimensional intermediate features or soft probability vectors, which imposes a substantial communication burden on bandwidth-limited wireless links. To address this challenge, we propose HintKD, a bandwidth-constrained
distillation framework for cloud-edge inference. The core idea is to compress the teacher's guidance into compact discrete hints rather than transmit raw features directly. Specifically, a cloud-side teacher is first trained with a maximum conditional mutual information (MCMI) objective to preserve informative intra-class variations. Its latent representation is then mapped to a compact codebook through differentiable vector quantization. During deployment, the cloud transmits only a hint index, and the UE uses a lightweight FiLM-based adapter to modulate student features according to the received codeword. Experiments on the DeepSense 6G beam prediction task show that HintKD preserves competitive accuracy while reducing the inference-side communication payload from O(D_t) for feature transmission or O(C) for logit transmission to O(1) for discrete hint transmission. In addition, experiments on the MNIST
dataset further verify the generality of the proposed
distillation method across different
datasets and tasks. These results show that HintKD achieves a favorable accuracy-bandwidth trade-off by replacing high-dimensional feature or logit transmission with a compact discrete hint index.
Read the full abstract ▾
All authors Wanling Luo, Jiayi Zhang, Bokai Xu, Ziheng Liu
Information Theory
Multi-Agent Reinforcement Learning for Movable Antenna-aided Cell-Free Massive MIMO Systems
Bokai Xu, Jiayi Zhang, Shuaifei Chen et al. (+4 more)
The inherent non-convex minimum-separation constraints introduced by movable
antennas present a formidable challenge to the joint optimization of
antenna positions and transmission strategies, rendering conventional methods computationally infeasible, particularly in large-scale
cell-free massive multiple-input multiple-output (
MIMO). In this work, we propose the graph-based learning individual intrinsic reward heterogeneous-
agent proximal policy optimization (GLIIR-HAPPO)
algorithm, a novel heterogeneous multi-
agent reinforcement learning (MARL) framework that fundamentally overcomes this impasse by systematically decomposing the original coupled optimization into coordinated subproblems. To ensure tractability, we embed the non-convex
geometric constraints into a penalty-augmented reward structure and develop a specialized
geometric solver that enables the positioning
agents to efficiently navigate the high-dimensional action space. Specifically, we propose an architecture featuring a dynamic-interaction graph critic for adaptive cross-role coordination, together with role-conditioned
federated distillation that synchronizes same-role policies through compact actor-output statistics. Beyond architectural design, we establish a rigorous theoretical analysis that derives monotonic performance improvement bounds and establishes convergence guarantees for the proposed bi-level optimization. Numerical simulations demonstrate that our framework yields significant sum-rate improvements over state-of-the-art MARL schemes. Moreover, the performance of our advanced architecture closely approaches its fully centralized counterpart, while drastically reducing communication overhead.
Read the full abstract ▾
All authors Bokai Xu, Jiayi Zhang, Shuaifei Chen, Ziheng Liu, Huahua Xiao, Derrick Wing Kwan Ng, Bo Ai
Information Theory
Optimal Conversion Bandwidth for MDS Convertible Codes in the Split Regime with $r^F<k^F<r^I$
Lewen Wang and Sihuang Hu
Erasure codes are widely used in distributed storage systems to provide fault tolerance. An $[n,k]$ erasure code encodes $k$ data symbols into $n$ coded symbols and distributes them across $n$ storage nodes. Once the code parameters are fixed, the achievable fault tolerance is also fixed. However, the failure rates of storage nodes may vary over time, and dynamically adapting the code parameters to these variations can substantially reduce storage overhead. Motivated by this observation, Maturana and Rashmi introduced convertible codes, which allow an $[n^I,k^I]$ initial code with redundancy $r^I=n^I-k^I$ to be transformed into an $[n^F,k^F]$ final code with redundancy $r^F=n^F-k^F$, while preserving the required code properties. Convertible codes whose initial and final codes are both MDS codes are called MDS convertible codes. They are of particular interest because MDS codes provide the maximum erasure tolerance for a given amount of storage overhead. Several works have established lower bounds and constructions for the bandwidth cost of MDS convertible codes in the split regime. However, the tight bound in the parameter range $r^F<k^F<r^I$ remains unknown. In this work, we establish a family of new rank inequalities for stable MDS convertible codes with linear conversion procedures and derive an improved lower bound on the bandwidth cost consisting of three cases for this remaining range. We prove that the bound is tight by presenting three explicit constructions, one for each case.
Read the full abstract ▾
Information Theory
On the State Evolution of Approximate Message Passing with Discontinuous Denoisers
Kuranage Roche Rayan Ranasinghe, Takumi Takahashi and Giuseppe Thadeu Freitas de Abreu
We consider approximate message passing (AMP) with discontinuous denoisers, such as hard slicers, quantizers and hard thresholding, for which the classical state
evolution (SE) analysis does not apply. It is first shown that the standard Onsager coefficient; i.e., the average pointwise derivative of the denoiser, misses a boundary term, namely, the sum over the jumps of the jump size times the density of the denoiser input at the jump. Then, for real independent and identically distributed (i.i.d.) Gaussian
matrices, a positive noise variance, a fixed number of iterations and a prior with finite second moment, we transfer the Lipschitz SE
theorem directly to denoisers, fixed in advance, that are continuously differentiable with bounded derivative except at finitely many jumps, by bounding the distance between the iterates of AMP and of AMP on Lipschitz ramps of the denoisers through the fraction of inputs that cross a jump. It is thereby proven that AMP follows SE, also for test functions such as symbol errors, if its Onsager coefficient contains the boundary term, evaluated either along SE or, as we propose, at the noise level estimated from the residual, with the density computed from the known prior. Simulation results with 4- and 8-level pulse amplitude modulation (PAM) demonstrate that AMP with a hard slicer, whose standard coefficient is identically zero, fails with this coefficient, while with the proposed one, it follows SE and comes close to Bayes-optimal AMP.
Read the full abstract ▾