Information Theory · Cross-listed from eess.AS
Co-Adaptive Entropy Modeling and Rate-Distortion-Optimal Bit Allocation for Perceptual Audio Coding
Soohyun Kim, Jiwon Jeong and Marina Bosi
We present a perceptual audio coder in which the
entropy-coding model and the bit allocation co-adapt across blocks without model side information. Mantissas are coded by a hybrid of bit packing and arithmetic coding with per-band probability models learned online. Encoder and decoder accumulate identical histograms from decoded blocks and advance the same model state. The model enters bit allocation directly: per-band rate–distortion frontiers combine measured perceptual distortion with
entropy-coded rate estimates under the current model, and a dynamic program attains the exact minimum of total objective perceptual distortion for each block's budget under the model's rate estimates, with reservoir carry-over. Under a strict per-block rate,
entropy-coding-aware bit allocation lowers this distortion (band noise-to-mask ratio in linear scale) by 15-24% relative to the same solver without it, and co-adaptation removes a further 3-4% (up to 15% on oboe) that a backward-adaptive coder alone cannot realize in one pass; a MUSHRA test at 80 kbps per channel confirms the ranking.
Read the full abstract ▾
Information Theory · Cross-listed from cs.LG
Effect of Spherical Parameter Space on Number of Non-Forgetting Tasks and Capacity of Continual Learning
Nazanin Mirhosseini
We propose a continuous learning formulation for $T$ non-forgetting tasks with the spherical parameter space $\mathbb{S}^{d-1}(\sqrt{d})$ in asymptotic regime $d\rightarrow \infty$. Each task $i$ has an optimum $\bm{\Theta}_i$, minimizing its corresponding loss function $R_i(\bm{w})$ on sphere. We reduce each task $i$ to an acceptable loss region modeled by a geodesic ball $B_{\bm{\Theta}_i}(r_i(d))$ on $\mathbb{S}^{d-1}(\sqrt{d})$ centered at $\bm{\Theta}_i$ with radius $r_i(d)$. To model task dependencies (similarities), task optima $\bm{\Theta}_i$ are distributed by a Neyman-Scott clustered point process $\Phi$. Cluster centers $\{\bm{Y}_j\}_{j=1}^{N_p}$ form a homogeneous Poisson point process on sphere, while intra-cluster task optima follow a von Mises-Fisher distribution with mean $\bm{Y}_j$ and concentration $\kappa$. Each task $t\in \{1, ..., T\}$ learning
algorithm is considered to be a channel $Q_{t}(\mathrm{d}\bm{w}_t | \bm{w}_{t-1}, \mathcal{K}_{t-1}(\bm{\theta}_{1:t-1}), \bm{\theta}_t)$, where $\bm{W}_t$ is the learned parameter of task $t$ and $\mathcal{K}_{t-1}(\bm{\Theta}_{1:t-1})=\bigcap_{i=1}^{t-1}B_{\bm{\Theta}_i}(r_i(d))$. Since in continual learning, the learner does not need to reach $\bm{\Theta}_t$ exactly to successfully learn task $t$, to avoid forgetting as well as reliable learning, it only needs for the learned parameter vector $\bm{W}_{t}$ to enter $B_{\bm{\Theta}_t}(r_t(d)) \cap \mathcal{K}_{t-1}(\bm{\Theta}_{1:t-1})$. Using this model, we derive an upper bound on $T$ ensuring $\mathbb{P}[\mathcal{K}_{T}(\bm{\Theta}_{1:T}) \neq \emptyset] \rightarrow 1$ as $d\rightarrow \infty$ and $\kappa/d \rightarrow \alpha$, where $\alpha>0$ is a constant. Finally, for $T$ tasks sampled uniformly from $\Phi$, we derive an upper bound for total information theoretic capacity under the non-forgetting constraint for each task.
Read the full abstract ▾
Information Theory · Cross-listed from cs.LG
Measuring Learned Equivariance: An Information Theoretic Approach
Claire Schlesinger, Wolfgang Gatterbauer and Robin Walters
Learning
symmetries through data augmentation offers an alternative to architectural equivariance, but does not guarantee that a network respects those
symmetries. Measuring learned equivariance is especially difficult for intermediate representations, whose group actions are not specified. In a finite discrete setting, we show that equivariance requires representations of original and transformed inputs to determine one another exactly, yielding a test based on normalized mutual information (NMI). For high-dimensional activations, we use centered kernel
alignment (CKA) as a scalable test, with an exact guarantee for orthogonal feature transformations. Across architectures and tasks, augmentation usually increases transformation consistency, showing that learned equivariance can be probed directly from representations without recovering the hidden transformation rules that relate them.
Read the full abstract ▾
Information Theory · Cross-listed from cs.LG
Can Rényi Accountants Beat Lossless Accountants for DP-SGD?
Jinho Bok
Recent frameworks in differential
privacy (DP), such as
privacy loss distributions and $f$-DP, provide lossless representations of the
privacy leakage of mechanisms. These approaches are often considered to yield tighter $(\varepsilon, \delta)$-DP guarantees than those obtained from relaxed notions such as Rényi DP. But is that always the case? We study this question for differentially private
stochastic gradient descent (DP-SGD) with fixed-size batches under replace adjacency. Our key observation is that in this setting,
privacy accounting based on Rényi DP can use a pair of worst-case distributions (i.e., a dominating pair) that differs from the one used in the lossless frameworks. In particular, we characterize the worst-case distributions as two-dimensional Gaussian mixtures and derive an improved Rényi DP bound for DP-SGD. Numerical and asymptotic comparisons show that across various hyperparameter settings, a Rényi DP accountant based on the new bound can yield better
privacy guarantees than those obtained from the lossless frameworks.
Read the full abstract ▾
Information Theory
The Right to Be Forgotten under Correlation: Privacy-Utility Tradeoffs
Luis Massny, Rawad Bitar, Fangwei Ye et al. (+1 more)
We study
privacy-preserving
algorithms that redact information from a correlated
dataset. The goal is to reduce the information leakage about sensitive parts. We analytically characterize the optimal tradeoff between
privacy (leakage through correlation) and utility (number of redactions) for local mechanisms and binary Markovian correlations. Our result shows how to decompose the mechanism design problem into a sequence of local mechanism design problems and reveals their sequential relation. The optimal mechanism is the solution to a Finite-Horizon Deterministic Markov Decision Process.
Read the full abstract ▾
All authors Luis Massny, Rawad Bitar, Fangwei Ye, Salim El Rouayheb
Information Theory · Cross-listed from math.CO
Construction of Regular Fat Linear Sets and Non-Representable q-Matroids
Yufei Qiu, Valentino Smaldore, Zhenpeng Wang et al. (+2 more)
Let $q$ be a
prime power. An $(r, i)$-regular fat $\mathbb F_q$-linear set has exactly $r$ points of weight $i\ge2$, called fat points. All of its remaining points have weight one, and at least one such point occurs. We solve two open problems posed by three of the authors of this paper. First, we derive divisibility constraints on the point weights using the scalar stabilizer of the defining subspace. Consequently, a regular fat linear set of rank $2i$ in $\operatorname {PG}(1,q^n)$ has at most two or exactly $q + 1$ fat points. In the latter case, the fat points form an $\mathbb F_q$-subline. Second, we construct higher-dimensional regular fat linear sets whose fat points do not form an $\mathbb F_q$-subgeometry. In particular, for every $k\geq 3$ we obtain examples of rank $2k$ whose fat locus consists of an $\mathbb F_q$-subgeometry $\operatorname {PG}(k-2,q)$ in $\operatorname {PG}(k-1,q^4)$ together with one external point. This construction is then generalized further. Finally, we apply our projective-line results to $q$-matroid representability and present an infinite family of non-representable rank-two $q$-matroids arising from partial spreads.
Read the full abstract ▾
All authors Yufei Qiu, Valentino Smaldore, Zhenpeng Wang, Corrado Zanella, Ferdinando Zullo
Information Theory
On the Multi-Weight Shift-Sum Iterative Decoding of Short BCH Codes
Yinglei Yang, Di Zhang, Aimin Li et al. (+3 more)
This paper investigates a shift-sum decoding
algorithm that employs dual codewords. Existing dual codeword shift-sum decoding schemes mainly rely on minimum-weight dual codewords. However, the number of such codewords is often limited, which restricts the amount of reliability information available to the decoder. To address this issue, we extend the component counting mechanism from minimum-weight dual codewords to dual codewords whose Hamming weights lie within a prescribed upper bound. We further introduce a normalized component counting rule to suppress the overcounting bias caused by the much larger populations of higher-weight classes. Based on this framework, we develop a hard-decision iterative error reduction decoder and a soft information-aided iterative error reduction decoder. Numerical results show that the proposed method consistently outperforms existing minimum-weight shift-sum baselines especially for short codes where additional dual codeword information is particularly valuable.
Read the full abstract ▾
All authors Yinglei Yang, Di Zhang, Aimin Li, Yifei Shen, Alexey Frolov, Bo Gu
Information Theory · Cross-listed from math.OC
Calibrated $\alpha$-Barycenters for Multiobjective $f$-Divergence Optimization
Dier Tang, Peijie Li and Xiaoyuan Zhang
Matching one distribution to many targets can require repeated evaluations of every target. CAL-$\alpha$-MODM aggregates a nonlinear predictor into one barycenter, calibrates its residual derivatives, and checks steps against the original $f$-divergence objective. Under smoothness assumptions, calibration controls parameter-step model error without requiring targets to be close to the model. Exact-oracle stationarity and local convergence guarantees are established. At common matching, canonical geometry cancels the leading quadratic iteration error, giving cubic convergence with second-order calibration and sufficiently accurate inner solves. High-precision experiments support this prediction. Sensitivity-based initialization also accelerates preference continuation along regular stationary branches. On 300-target Jensen–Shannon problems, CAL-p2 achieves median paired speedups of $3.15\times$ over L-BFGS and $1.10\times$ over second-order Direct-TR, across 23/25 and 24/25 objective-comparable stationary pairs. The savings arise when cheaper predictor queries offset calibration overhead.
Read the full abstract ▾
Information Theory
Information-Theoretic Limits for Rectangular Spiked Matrix Models with Rotationally Invariant Noise
Qun Chen, Rudy Skerk, Jean Barbier et al. (+3 more)
We study the information-theoretic limits of inference in rectangular rank-one spiked
matrix models with additive rotationally invariant noise, thereby going beyond the standard assumption of independent noise entries. Using the replica method from statistical physics, we derive a variational formula for the mutual information between the signal and the data, together with a system of saddle point equations predicting the minimum mean squared error in estimating the rank-one spike. We further show that, in the Bayes-optimal setting, these saddle point equations coincide with the fixed point equations of the state
evolution of a recently proposed orthogonal approximate message passing
algorithm with Bayes-optimal denoisers. This equivalence provides an algorithmic interpretation of the replica prediction and supports the Bayes-optimality of orthogonal approximate message passing whenever the relevant fixed point is unique. Numerical evaluations of the saddle point equations for several rotationally invariant noise ensembles and signal priors, together with finite-size simulations of the
algorithm, confirm the asymptotic predictions. They illustrate both regimes with a unique informative fixed point, where orthogonal approximate message passing is predicted to attain the minimum mean squared error, and regimes with multiple fixed points, where a statistical-to-computational gap exists.
Read the full abstract ▾
All authors Qun Chen, Rudy Skerk, Jean Barbier, Francesco Camilli, Junjie Ma, Haochuan Zhang
Information Theory · Cross-listed from eess.SP
Generative Deep Learning for CSI Feedback in Massive MIMO via Channel Distribution Prediction
Liujia Yao, Changsheng You, Chao Zhou et al. (+2 more)
Existing deep-learning-based channel state information (CSI) feedback methods reduce feedback overhead by compressing high-dimensional CSI into low-dimensional representations and/or exploiting historical CSI to predict future CSI. However, these methods may become unreliable when user equipment (UE) suddenly changes its motion. To address this issue, we propose a prediction-aided coding (PAC) framework for generative CSI feedback, which adaptively encodes CSI based on predicted channel/CSI distributions and the observed CSI in real time. Each CSI frame is represented by multiple CSI
tokens obtained without using historical CSI, making
token generation robust to abrupt UE motion. In the first stage, the conditional distribution, instead of a deterministic value, of the next-frame CSI
tokens is predicted based on historical
tokens. In the second stage, once the next frame arrives, the predicted distribution is used for variable-length
entropy coding of its CSI
tokens. To further reduce feedback overhead and inference latency, we propose efficient designs for both stages. For the first stage, a customized causal
Transformer with a new positional
embedding strategy accurately predicts next-frame
token probabilities. For the second stage, a parallel causal decoding strategy reduces the inference latency caused by sequential
token decoding. Extensive experiments show that, compared with existing history-aware baselines, PAC reduces feedback overhead by approximately 72% to 84% while achieving higher reconstruction accuracy. Additionally, PAC maintains CSI reconstruction accuracy under random-walk trajectories, whereas existing history-aware baselines suffer sharp degradation in reconstruction accuracy.
Read the full abstract ▾
All authors Liujia Yao, Changsheng You, Chao Zhou, Yi Gong, Chengwen Xing
Information Theory
Digital-Twin-Referenced Belief-Guided Q-Learning for Jammer Identification and Beam-Power Control in a mmWave Model
Parmida Geranmayeh and Onur Günlü
Digital Twin (DT) combined with reinforcement learning (RL) can support adaptive resource allocation and security-aware decision-making by comparing expected and observed network behavior. This paper presents a simulation-based DT reference and a belief-guided Q-learning (BGQL) for joint jammer identification and beam-power control in a site-specific 28-GHz network. For each action, the same channel simulator is evaluated under an attack-free and an attacked configuration. Receiver-level residuals are combined with spatial jamming footprints and accumulated to form a normalized suspicion vector. BGQL incorporates this vector into an aggregated state, a belief-concentration reward, and exploratory beam-selection rule. The BGQL scheme is compared with SARSA, Greedy, and Random under matched attacker sets. Across 30 paired Monte Carlo runs in four configurations, with the number of active jammers supplied to every method, BGQL achieves mean aggregate full-band link-rate scores of approximately 437.4–564.2~Mbit/s and rate-to-device-power scores of 60.37–76.79~Mbit/J. Its detection accuracy is 100% in one-jammer configurations and approximately 78%–81% in two-jammer configurations. BGQL requires approximately 80–130~s per run, compared with 5–14~s for SARSA, reflecting evaluation of 625 beam pairs during exploratory steps. These findings provide a matched-model simulation proof of concept for DT-referenced belief-guided learning for security-aware beam-power control in 6G networks.
Read the full abstract ▾
Information Theory
Universal Distributed Feature Extraction Complementary to Side Information via H-Score
Yuan Xinjie, Dier Tang and Shaolun-Huang
This paper considers a distributed estimation system where edge nodes transmit averaged features to a center node for parameter estimation. A key challenge arises when the center node possesses auxiliary side information: how to design local extractors that are universally optimal for any downstream task, while avoiding redundancy with the side information. To tackle this problem, we orthogonally decompose the joint function space and demonstrate in the weak-dependence regime that the modal decomposition of the canonical dependence kernel's conditional-additive component yields optimal, task-independent features. We construct a distributed nested H-score to learn this decomposition and show that, under a limited feature budget, the mean-squared error (MSE) gap to the best task-specific features is determined solely by the task's
alignment with the discarded, least informative modes, and vanishes to leading order for every square-integrable task once the budget is sufficient.
Read the full abstract ▾
Information Theory · Cross-listed from eess.SP
Investigation of RIS Integration in Satellite Systems Guided by ITU-R VSAT Radiation Diagram
Symeon Vaidanis and Athanasios D. Panagopoulos
The introduction of Reconfigurable Intelligent Surfaces (RIS) technology marks a transformative era in modern communication systems, enhancing coverage and facilitating higher data transmission rates. While recent research has delved into the integration of RIS into satellite networks, a critical and practical gap persists concerning its implementation in the satellite-Very Small Aperture Terminal (VSAT) downlink, specifically amplifying contributions from the VSAT side. This paper addresses this gap by proposing a simple and engineering methodology to optimize RIS position and orientation, incorporating considerations for the
antennas' radiation patterns and the system's
topology to streamline optimization variables. The primary objective of this paper is to alleviate the double fading constraint in the reflect link without the need for active elements, achieved by leveraging the Fresnel zone of the RIS. The improvement power ratio depends on the frequency and the VSAT 's diameter. The results show that for a VSAT with diameter 1m the gain is up to 9dB at 2 GHz (S band) and up to 14dB at 4 GHz (C band). Additionally, the RIS channel equation is refined, encompassing the tropospheric attenuation and accounting for electromagnetic wave polarization.
Read the full abstract ▾
Information Theory
On Measures of Dissimilarity on the Majorization Simplex
Tarun Hariharan, Prasad Krishnan and Abhishek Deshpande
The majorization simplex $\mathcal{P}_n$ is the set of probability vectors of length $n$ whose entries are arranged in non-increasing order. Under the majorization order, $\mathcal{P}_n$ is a
lattice, and it is the natural space for comparing distributions when the labels of the outcomes are not relevant. In this work, we study measures of dissimilarity (and, similarity) on $\mathcal{P}_n$. Our first set of results concerns the Rényi-
entropy based distance $d_\alpha(p,q)=H_\alpha(p)+H_\alpha(q)-2H_\alpha(p\vee q)$ proposed by Yadav and Shkel (2026), which generalizes a metric of Cicalese, Gargano and Vaccaro (2013). We show that $d_\alpha$ is a metric on $\mathcal{P}_n$ for $\alpha\in[1,\infty)$, while for $\alpha\in[0,1)\cup\{\infty\}$, $d_\alpha$ is not a metric in general. Our main result in this part identifies the necessary and sufficient conditions for equality in the triangle inequality of $d_\alpha$ when $\alpha>1$, for which we identify the equality conditions for the supermodularity of the Rényi
entropy $H_\alpha$. Our second set of results concerns the total variation distance and the Hellinger affinity (also called the Bhattacharyya coefficient) on $\mathcal{P}_n$. We obtain a number of bounds on these measures, along with some of their equality conditions. The resulting Hellinger affinity based upper bound on the total variation distance can be smaller than the bounds obtained from Pinsker's and the Bretagnolle–Huber inequalities.
Read the full abstract ▾
Information Theory · Cross-listed from eess.SP
Properties of Array Factor of Reconfigurable Intelligent Surfaces for Radio Communications
Symeon Vaidanis and Athanasios D. Panagopoulos
This paper delves into the intricate aspects of Reconfigurable Intelligent Surfaces (RIS) in the
antenna design. It investigates the properties of the array factor of RIS, highlighting the significance of phase shifts in establishing of the reflect path. Additionally, the paper investigates spatial positions requiring identical phase shifts, nullification of the array factor, the count of total maxima, second maximum determination (amplitude of the second lobe), and the calculation of the 3dB bandwidth. They are very useful for RIS-assisted
radio communication systems.
Read the full abstract ▾
Information Theory
Covering Radius and Deep Holes of Hyperderivative Reed-Solomon Codes under the NRT Metric
Zhihao Zhu, Haojie Gu, Jun Zhang et al. (+1 more)
The deep hole problem is a fundamental problem in coding theory, and it has many important applications in code constructions and cryptography. The deep hole problem of Reed-Solomon codes has gained a lot of
attention. As a generalization of Reed-Solomon codes under Hamming metric, we study the covering radius and deep holes of Hyperderivative Reed-Solomon (HRS) codes under the Niederreiter-Rosenbloom-Tsfasman (NRT) metric. We determine the NRT covering radius of HRS code and characterize its NRT deep holes via Maximum Distance Separable (MDS) extensions and
polynomial remainder conditions. We further relate the enumeration of NRT deep holes to the characteristic
polynomial of an associated hyperplane arrangement. We also provide three classes of explicit NRT deep-hole constructions, propose a Cheng-Murray type
conjecture for HRS codes under the NRT metric and obtain a conditional low-layer description. Using these characterizations, we derive exact enumeration formulas in several special cases.
Read the full abstract ▾
All authors Zhihao Zhu, Haojie Gu, Jun Zhang, Yu Li
Information Theory · Cross-listed from cs.SD
PRB-WM: Physics-Regularized Beta World Models for Predictive Link Adaptation on Real Time-Varying Underwater Acoustic Channels
Shengjie Zhong, Zhongliang Zhao, Jingxuan Chen et al. (+5 more)
Adaptive modulation and coding (AMC) improves the spectral efficiency of underwater acoustic (UWA) links, but its feedback is inevitably stale: sound propagates slowly, and by the time channel state information (CSI) arrives, the fast time-varying multipath channel has moved on. Predictive link adaptation therefore needs more than a point forecast: it needs a distribution over each candidate action's consequences. We propose PRB-WM, a physics-regularized Beta world model for predictive per-subcarrier link adaptation on real time-varying UWA channels. A persistence-anchored
stochastic latent-space dynamics model learns how the channel evolves; a counterfactual outcome head evaluates, for every candidate modulation and coding scheme (MCS), a Beta-binomial distribution of bit errors on shared imagined futures; and a chance-constrained receding-horizon model predictive controller maximizes expected spectral efficiency subject to a bounded probability of violating the bit error rate (BER) target. To the best of our knowledge, PRB-WM is the first world model for UWA communications. In closed-loop replay of six measured sea-trial links from the public UWA Channel Library against a 24-method pool spanning the full methodological spectrum, PRB-WM attains a mean goodput 87% above the strongest baseline, ranks first on all six time-varying links at the longest lookahead, and is the only method whose realized frame risk decreases systematically with lookahead, meeting the BER-violation contract where no baseline does. Ablations confirm that rollout sampling dispersion is essential to the chance constraint and that the physics anchor is necessary for trainability.
Read the full abstract ▾
All authors Shengjie Zhong, Zhongliang Zhao, Jingxuan Chen, Dianrun Huang, Zichen Zhao, Yang Liu, Dapeng O. Wu, Tony Q. S. Quek
Information Theory · Cross-listed from eess.SY
Design of Privacy-Preserving Estimators Under Prior Information Exposure
Le Liu, Yu Kawano and Ming Cao
State estimation is widely used in networked dynamical systems, but released estimates may inadvertently reveal sensitive information about private state variables. Existing
privacy-preserving estimation frameworks often do not explicitly account for
adversarial prior knowledge. Motivated by this gap, we study
privacy-preserving state estimation for discrete-time linear
stochastic systems under prior information exposure using pointwise maximal leakage (PML). We first reveal a
privacy anti-optimality property of the Kalman filter: in the absence of additive release noise, the steady-state Kalman filter maximizes PML
privacy leakage among the considered class of stable linear
estimators. We further establish a nontrivial lower bound on the steady-state estimation error enforced by $(\varepsilon,\delta)$-PML
privacy. We then derive sufficient release-noise conditions that guarantee $(\varepsilon,\delta)$-PML
privacy, first for the Kalman filter and then for stable linear
estimators. Finally, a convex semidefinite program is formulated for a joint design of the
estimator gain and release noise covariance, providing a tractable
privacy-preserving
estimator with an explicit upper bound on the steady-state estimation error. Simulations on a source–load power system demonstrate the proposed
privacy–utility tradeoff.
Read the full abstract ▾
Information Theory
Multiple-Function Approximated Coded Computing: Constructions and Error Bounds
Rimpi Borah, Wilton Kim, J. Harshan et al. (+2 more)
Coded computing mitigates stragglers in distributed computation. While Lagrange coded computing (LCC) enables exact
polynomial computation over finite fields, Berrut approximated coded computing (BACC) enables approximate computation of arbitrary, including non-
polynomial, functions in the analog domain. In applications such as multi-task learning and inference, multiple functions must be evaluated over different subsets of a common
dataset, making single-round computation desirable to reduce communication overhead. An existing LCC approach addresses this requirement but is restricted to
polynomial functions over finite fields, leaving single-round multi-function computation in the analog domain an open problem. In this work, we develop a single-round construction for computing multiple arbitrary functions using BACC. We first extend the existing multi-function computation approach in LCC to the BACC setting and derive an approximation-error bound. Building on this, we propose a construction based on Berrut basis functions for computing multiple arbitrary functions and derive its approximation-error bound. We analyze the scaling of both bounds with the number of functions and the number of data points per function, and numerically compare the two constructions over a wide range of arbitrary functions. The results demonstrate that the proposed construction achieves lower relative errors than the existing approach in the tested settings.
Read the full abstract ▾
All authors Rimpi Borah, Wilton Kim, J. Harshan, Stanislav Kruglik, Han Mao Kiah