Information Theory · Cross-listed from cs.LG
Perfect Reconstruction Cannot Show Whether Two Codebooks on a VAE's Latent Space Agree
Yusuke Hayashi
Many systems read the latent space of a variational autoencoder (VAE) as discrete codes, and rely on each latent being read back as the code it was written as. Shannon (1948) took such a shared codebook as given; inside a VAE nothing enforces it. We prove that no item of a standard training report can reveal whether the written and read codes agree on encoded data. We construct two VAEs that reconstruct every input perfectly, infer exactly, and match on every per-example ELBO and every code histogram, yet agreement is one in the first and zero in the second. We then show that the ELBO's own terms bound how far a measured agreement can move: the variational gap bounds the shift to exact inference, and the rate bounds the shift to generation from the prior. For Gaussian posteriors, a mean-field encoder pays an exact price in the gap that depends on its choice of axes. On probabilistic PCA the gap certifies a trained encoder in all 75 settings we test, at agreement between 0.76 and 0.97. On 240 trained VAEs, a floor set by the rate and the
dataset size rules out rate-side certification in advance for 66 models, and none of them certifies. The results yield a simple protocol, Certify-or-Measure, for any pipeline that reads a VAE as codes.
Read the full abstract ▾
Information Theory · Cross-listed from cs.LG
Bayes-Sufficient Compression Is Not Enough: How Does Communication Help Multi-Agent Systems?
Yi Xie, Zhanke Zhou, Yi Fan et al. (+3 more)
Multi-
agent LLM systems pair a sender with broad context and an executor with a limited local view. We study when a short message improves the executor's next decision, when raw context is preferable, and when a stronger sender helps. Our framework, receiver-relative bounded coordination, expresses message utility as receiver gain minus protocol tax. Compression beats raw context when tax savings exceed losses from omitted information and decoder mismatch. Even Bayes-sufficient compression can fail when a bounded executor cannot use its surface form. A three-stage decomposition separates externalization, absorption, and action closure, explaining how errors remain after the correct content reaches the receiver. Under a single-crossing condition, sender upgrades help above a receiver-burden threshold. Across six
benchmarks, the same Qwen protocol raises ContextBench joint accuracy from $0.633$ to $0.775$ but lowers ToolSandbox from $0.889$ to $0.653$. Fixed-message replay reveals closure failures despite correct artifact recovery. These results guide an inference-time selector that improves the accuracy-cost frontier on the evaluated communication regimes.
Read the full abstract ▾
All authors Yi Xie, Zhanke Zhou, Yi Fan, Yong Ge, Bo Han, Bo Liu
Information Theory · Cross-listed from cs.CR
Z-Sigil: A Public-Key Cryptosystem with Chained Selection over a Fiber Bundle of Module-Lattice Keys
Andrea Rondelli
Z-Sigil is a public-key cryptosystem in which the plaintext selects successive keys from a fixed module-
lattice family. Messages are length-prefixed, zero-padded and divided into 32-byte blocks. Each public vector is a shared public
matrix applied to a small secret vector, plus a small error. Key indices label torsion points of a flat Kähler
torus; the secret family forms a section of a key bundle over them. A public nonce initializes a hash state that selects each block's key and bit mask. The sender updates the state with the plaintext block; the receiver does so after recovering it. The stream and nonce determine a discrete walk along which decryption reads the secret section. We specify the
algorithms, prove correctness under an explicit noise condition and bound decoding failure for messages chosen after the public key. Under stated decisional Module-LWE assumptions, we establish IND-CPA confidentiality for the chain without modelling the state hash as a random oracle. The reduction covers
quantum adversaries under
quantum hardness assumptions, with classical keys, messages and ciphertexts; no concrete security level is established. With independent uniform selectors, a restricted direct-decryption model quantifies reduced fragment recovery under partial key exposure, without improving full-message recovery probability over an independent-block baseline. Known-plaintext and candidate-message attacks, and parallel candidate-table decryption, delimit this result. Neither a universal sequential lower bound nor authentication or chosen-ciphertext security is established. Replacing an earlier scalar-exposing proposal, we give an augmented-
lattice interpretation, integral-transport obstructions and a noise budget for research on curved, nontrivial bundles. A byte-level specification, pseudocode, test vectors and numerical checks support verification.
Read the full abstract ▾
Information Theory
Derandomizing Dense Binary Hypervector Codebooks for Quantized Scalars
Dmitri Rachkovskij, Evgeny Osipov, Olexander Volkov et al. (+2 more)
Hyperdimensional computing and vector symbolic architectures often represent quantized scalar levels by dense binary codebooks whose level-to-level similarity is intended to follow a prescribed function of scalar separation. At finite dimensionality, randomized scalar codebook constructions deviate from this target because of sampling noise, random-start imbalance, update-count fluctuations, component dependence, and finite-capacity effects. We develop a transition-based derandomization framework for dense binary scalar codebooks across two target-similarity families, with similarity decaying exponentially or linearly with level separation. The framework separates the target similarity law, the derandomization variant, and the concrete generator construction, making explicit how initialization, selection, update, and capacity-handling mechanisms shape the induced similarity profile. We formalize derandomization variants that separately constrain initial Hamming weight, update-count variability, and update balance, thereby controlling distinct sources of finite-dimensional error. For each family and variant, we derive the induced mean similarity, identify realization-wise and mean target-matching regimes, and derive exact finite-dimensional expressions for bias, variance, and root-mean-square error. Simulations across dimensions, quantization ranges, reference scalar levels, and generator constructions validate the theory and show how each constraint removes or reduces a specific source of similarity mismatch. The results provide practical guidance for choosing scalar codebook generators that more closely match a desired similarity law under finite-dimensional and hardware-relevant constraints.
Read the full abstract ▾
All authors Dmitri Rachkovskij, Evgeny Osipov, Olexander Volkov, Denis Kleyko, Vaclav Snasel
Information Theory
Systematic Codes Correcting Two Deletions with Three Redundant Symbols over Large Alphabets
Valery Grishin and Aryeh Lev Zabokritskiy
A literal-systematic $q$-ary code leaves its $n$ information symbols unchanged and appends deletion-prone checks. For $n\ge2$, a known lower bound requires three such checks to correct two arbitrary deletions. We give a uniform construction meeting this minimum over a power-of-two alphabet $q<2^{32}n^{10}$, with
polynomial-time encoding and decoding. It combines boundary authentication, symmetric field checks, and a pointwise computable coloring of a sparse conflict graph. For one adjacent burst of length at most two, we give a closed-form construction with three check symbols over $q_{\mathrm{adj}}<(n+2)^4$. Alphabet size and fixed-alphabet bit redundancy remain separate objectives.
Read the full abstract ▾
Information Theory
Approximate Uniformity in Finite Convolution Models
Nilava Metya and Satyaki Mukherjee
The problem of the uniform law being a sum of two independent distributions has been well studied. Here, we study the approximation of the uniform law to a sum of two distributions with fixed support, under the following discrepancies: the Manhattan distance, the Euclidean distance, the forward Kullback–Leibler divergence and the Wasserstein-one distance based on the line metric. The problem of membership of the uniform law in this model has been well studied. Explicit results are obtained in two models, one where one of the distributions is a Bernoulli and the other when both distributions have the same support, including one
conjecture. Finally, we show an application of our reconstruction method to a recent
conjecture about the coding capacity of an additive noise channel.
Read the full abstract ▾
Information Theory
An information identity reveals the geometry of deviation events
Akshay Balsubramani
In a deviation event, the empirical measure of $n$ independent draws lands in a set of distributions. The exponent of its probability splits, at every $n$, into two nonnegative terms. The first is the rate: $n$ times the relative
entropy, from the population, of the distribution of a typical draw under the event. The second is the dependence that conditioning induces among the draws, measured by their total correlation. Their relative sizes depend on the geometry of the set. This paper focuses on this exact split and on the geometry that the interplay of its two terms reveals. The dependence vanishes exactly when the event confines every draw to a single set, and the classical rate of that constraint is exact. On a set invariant under a compact group that fixes the population and leaves no invariant set of intermediate probability, the rate vanishes; uniform convergence over a hypothesis class that such a group permutes is one. On a half-space whose threshold sits a fixed number of standard errors above the mean, the fraction of the exponent that is dependence tends to a function of the event's probability alone. On a set split into pieces, the dependence is the pieces' average plus $n$ times the Jensen-Shannon divergence of their marginals, minus the
entropy of their weights. The split extends to a general reference law. When the information projection is such a law, the split gives the exact value of the expectation in the exponential change of measure to the projection.
Read the full abstract ▾
Information Theory · Cross-listed from quant-ph
The exact strong converse exponent for private communication over quantum channels
Hao-Chung Cheng, Christoph Hirche and Marco Tomamichel
We determine the exact strong converse exponent for secret-key transmission and generation over general finite-dimensional
quantum wiretap channels, measuring reliability and secrecy jointly by squared fidelity to an ideal secret key. We introduce a novel Renyi private information whose regularization characterizes this exponent. An exact integral representation in terms of ordinary private information gives a uniform continuity bound, showing that the regularized quantity converges to private capacity as the Renyi order approaches one. This establishes for any wiretap channel an exponential fidelity decay at every rate above private capacity.
Read the full abstract ▾
Information Theory · Cross-listed from physics.optics
Receiver-Informed Transmitter for 200Gbps C-Band Direct-Detection PON With FFE-Free ONU
Safiya Dabwan, Geraldo Gomes and Stylianos Sygletos
We introduce and numerically demonstrate receiver-informed transmitter (RITX) waveform design for 200 Gb/s PAM-4 C-band direct-detection PON. Using 50 GHz optoelectronics, RITX achieves N1/E1 loss classes with FFE-Free ONU, enabling 40 km reach and 48.1%/66.7% network-wide complexity savings at 1:64/1:128 splits
Read the full abstract ▾
Information Theory
Mutual Information Constrained Chernoff Bottleneck
Dier Tang and Guangyue Han
The classical information bottleneck (IB) measures the relevance of a representation $U$ of $X$ to a target $Y$ by $I(U;Y)$, which does not directly characterize the error of downstream decisions. For a binary hypothesis $Y$ inferred from many separately encoded observations, the optimal error exponent is the Chernoff information between the two conditional distributions of $U$ given $Y$. We study the mutual information constrained Chernoff bottleneck, which seeks an encoder that maximizes this Chernoff information subject to a rate constraint $I(U;X) \leq R$. We show that its optimal value $C(R)$ increases strictly up to $R = H(V)$, where $V$ merges the symbols of $X$ with equal likelihood ratio, remains at the uncompressed exponent beyond, and, unlike the IB curve, need not be concave. We further show that $k+1$ outputs suffice to attain $C(R)$, where $k$ is the cardinality of $V$. We propose an alternating
algorithm that updates the encoder via a generalized Blahut–Arimoto
algorithm and the Chernoff parameter $s$ via a nonlinear equation, and prove that its iterates remain feasible, with nondecreasing and convergent Chernoff information. Numerical experiments confirm the theory, and on real topic-detection data from the 20 Newsgroups corpus, compressing each word to only $17\%$ of its
entropy retains $90\%$ of the error exponent and nearly the accuracy of the uncompressed classifier.
Read the full abstract ▾
Information Theory
Innovation-Based Sampling for New Information
Jiping Luo, Anthony Ephremides and Nikolaos Pappas
This work introduces innovation-based sampling of continuous
stochastic processes, in which a sample is generated only when the process reveals “sufficiently new” information relative to all previous samples. The resulting innovation process admits a
lattice structure and a three-dimensional state representation. For a Wiener process, we characterize the direction, timing, and frequency of innovations. We show that direction reversals become increasingly rare and establish limit
theorems for their number. Innovations also become progressively sparser: the expected number of innovations grows only as the square root of time, and hence the sampling rate vanishes asymptotically. We then study remote estimation from sparsely received innovations. Notably, the absence of an innovation carries information: the minimum mean-square error (MMSE) estimate evolves with the age of information (AoI) and achieves a substantial MSE reduction over the conventional silence-ignorant zero-order hold (ZOH)
estimator. Finally, we develop tractable affine-age and exponential-age approximations for practical use. Overall, information is conveyed not only by the content of innovations, but also by their direction and timing.
Read the full abstract ▾
Information Theory · Cross-listed from quant-ph
Resonance Breaking in Noisy Shor's Algorithm
Zhengwei Liu
We argue that the exponential
quantum advantage in Shor's
algorithm is broken under one-layer depolarizing noise of arbitrary small error rate, by analyzing a resonance breaking phenomenon in noisy
quantum circuits. First, we express the distribution of measurements on $n$-
qubit strings as the superposition of the $4^n$ wave functions in the Pauli path integral. In the noiseless case, the bit-strings achieving resonant peaks guaranteed a constant rate of successful measurements to factor a large number. Secondly, when the middle layer of the
quantum circuit has independent depolarizing noise of rate $\lambda$, for any Hamming weight $d$, we obtain corresponding measurement rate bounded by $2(1-\lambda)^d$ for higher frequency terms and by $O(n^d)/2^{n/2}$ for low frequency terms. The rate approaches to zero as $n$ and $d$ approaches to infinity. Resonance breaking destroys the exponential
quantum advantage. Furthermore, we design a classical factoring
algorithm in
polynomial time $O(n^d)$ to substitute the low frequency contribution.
Read the full abstract ▾
Information Theory
Order-Optimal Systematic Permutation Codes for Correcting t Deletions
Bolin Wu, Quan D. Bui, Kai Niu et al. (+2 more)
This paper investigates the construction of full-systematic permutation codes capable of correcting multiple deletions under two complementary models, namely symbol-invariant deletions (SIDs), where surviving symbol values are preserved, and permutation-invariant deletions (PIDs), where the surviving sequence is standardized to a permutation. For any fixed integer $t \ge 1$ and all sufficiently large message lengths $n$, our proposed encoders map any message permutation of length $n$ to a codeword by inserting distinct redundancy symbols while strictly preserving the sequence order of the original message symbols. The proposed constructions correct up to $t$ deletions using $7t-1$ redundancy markers for PIDs and $4t$ redundancy markers for SIDs, achieving redundancies of $(7t-1)\log n + O_t(1)$ bits and $4t\log n + O_t(1)$ bits, respectively. Both code families are uniformly constructible, encodable, and decodable in $n^{O(t)}$ time. The underlying framework stores an inner deletion-correcting syndrome in the relative positions of redundancy markers via an algebraic outer code based on integer moments and residual graph coloring. We further extend this framework to fixed-composition and strictly $\lambda$-regular multipermutations, proving that the PID and SID channels coincide whenever the common multiplicity satisfies $\lambda > t$.
Read the full abstract ▾
All authors Bolin Wu, Quan D. Bui, Kai Niu, Van Khu Vu, Shuche Wang
Information Theory
On Task Scope and Information Retention in Source Coding
Alireza Furutanpey and Kerstin Bunte
We argue that dividing codec design into Coding for Machines (CfM) and Coding for Humans (CfH) is a misleading distinction for deciding what information a codec may discard. Receiver identity does not determine admissible information loss. The required rate depends on task scope, including the predictions to support, their losses and tolerated risks, the encoder observation, and the permitted decoding procedures. Notably, a machine task may have a higher minimum rate than a restricted human decision. Rate savings on selected machine tasks apply only to the stated requirements, not to an intrinsic ordering by receiver type. We extend source and feature coding to finite task families, derive when restricting the encoder observation preserves the minimum rate, and show that equality between source and split-feature coding rates can no longer hold as the task scope expands.
Read the full abstract ▾
Information Theory · Cross-listed from cs.MA
RAVEN: Receiver-Conditioned Action-Value Encoding for Finite-Alphabet Multi-Agent Communication
Shuwei Sun, Chenxi Wang, Jian Huang et al. (+2 more)
A message drawn from a small alphabet helps a teammate only if it keeps the distinctions that change that teammate's next decision. We show that scoring messages by action values averaged over the receiver's situation can erase exactly these distinctions, and we propose RAVEN (Receiver-conditioned Action-Value ENcoding), which trains a four-symbol, one-step-delayed channel to preserve each receiver's centered action-value profile within the receiver's own context. The sender never needs to know that context: the receiver decodes every symbol with its private information. We give two
estimators of this target. With a teacher, offline RAVEN selects the codebook that exactly minimizes an empirical conditional distortion and distills it into a frozen sender; we bound the resulting codebook-selection error and one-step decision loss. Without a teacher, online RAVEN aligns, inside a QMIX learner, the deployed symbol pathway with a training-only continuous reference that shares its routing. Against five recent communication methods on eight navigation settings, offline RAVEN attains the highest return in seven, and removing receiver conditioning forfeits 83% of its communication gain. Online RAVEN raises predator-prey capture success from 53.2% to 96.0% over the same QMIX backbone without communication, and on SMAC and MPE it attains the best mean normalized score of 14 methods, including methods that exchange kilobit messages. Every RAVEN message costs 2 bits, 12-1,024x fewer than those of NDQ, CACOM and ExpoComm on navigation.
Read the full abstract ▾
All authors Shuwei Sun, Chenxi Wang, Jian Huang, Weiyun Ru, Hui Cao