Bayesian Effective Dimension: A Mutual Information Approach

This paper introduces the Bayesian effective dimension, a novel concept for understanding dimension reduction in high-dimensional Bayesian inference. It uses mutual information to quantify the number of statistically learnable directions in the parameter space, offering a unifying perspective on shrinkage priors, regularization, and approximate Bayesian methods. The paper's significance lies in providing a formal, quantitative measure of effective dimensionality, moving beyond informal notions like sparsity and intrinsic dimension. This allows for a better understanding of how these methods work and how they impact uncertainty quantification.