Skip to main content

Non-vacuous Generalization Bounds at the ImageNet Scale: a PAC-Bayesian Compression Approach

Author(s): Zhou, Wenda; Veitch, Victor; Austern, Morgane; Adams, Ryan P; Orbanz, Peter

To refer to this page use:
Abstract: Modern neural networks are highly overparameterized, with capacity to substantially overfit to training data. Nevertheless, these networks often generalize well in practice. It has also been observed that trained networks can often be compressed to much smaller representations. The purpose of this paper is to connect these two empirical observations. Our main technical result is a generalization bound for compressed networks based on the compressed size that, combined with off-the-shelf compression algorithms, leads to state-of-the-art generalization guarantees. In particular, we provide the first non-vacuous generalization guarantees for realistic architectures applied to the ImageNet classification problem. Additionally, we show that compressibility of models that tend to overfit is limited. Empirical results show that an increase in overfitting increases the number of bits required to describe a trained network.
Publication Date: 2019
Citation: Zhou, Wenda, Victor Veitch, Morgane Austern, Ryan P. Adams, and Peter Orbanz. "Non-vacuous Generalization Bounds at the ImageNet Scale: a PAC-Bayesian Compression Approach." International Conference on Learning Representations (2019).
Type of Material: Conference Article
Journal/Proceeding Title: International Conference on Learning Representations (ICLR)
Version: Final published version. Article is made available in OAR by the publisher's permission or policy.

Items in OAR@Princeton are protected by copyright, with all rights reserved, unless otherwise indicated.