Skip to main content

Incremental Learning Using a Grow-and-Prune Paradigm With Efficient Neural Networks

Author(s): Dai, Xiaoliang; Yin, Hongxu; Jha, Niraj K

To refer to this page use:
Abstract: Deep neural networks (DNNs) have become a widely deployed model for numerous machine learning applications. However, their fixed architecture, substantial training cost, and significant model redundancy make it difficult to efficiently update them to accommodate previously unseen data. To solve these problems, we propose an incremental learning framework based on a grow-and-prune neural network synthesis paradigm. When new data arrive, the neural network first grows new connections based on the gradients to increase the network capacity to accommodate new data. Then, the framework iteratively prunes away connections based on the magnitude of weights to enhance network compactness, and hence recover efficiency. Finally, the model rests at a lightweight DNN that is both ready for inference and suitable for future grow- and-prune updates. The proposed framework improves accuracy, shrinks network size, and significantly reduces the additional training cost for incoming data compared to conventional approaches, such as training from scratch and network fine-tuning. For the LeNet-300-100 (LeNet-5) neural network architectures derived for the MNIST dataset, the framework reduces training cost by up to 64 (67), 63 (63), and 69 (73 percent) compared to training from scratch, network fine-tuning, and grow-and-prune from scratch, respectively. For the ResNet-18 architecture derived for the ImageNet dataset (DeepSpeech2 for the AN4 dataset), the corresponding training cost reductions against training from scratch, network fine-tunning, and grow-and-prune from scratch are 64 (67), 60 (62), and 72 (71 percent), respectively. Our derived models contain fewer network parameters but achieve higher accuracy relative to conventional baselines.
Publication Date: 10-Nov-2020
Citation: Dai, Xiaoliang, Yin, Hongxu, Jha, Niraj K. (2022). Incremental Learning Using a Grow-and-Prune Paradigm With Efficient Neural Networks. IEEE Transactions on Emerging Topics in Computing, 10 (2), 752 - 762. doi:10.1109/tetc.2020.3037052
DOI: doi:10.1109/tetc.2020.3037052
EISSN: 2168-6750
Pages: 752 - 762
Type of Material: Journal Article
Journal/Proceeding Title: IEEE Transactions on Emerging Topics in Computing
Version: Author's manuscript

Items in OAR@Princeton are protected by copyright, with all rights reserved, unless otherwise indicated.