PruneTrain: Fast Neural Network Training by Dynamic Sparse Model Reconfiguration

Authors: Sangkug Lym (University of Texas), Esha Choukse (University of Texas), Siavash Zangeneh (University of Texas), Wei Wen (Duke University), Sujay Sanghavi (University of Texas), Mattan Erez (University of Texas)

Abstract: State-of-the-art convolutional neural networks (CNNs) used in vision applications have large models with numerous weights. Training these models is very compute- and memory-resource intensive. Much research has been done on pruning or compressing these models to reduce the cost of inference, but little work has addressed the costs of training. We focus precisely on accelerating training. We propose PruneTrain, a cost-efficient mechanism that gradually reduces the training cost during training. PruneTrain uses a structured regularization approach that drives the training optimization toward both high accuracy and small weight values. Small weights can then be periodically removed by reconfiguring the network model to a smaller one. By using a structured-pruning approach and additional reconfiguration techniques we introduce, the pruned model can still be efficiently processed on a GPU accelerator. Overall, PruneTrain achieves a reduction of 39% in the training time of modern CNNs.


Presentation: file


Back to Technical Papers Archive Listing