Presentation
PruneTrain: Fast Neural Network Training by Dynamic Sparse Model Reconfiguration
Event Type
Paper
TP
Applications
Data Management
Deep Learning
GPUs
Machine Learning
Performance
Scalable Computing
BSP Finalist
TimeWednesday, 20 November 201910:30am - 11am
Location401-402-403-404
DescriptionState-of-the-art convolutional neural networks (CNNs) used in vision applications have large models with numerous weights. Training these models is very compute- and memory-resource intensive. Much research has been done on pruning or compressing these models to reduce the cost of inference, but little work has addressed the costs of training. We focus precisely on accelerating training. We propose PruneTrain, a cost-efficient mechanism that gradually reduces the training cost during training. PruneTrain uses a structured regularization approach that drives the training optimization toward both high accuracy and small weight values. Small weights can then be periodically removed by reconfiguring the network model to a smaller one. By using a structured-pruning approach and additional reconfiguration techniques we introduce, the pruned model can still be efficiently processed on a GPU accelerator. Overall, PruneTrain achieves a reduction of 39% in the training time of modern CNNs.
Download PDF
Archive
