Abstract Pruning is an effective way to reduce the complexity of the model. In the existing pruning methods, only the influence of the convolutional layer on the feature map is taken into account,and therefore the redundant filter cannot be determined accurately. In this paper, a dynamic network structured pruning method based on layer fusion feature coefficients is proposed. Considering the influence of convolutional layer and Batch Norm layer on the feature map, the importance of the filter is determined by multiple dynamic parameters, and the redundant filter is dynamically searched to obtain the optimal network structure. Experiments on the standard datasets of CIFAR-10 and CIFAR-100 shows that both the residual network and the lightweight network maintain high accuracy while using large pruning rates.
Fund:Supported by National Natural Science Foundation of China(No.61876090,61936005 ), National Major Project of China for New Generation of AI(No. 2018AAA0100401)
Corresponding Authors:
YUAN Xiaotong, Ph.D., professor. His research interests include machine learning and pattern recognition.
About author:: LU Haiwei, master student. His research interests include neural network model compression.
[1] DENIL M, SHAKIBI B, DINH L, et al. Predicting Parameters in Deep Learning // Proc of the 26th International Conference on Neural Information Processing Systems. Cambridge, USA: The MIT Press, 2013: 2148-2156. [2] DENTON E, ZAREMBA W, BRUNA J, et al. Exploiting Linear Structure with Convolutional Networks for Efficient Evaluation // Proc of the 27th International Conference on Neural Information Processing Systems. Cambridge, USA: The MIT Press, 2014: 1269-1277. [3] IANDOLA F N, MOSKEWICZ M W, ASHRAF K, et al. Squeeze-Net: AlexNet-Level Accuracy with 50x Fewer Parameters and< 0.5 MB Model Size[C/OL]. [2019-07-06]. https://arxiv.org/pdf/1602.07360v2.pdf. [4] HOWARD A G, ZHU M L, CHEN B, et al. Mobilenets: Efficient Convolutional Neural Networks for Mobile Vision Applications[C/OL]. [2019-07-06]. https://arxiv.org/pdf/1704.04861.pdf. [5] ZHANG X Y, ZHOU X Y, LIN M X, et al. ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devices // Proc of the IEEE Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2018: 6848-6856. [6] HUANG G, LIU Z, VAN DER MAATEN L, et al. Densely Connected Convolutional Networks // Proc of the IEEE Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2017: 2261-2269. [7] HINTON G, VINYALS O, DEAN J. Distilling the Knowledge in a Neural Network[C/OL]. [2019-07-06]. https://arxiv.org/pdf/1503.02531.pdf. [8] BA L J, CARUANA R. Do Deep Nets Really Need to be Deep? // Proc of the 27th International Conference on Neural Information Processing Systems[C/OL]. [2019-07-06]. https://arxiv.org/pdf/1312.6184.pdf. [9] COURBARIAUX M, BENGIO Y, DAVID J P. Binaryconnect: Trai-ning Deep Neural Networks with Binary Weights during Propagations // Proc of the 28th International Conference on Neural Information Processing Systems. Cambridge, USA: The MIT Press, 2015: 3123-3131. [10] RASTEGARI M, ORDONEZ V, REDMON J, et al. Xnor-Net: Imagenet Classification Using Binary Convolutional Neural Networks // Proc of the European Conference on Computer Vision. Berlin, Germany: Springer, 2016: 525-542. [11] ZHOU A J, YAO A B, GUO Y W, et al. Incremental Network Quantization: Towards Lossless CNNs with Low-Precision Weights[C/OL]. [2019-07-06]. http://ariv.prg/pdf/1702.03044.pdf. [12] HAN S, POOL J, TRAN J, et al. Learning Both Weights and Connections for Efficient Neural Network // Proc of the 28th International Conference on Neural Information Processing Systems. Cambridge, USA: The MIT Press, 2015: 1135-1143. [13] GUO Y W, YAO A B, CHEN Y R. Dynamic Network Surgery for Efficient DNNs // Proc of the 29th International Conference on Neural Information Processing Systems. Cambridge, USA: The MIT Press, 2016: 1379-1387. [14] JIN X J, YUAN X T, FENG J S, et al. Training Skinny Deep Neural Networks with Iterative Hard Thresholding Methods[C/OL]. [2019-07-06]. https://arxiv.org/pdf/1607.05423.pdf. [15] HE Y H, ZHANG X Y, SUN J. Channel Pruning for Accelerating Very Deep Neural Networks // Proc of the IEEE International Conference on Computer Vision. Washington, USA: IEEE, 2017: 1398-1406. [16] IOFFE S, SZEGEDY C. Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift[C/OL]. [2019-07-06]. https://arxiv.org/pdf/1502.03167.pdf. [17] SRIVASTAVA N, HINTON G, KRIZHEVSKY A, et al. Dropout: A Simple Way to Prevent Neural Networks from Overfitting[C/OL]. [2019-07-06]. https://arxiv.org/pdf/1607.05423.pdf. [18] HE K M, ZHANG X Y, REN S Q, et al. Deep Residual Learning for Image Recognition // Proc of the IEEE Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2016: 770-778. [19] DONG X Y, HUANG J S, YANG Y, et al. More Is Less: A More Complicated Network with Less Inference Complexity // Proc of the IEEE Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2017: 1895-1903. [20] HE Y, KANG G L, DONG X Y, et al. Soft Filter Pruning for Accelerating Deep Convolutional Neural Networks[C/OL]. [2019-07-06]. https://arxiv.org/pdf/1808.06866v1.pdf. [21] LI H, KADAV A, DURDANOVIC I, et al. Pruning Filters for Efficient Convnets[C/OL]. [2019-07-06]. https://arxiv.org/pdf/ 1608.08710.pdf.
2019, Vol. 32 
