SU Qihang1,2, QIAN Yeqiang3, YUAN Wei3, YANG Ming1,2, WANG Chunxiang1,2
1. Department of Automation, Shanghai Jiao Tong University, Shanghai 200240; 2. Key Laboratory of System Control and Information Processing, Ministry of Education, Shanghai Jiao Tong University, Shanghai 200240; 3. UM-SJTU Joint Institute, Shanghai Jiao Tong University, Shanghai 200240
Abstract：To alleviate the problem of the application of deep neural network being restricted owing to the massive computation resources, many network compression strategies including network pruning are put forward. Most of the network pruning methods based on greedy algorithm include training, pruning and fine-tuning, and therefore the optimal pruned structure cannot be obtained. In this paper, combining the rule-based method and the automatic search method, a network pruning method via automatic mending strategy is proposed. The whole pruning process is comprised of four stages: training, pre-pruning, mending and fine-tuning. The structure of the pre-pruned model is improved in the additional mending stage. Particularly, the neural architecture search is utilized to implement network mending. The search space and an efficient search strategy are designed. The estimation process is accelerated based on the filter ranking of the pre-pruning stage. Experiments show that the proposed method can guarantee the network accuracy in the case of high pruning rate.
[1] SIMONYAN K, ZISSERMAN A. Very Deep Convolutional Networks for Large-Scale Image Recognition[C/OL]. [2021-06-20]. https://arxiv.org/pdf/1409.1556.pdf. [2] REN S Q, HE K M, GIRSHICK R, et al. Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017, 39(6): 1137-1149. [3] LONG J, SHELHAMER E, DARRELL T.Fully Convolutional Networks for Semantic Segmentation // Proc of the IEEE Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2015: 3431-3440. [4] RASTEGARI M, ORDONEZ V, REDMON J, et al. XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks // Proc of the European Conference on Computer Vision. Berlin, Germany: Springer, 2016: 525-542. [5] HINTON G, VINYALS O, DEAN J.Distilling the Knowledge in a Neural Network [C/OL]. [2021-06-20].https://arxiv.org/pdf/1503.02531.pdf. [6] DENTON E, ZAREMBA W, BRUNA J, et al. Exploiting Linear Structure within Convolutional Networks for Efficient Evaluation [C/OL]. [2021-06-20]. https://arxiv.org/pdf/ Exploiting Linear Structure within Convolutional Networks for Efficient Evaluation [C/OL]. [2021-06-20]. https://arxiv.org/pdf/1404.0736.pdf. [7] LIU Z, LI J G, SHEN Z Q, et al. Learning Efficient Convolutional Networks through Network Slimming // Proc of the IEEE International Conference on Computer Vision. Washington, USA: IEEE, 2017: 2755-2763. [8] GUO Y W, YAO A B, CHEN Y R.Dynamic Network Surgery for Efficient DNNs // Proc of the 30th International Conference on Neural Information Processing Systems. Cambridge, USA: The MIT Press, 2016: 1387-1395. [9] LIN M B, JI R R, ZHANG Y X, et al. Channel Pruning via Automatic Structure Search // Proc of the 29th International Joint Confe-rence on Artificial Intelligence. New York, USA: ACM, 2020: 673-679. [10] LI H, KADAV A, DURDANOVIC I, et al. Pruning Filters for Efficient ConvNets[C/OL].[2021-06-20]. https:// arxiv.org/pdf/1608.08710.pdf. [11] HE Y H, ZHANG X Y, SUN J.Channel Pruning for Accelerating very Deep Neural Networks // Proc of the IEEE International Conference on Computer Vision. Washington, USA: IEEE, 2017: 1389-1397. [12] LIN S H, JI R R, LI Y C, et al. Accelerating Convolutional Networks via Global & Dynamic Filter Pruning // Proc of the 27th International Joint Conference on Artificial Intelligence. New York, USA: ACM, 2018: 2425-2432. [13] HUANG Z H, WANG N Y.Data-Driven Sparse Structure Selection for Deep Neural Networks // Proc of the European Conference on Computer Vision. Berlin, Germany: Springer, 2018: 317-334. [14] LUO J H, WU J X, LIN W Y.ThiNet: A Filter Level Pruning Method for Deep Neural Network Compression // Proc of the IEEE International Conference on Computer Vision. Washington, USA: IEEE, 2017: 5068-5076. [15] HE Y, LIU P, WANG Z W, et al. Filter Pruning via Geometric Median for Deep Convolutional Neural Networks Acceleration // Proc of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2019: 4340-4349. [16] YE J B, LU X, LIN Z, et al. Rethinking the Smaller-Norm-Less-Informative Assumption in Channel Pruning of Convolution Layers [C/OL].[2021-06-20]. https://arxiv.org/pdf/1802.00124.pdf. [17] DING X H, DING G G, GUO Y C, et al. Centripetal SGD for Pruning Very Deep Convolutional Networks with Complicated Structure // Proc of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2019: 4938-4948. [18] LIN M B, JI R R, WANG Y, et al. HRank: Filter Pruning Using High-Rank Feature Map // Proc of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2020: 1526-1535. [19] HE Y, KANG G L, DONG X Y, et al. Soft Filter Pruning for Accelerating Deep Convolutional Neural Networks // Proc of the 27th International Joint Conference on Artificial Intelligence. New York, USA: ACM, 2018: 2234-2240. [20] CHIN T W, DING R Z, ZHANG C, et al. Towards Efficient Mo-del Compression via Learned Global Ranking // Proc of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2020: 1515-1525. [21] LIU Z C, MU H Y, ZHANG X Y, et al. MetaPruning: Meta Learning for Automatic Neural Network Channel Pruning // Proc of the IEEE/CVF International Conference on Computer Vision. Washington, USA: IEEE, 2019: 3295-3304. [22] LIU Z, SUN M J, ZHOU T H, et al. Rethinking the Value of Network Pruning [C/OL].[2021-06-20]. https://arxiv.org/pdf/1810.05270.pdf. [23] REAL E, MOORE S, SELLE A, et al. Large-Scale Evolution of Image Classifiers // Proc of the 34th International Conference on Machine Learning. New York, USA: ACM, 2017: 2902-2911. [24] ZOPH B, LE Q V.Neural Architecture Search with Reinforcement Learning [C/OL]. [2021-06-20].https://arxiv.org/pdf/1611.01578.pdf. [25] ELSKEN T, METZEN J H, HUTTER F.Neural Architecture Search: A Survey. Journal of Machine Learning Research, 2019, 20: 1-21. [26] REAL E, AGGARWAL A, HUANG Y P, et al. Regularized Evolution for Image Classifier Architecture Search[C/OL].[2021-06-20]. https://arxiv.org/pdf/1802.01548.pdf. [27] ZOPH B, VASUDEVAN V, SHLENS J, et al. Learning Transferable Architectures for Scalable Image Recognition // Proc of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2018: 8697-8710. [28] ZHOU D Z, ZHOU X C, ZHANG W W, et al. EcoNAS: Finding Proxies for Economical Neural Architecture Search // Proc of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2020: 11393-11401. [29] KRIZHEVSKY A. Learning Multiple Layers of Features from Tiny Images[C/OL]. [2021-06-20]. http://www.cs.toronto.edu/~kriz/learning-features-2009-TR.pdf. [30] WAH C, BRANSON S, WELINDER P, et al. The Caltech-UCSD Birds-200-2011 Dataset[C/OL]. [2021-06-20]. http://www.vision.caltech.edu/visipedia/papers/CUB_200_2011.pdf. [31] HE K M, ZHANG X Y, REN S Q, et al. Deep Residual Learning for Image Recognition // Proc of the IEEE Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2016: 770-778. [32] SUTSKEVER I, MARTENS J, DAHL G, et al. On the Importance of Initialization and Momentum in Deep Learning. Journal of the Machine Learning Research, 2013, 28(3): 1139-1147.
2022, Vol. 35 
