基于指数移动平均知识蒸馏的神经网络低比特量化方法

doi:10.16451/j.cnki.issn1003-6059.202112007

摘要
图/表
参考文献
相关文章 (15)

全文: PDF (681 KB) HTML (1 KB)
输出: BibTeX | EndNote (RIS)

摘要目前存储和计算成本严重阻碍深度神经网络应用和推广,而神经网络量化是一种有效的压缩方法.神经网络低比特量化存在的显著困难是量化比特数越低,网络分类精度也越低.为了解决这一问题,文中提出基于指数移动平均知识蒸馏的神经网络低比特量化方法.首先利用少量图像进行自适应初始化,训练激活和权重的量化步长,加快量化网络收敛.再引入指数移动平均(EMA)知识蒸馏的思想,利用EMA对蒸馏损失和任务损失进行归一化,指导量化网络训练.在ImageNet、CIFAR-10数据集上的分类任务表明,文中方法可获得接近或超过全精度网络的性能.

	服务

	把本文推荐给朋友
	加入我的书架
	加入引用管理器
	E-mail Alert
	RSS
	作者相关文章

关键词 ：深度学习, 网络量化, 知识蒸馏, 模型压缩

Abstract：Now the memory and computational cost restrict the popularization of deep neural network application, whereas neural network quantization is an effective compression method. As the number of quantized bits is lower, the classification accuracy of neural networks becomes poorer in low-bit quantization of neural networks. To solve this problem, a low-bit quantization method of neural networks based on knowledge distillation is proposed. Firstly, a few images are exploited for adaptive initialization to train the quantization step of activation and weight to speed up the convergence of the quantization network. Then, the idea of exponential moving average knowledge distillation is introduced to normalize distillation loss and task loss and guide the training of quantization network. Experiments on ImageNet and CIFAR-10 datasets show that the performance of the proposed method is close to or better than that of the full precision network.

Key words： Deep Learning Network Quantization Knowledge Distillation Model Compression

收稿日期: 2021-04-20

ZTFLH:

TP 391

基金资助:国家重点研发计划项目(No.2019YFB2204200)、中央高校基本科研业务费项目(No.2020JBM020)、北京市自然科学基金项目(No.4202063)资助

通讯作者: 王东,博士,教授,主要研究方向为图像处理、模型压缩、芯片设计等.E-mail:wangdong@bjtu.edu.cn.

作者简介: 吕君环,硕士研究生,主要研究方向为神经网络量化、模型压缩等.E-mail:lvjunhuan@bjtu.edu.cn.
许柯,博士研究生,主要研究方向为神经网络模型剪枝、量化、模型压缩等.E-mail:xuke225@bjtu.edu.cn.

引用本文:

吕君环, 许柯, 王东. 基于指数移动平均知识蒸馏的神经网络低比特量化方法[J]. 模式识别与人工智能, 2021, 34(12): 1143-1151. LÜ Junhuan, XU Ke, WANG Dong. Low-Bit Quantization of Neural Network Based on Exponential Moving Average Knowledge Distillation. , 2021, 34(12): 1143-1151.

链接本文:

http://manu46.magtech.com.cn/Jweb_prai/CN/10.16451/j.cnki.issn1003-6059.202112007 或 http://manu46.magtech.com.cn/Jweb_prai/CN/Y2021/V34/I12/1143

[1] KRIZHEVSKY A, SUTSKEVER I, HINTON G E.ImageNet Classification with Deep Convolutional Neural Networks // Proc of the 25th International Conference on Neural Information Processing Systems. Cambridge, USA: The MIT Press, 2012: 1097-1105.
[2] HINTON G, DENG L, YU D, et al. Deep Neural Networks for Acoustic Modeling in Speech Recognition: The Shared Views of Four Research Groups. IEEE Signal Processing Magazine, 2012, 29(6): 82-97.
[3] REDMON J, DIVVALA S, GIRSHICK R, et al. You Only Look Once: Unified, Real-Time Object Detection // Proc of the IEEE Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2016: 779-788.
[4] HAN S, POOL J, TRAN J, et al. Learning Both Weights and Connections for Efficient Neural Networks // Proc of the 28th International Conference on Neural Information Processing Systems. Cambridge, USA: The MIT Press, 2015, I: 1135-1143.
[5] WANG P S, HU Q H, ZHANG Y F, et al. Two-Step Quantization for Low-Bit Neural Networks // Proc of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2018: 4376-4384.
[6] HINTON G, VINYALS O, DEAN J.Distilling the Knowledge in a Neural Network[C/OL]. [2021-03-25].https://export.arxiv.org/pdf/1503.02531.
[7] JACOB B, KLIGYS S, CHEN B, et al. Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference // Proc of the IEEE/CVF Conference on Computer Vision and Pa-ttern Recognition. Washington, USA: IEEE, 2018: 2704-2713.
[8] CHOI J, WANG Z, VENKARANI S, et al. PACT: Parameterized Clipping Activation for Quantized Neural Networks[C/OL].[2021-03-25]. https://arxiv.org/pdf/1805.06085v2.pdf.
[9] JUNG S, SON C, LEE S, et al. Learning to Quantize Deep Networks by Optimizing Quantization Intervals with Task Loss // Proc of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2019: 4345-4354.
[10] ESSER S K, MCKINSTRY J L, BLANI D, et al. Learned Step Size Quantization[C/OL].[2021-03-25]. https://arxiv.org/pdf/1902.08153v1.pdf.
[11] JAIN S R, GURAL A, WU M, et al. Trained Quantization Thre-sholds for Accurate and Efficient Fixed-Point Inference of Deep Neural Networks[C/OL].[2021-03-25]. https://arxiv.org/pdf/1903.08066v2.pdf.
[12] ZHOU A J, YAO A B, GUO Y W, et al. Incremental Network Quantization: Towards Lossless CNNs with Low-Precision Weights[C/OL].[2021-03-25]. https://arxiv.org/pdf/1702.03044v2.pdf.
[13] ZHANG D Q, YANG J L, YE D Q Z, et al. LQ-Nets: Learned Quantization for Highly Accurate and Compact Deep Neural Networks // Proc of the European Conference on Computer Vision. Berlin, Germany: Springer, 2018: 373-390.
[14] GAO M Y, SHEN Y J, LI Q Q, et al. Residual Knowledge Disti-llation[C/OL].[2021-03-25]. https://arxiv.org/pdf/2002.09168v1.pdf.
[15] NOWAK T S, CORSO J J.Deep Net Triage: Analyzing the Importance of Network Layers via Structural Compression[C/OL]. [2021-03-25].https://arxiv.org/pdf/1801.04651.pdf.
[16] POLINO A, PASCANU R, ALISTARH D.Model Compression via Distillation and Quantization[C/OL]. [2021-03-25].https://arxiv.org/pdf/1802.05668.pdf.
[17] WEI Y, PAN X Y, QIN H W, et al. Quantization Mimic: Towards Very Tiny CNN for Object Detection // Proc of the European Conference on Computer Vision. Berlin, Germany: Springer, 2018: 274-290.
[18] MISHRA A, MARR D.Apprentice: Using Knowledge Distillation Techniques to Improve Low-Precision Network Accuracy[C/OL]. [2021-03-25].https://arxiv.org/pdf/1711.05852.pdf.
[19] BENGIO Y, LÉONARD N, COURVILLE A. Estimating or Propagating Gradients through Stochastic Neurons for Conditional Computation[C/OL].[2021-03-25]. https://arxiv.org/pdf/1308.3432v1.pdf.
[20] KRIZHEVSKY A.Learning Multiple Layers of Features from Tiny Images[C/OL]. [2021-03-25]. https://www.cs.toronto.edu/~kriz/learning-features-2009-TR.pdf
[21] RUSSAKOVSKY O, DENG J, SU H, et al. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision, 2015, 115(3): 211-252.
[22] HOWARD A G, ZHU M L, CHEN B, et al. MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications[C/OL].[2021-03-25]. https://arxiv.org/pdf/1704.04861.pdf.
[23] SANDLER M, HOWARD A, ZHU M L, et al. MobileNetV2: Inverted Residuals and Linear Bottlenecks // Proc of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2018: 4510-4520.
[24] REDMON J, FARHADI A.YOLO9000: Better, Faster, Stronger // Proc of the IEEE Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2017: 6517-6525.
[25] HE K M, ZHANG X Y, REN S Q, et al. Deep Residual Learning for Image Recognition // Proc of the IEEE Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2016: 770-778.
[26] KINGMA D P, BA J.Adam: A Method for Stochastic Optimization[C/OL]. [2021-03-25]. https://arxiv.org/pdf/1412.6980.pdf.
[27] GONG R H, LIU X L, JIANG S H, et al. Differentiable Soft Quan-tization: Bridging Full-Precision and Low-Bit Neural Networks // Proc of the IEEE/CVF International Conference on Computer Vision. Washington, USA: IEEE, 2019: 4851-4860.
[28] BHALGAT Y, LEE J, NAGEL M, et al. LSQ+: Improving Low-Bit Quantization through Learnable Offsets and Better Initialization // Proc of the IEEE/CVF Conference on Computer Vision and Pa-ttern Recognition Workshops. Washington, USA: IEEE, 2020: 2978-2985.
[29] WANG K, LIU Z J, LIN Y J, et al. HAQ: Hardware-Aware Automated Quantization with Mixed Precision // Proc of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2019: 8604-8612.