1. College of Computer Science and Technology, Ocean University of China, Qingdao 266404; 2. National Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences, Beijing 100190; 3. School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing 100049
Abstract：Network architecture adaptation aims to automatically design and optimize the neural network architectures based on specific learning tasks and data to meet the comprehensive needs of intelligent perception learning tasks in open environment. This paper is intended to provide a comprehensive review of network architecture adaptation methods. Firstly, the main methods of neural architecture search are elucidated and analyzed. Then, the research progress of network architecture adaptation is presented from three aspects: lightweight neural architecture search, intelligent perception tasks and continuous learning. On this basis, an adaptive evaluation index system of deep neural network components and architectures for open environment applications is established, and a network architecture adaptive method is proposed. Through the attention-guided micro-architecture adaptive mechanism and progressive discretization strategy, adaptive adjustment, optimization and gradual discretization of network structures are realized in the optimization process. The proposed method is compared with the existing methods. Finally, problems and challenges of current methods are discussed, and the future research directions are prospected.
[1] ZOPH B, VASUDEVAN V, SHLENS J, et al.Learning Transfe-rable Architectures for Scalable Image Recognition // Proc of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2018: 8697-8710. [2] WANG N, GAO Y, CHEN H, et al.NAS-FCOS: Fast Neural Architecture Search for Object Detection // Proc of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2020: 11940-11948. [3] GUO J Y, HAN K, WANG Y H, et al.Hit-Detector: Hierarchical Trinity Architecture Search for Object Detection // Proc of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2020: 11402-11411. [4] ZHANG X, XU H M, MO H, et al.DCNAS: Densely Connected Neural Architecture Search for Semantic Image Segmentation // Proc of the IEEE/CVF Conference on Computer Vision and Pattern Re-cognition. Washington, USA: IEEE, 2021: 13951-13962. [5] YU Q H, YANG D, ROTH H, et al.C2FNAS: Coarse-to-Fine Neu-ral Architecture Search for 3D Medical Image Segmentation // Proc of the IEEE/CVF Conference on Computer Vision and Pattern Re-cognition. Washington, USA: IEEE, 2020: 4125-4134. [6] CHEN Z K, HUANG Y, YU H Y, et al.Learning a Robust Part-Aware Monocular 3D Human Pose Estimator via Neural Architecture Search. International Journal of Computer Vision, 2022, 130: 1-20. [7] NIE X, LIU Y C, CHEN S H, et al.Differentiable Convolution Search for Point Cloud Processing // Proc of the IEEE/CVF International Conference on Computer Vision. Washington, USA: IEEE, 2021: 7417-7426. [8] LIU H X, SIMONYAN K, VINYALS O K, et al. Hierarchical Re-presentations for Efficient Architecture Search[C/OL].[2023-09-19]. https://arxiv.org/abs/1711.00436. [9] SUGANUMA M, SHIRAKAWA S, NAGAO T.A Genetic Progra-mming Approach to Designing Convolutional Neural Network Architectures // Proc of the Genetic and Evolutionary Computation Confe-rence. New York, USA: ACM, 2017: 497-504. [10] BAKER B, GUPTA O, NAIK N, et al. Designing Neural Network Architectures Using Reinforcement Learning[C/OL].[2023-09-19]. https://arxiv.org/pdf/1611.02167.pdf. [11] CAI H, CHEN T Y, ZHANG W N, et al. Efficient Architecture Search by Network Transformation[C/OL].[2023-09-19]. https://arxiv.org/pdf/1707.04873.pdf. [12] LECUN Y, BOTTOU L, BENGIO Y, et al.Gradient-Based Lear-ning Applied to Document Recognition. Proceedings of the IEEE, 1998, 86(11): 2278-2324. [13] KRIZHEVSKY A, SUTSKEVER I, HINTON G E.ImageNet Cla-ssification with Deep Convolutional Neural Networks. Communications of the ACM, 2017, 60(6): 84-90. [14] SZEGEDY C, LIU W, JIA Y Q, et al.Going Deeper with Convolutions // Proc of the IEEE Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2015. DOI: 10.1109/CVPR.2015.7298594. [15] HE K M, ZHANG X Y, REN S Q, et al.Deep Residual Learning for Image Recognition // Proc of the IEEE Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2016: 770-778. [16] HUANG G, LIU Z, VAN DER MAATEN L, et al. Densely Connected Convolutional Networks // Proc of the IEEE Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2017: 2261-2269. [17] PLEISS G, CHEN D L, HUANG G, et al. Memory-Efficient Implementation of Densenets[C/OL].[2023-09-19]. https://arxiv.org/abs/1707.06990. [18] SIMONYAN K, ZISSERMAN A.Very Deep Convolutional Networks for Large-Scale Image Recognition[C/OL]. [2023-09-19].https://arxiv.org/abs/1409.1556. [19] LIU H X, SIMONYAN K, YANG Y M.DARTS: Differentiable Architecture Search[C/OL]. [2023-09-19].https://arxiv.org/pdf/1806.09055.pdf. [20] CHU X X, WANG X X, ZHANG B, et al. DARTS-: Robustly Stepping Out of Performance Collapse without Indicators[C/OL].[2023-09-19]. https://arxiv.org/pdf/2009.01027.pdf. [21] CHEN X, XIE L X, WU J, et al.Progressive Differentiable Architecture Search: Bridging the Depth Gap between Search and Evaluation // Proc of the IEEE/CVF International Conference on Computer Vision. Washington, USA: IEEE, 2019: 1294-1303. [22] XU Y H, XIE L X, ZHANG X P, et al. PC-DARTS: Partial Channel Connections for Memory-Efficient Architecture Search[C/OL].[2023-09-19]. https://arxiv.org/pdf/1907.05737.pdf. [23] ZHANG M, PAN S K, CHANG X J, et al.BaLeNAS: Differentiable Architecture Search via the Bayesian Learning Rule // Proc of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2022: 11861-11870. [24] CHU X X, ZHOU T B, ZHANG B, et al.Fair DARTS: Eliminating Unfair Advantages in Differentiable Architecture Search // Proc of the European Conference on Computer Vision. Berlin, Germany: Springer, 2020: 465-480. [25] LI S, MAO Y X, ZHANG F C, et al.DLW-NAS: Differentiable Light-Weight Neural Architecture Search. Cognitive Computation, 2023, 15(2): 429-439. [26] YE P, LI B P, LI Y K, et al. β-DARTS: Beta-Decay Regularization for Differentiable Architecture Search // Proc of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2022: 10864-10873. [27] ZOPH B, LE Q V.Neural Architecture Search with Reinforcement Learning[C/OL]. [2023-09-19].https://arxiv.org/abs/1611.01578. [28] STANLEY K O, MIIKKULAINEN R.Evolving Neural Networks through Augmenting Topologies. Evolutionary Computation, 2002, 10(2): 99-127. [29] GOMEZ F J, MIIKKULAINEN R. Solving Non-Markovian Control Tasks with Neuroevolution // Proc of the 16th International Joint Confe-rence on Artificial Intelligence. San Francisco, USA: Morgan Kaufmann, 1999, II: 1356-1361 [30] SALIMANS T, HO J, CHEN X, et al. Evolution Strategies as a Scalable Alternative to Reinforcement Learning[C/OL].[2023-09-19]. https://arxiv.org/pdf/1703.03864.pdf. [31] REAL E, MOORE S, SELLE A, et al.Large-Scale Evolution of Image Classifiers // Proc of the 34th International Conference on Machine Learning. San Diego, USA: JMLR, 2017: 2902-2911. [32] REAL E, AGGARWAL A, HUANG Y P, et al.Regularized Evolution for Image Classifier Architecture Search. Proceedings of the 34th AAAI Conference on Artificial Intelligence, 2019, 33(1): 4780-4789. [33] LIANG H W, ZHANG S F, SUN J C, et al. DARTS+: Improved Differentiable Architecture Search with Early Stopping[C/OL].[2023-09-19]. https://arxiv.org/abs/1909.06035. [34] BI K F, HU C P, XIE L X, et al. Stabilizing DARTS with Amended Gradient Estimation on Architectural Parameters[C/OL].[2023-09-19]. https://arxiv.org/pdf/1910.11831.pdf. [35] ZELA A, ELSKEN T, SAIKIA T, et al. Understanding and Robustifying Differentiable Architecture Search[C/OL].[2023-09-19]. https://arxiv.org/pdf/1909.09656.pdf. [36] WANG R C, CHENG M H, CHEN X N, et al. Rethinking Architecture Selection in Differentiable NAS[C/OL].[2023-09-19]. https://arxiv.org/pdf/2108.04392.pdf. [37] HE C Y, YE H S, SHEN L, et al.MiLeNAS: Efficient Neural Architecture Search via Mixed-Level Reformulation // Proc of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2020: 11990-19999. [38] JIANG Y F, HU C, XIAO T, et al.Improved Differentiable Architecture Search for Language Modeling and Named Entity Recognition // Proc of the Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing. Stroudsburg, USA: ACL, 2019: 3585-3590. [39] YANG Y B, YOU S, LI H Y, et al.Towards Improving the Consistency, Efficiency, and Flexibility of Differentiable Neural Architecture Search // Proc of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2021: 6663-6672. [40] GU Y C, WANG L J, LIU Y, et al.DOTS: Decoupling Operation and Topology in Differentiable Architecture Search // Proc of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2021: 12306-12315. [41] MAO Y X, ZHONG G Q, WANG Y N, et al.Differentiable Light-Weight Architecture Search // Proc of the IEEE International Conference on Multimedia and Expo. Washington, USA: IEEE, 2021. DOI: 10.1109/ICME51207.2021.9428132. [42] WANG W N, ZHANG X W, CUI H F, et al.FP-DARTS: Fast Parallel Differentiable Neural Architecture Search for Image Classification. Pattern Recognition, 2023, 136. DOI: 10.1016/j.patcog.2022.109193. [43] YU H Y, PENG H W, HUANG Y, et al.Cyclic Differentiable Architecture Search. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023, 45(1): 211-228. [44] ZHANG X B, CHANG J L, GUO Y W, et al.DATA: Differenti-able Architecture Approximation with Distribution Guided Sampling. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021, 43(9): 2905-2920. [45] LI Y Y, ZHAO P, YUAN G, et al. Pruning-as-Search: Efficient Neural Architecture Search via Channel Pruning and Structural Reparameterization[C/OL].[2023-09-19]. https://arxiv.org/abs/2206.01198. [46] ZHANG X B, HUANG Z H, WANG N Y, et al.You Only Search Once: Single Shot Neural Architecture Search via Direct Sparse Optimization. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021, 43(9): 2891-2904. [47] LI C L, PENG J F, YUAN L C, et al.Block-Wisely Supervised Neural Architecture Search with Knowledge Distillation // Proc of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2020: 1986-1995. [48] WANG T Z, WANG K, CAI H, et al.APQ: Joint Search for Network Architecture, Pruning and Quantization Policy // Proc of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2020: 2075-2084. [49] WU B C, DAI X L, ZHANG P Z, et al.FBNet: Hardware-Aware Efficient ConvNet Design via Differentiable Neural Architecture Search // Proc of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2019: 10726-10734. [50] WAN A, DAI X L, ZHANG P Z, et al.FBNetV2: Differentiable Neural Architecture Search for Spatial and Channel Dimensions // Proc of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2020: 12962-12971. [51] DAI X L, WAN A, ZHANG P Z, et al.FBNetV3: Joint Architecture-Recipe Search Using Predictor Pretraining // Proc of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2021: 16271-16280. [52] TAN M X, CHEN B, PANG R M, et al.MnasNet: Platform-Aware Neural Architecture Search for Mobile // Proc of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2019: 2815-2823. [53] YANG T J, HOWARD A, CHEN B, et al.NetAdapt: Platform-Aware Neural Network Adaptation for Mobile Applications // Proc of the European Conference on Computer Vision. Berlin, Germany: Springer, 2018: 289-304. [54] YÜZÜGÜLER A C, DIMITRIADIS N, FROSSARD P. U-Boost NAS: Utilization-Boosted Differentiable Neural Architecture Search // Proc of the European Conference on Computer Vision. Berlin, Germany: Springer, 2022: 173-190. [55] YUAN Y, WANG L N, ZHONG G Q, et al.Adaptive Gabor Convolutional Networks. Pattern Recognition, 2022, 124. DOI: 10.1016/j.patcog.2021.108495. [56] ZHENG X W, FEI X, ZHANG L, et al.Neural Architecture Sear-ch with Representation Mutual Information // Proc of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2022: 11902-11911. [57] CHEN Y K, YANG T, ZHANG X Y, et al. DetNAS: Backbone Search for Object Detection[C/OL].[2023-09-19]. https://arxiv.org/pdf/1903.10979.pdf. [58] XIONG Y Y, LIU H X, GUPTA S, et al.MobileDets: Searching for Object Detection Architectures for Mobile Accelerators // Proc of the IEEE/CVF Conference on Computer Vision and Pattern Re-cognition. Washington, USA: IEEE, 2021: 3824-3833. [59] YAN B, PENG H W, WU K, et al.LightTrack: Finding Lightweight Neural Networks for Object Tracking via One-Shot Architecture Search // Proc of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2021: 15175-15184. [60] YAO F Q, WANG S K, DING L H, et al.Lightweight Network Learning with Zero-Shot Neural Architecture Search for UAV Images. Knowledge-Based Systems, 2023, 260. DOI: 10.1016/j.knosys.2022.110142. [61] SHAW A, HUNTER D, LANDOLA F, et al.SqueezeNAS: Fast Neural Architecture Search for Faster Semantic Segmentation // Proc of the IEEE/CVF International Conference on Computer Vision Workshop. Washington, USA: IEEE, 2019: 2014-2024. [62] PENG C, MYRONENKO A, HATAMIZADEH A, et al.HyperSeg-NAS: Bridging One-Shot Neural Architecture Search with 3D Me-dical Image Segmentation Using HyperNet // Proc of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2022: 20709-20719. [63] GAO S H, LI Z Y, HAN Q, et al.RF-Next: Efficient Receptive Field Search for Convolutional Neural Networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022, 45(3): 2984-3002. [64] THOMAS J B, SHIHABUDHEEN K V.Neural Architecture Search Algorithm to Optimize Deep Transformer Model for Fault Detection in Electrical Power Distribution Systems. Engineering Applications of Artificial Intelligence, 2023, 120. DOI: 10.1016/j.engappai.2023.105890. [65] LI X L, ZHOU Y B, WU T F, et al. Learn to Grow: A Continual Structure Learning Framework for Overcoming Catastrophic Forge-tting[C/OL].[2023-09-19]. https://arxiv.org/pdf/1904.00310.pdf. [66] TONIONI A, TOSI F, POGGI M, et al.Real-Time Self-Adaptive Deep Stereo // Proc of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2019: 195-204. [67] ZHANG C H, TIAN K, FAN B, et al.Continual Stereo Matching of Continuous Driving Scenes with Growing Architecture // Proc of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2022: 18879-18888. [68] TAN M X, LE Q V.EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks[C/OL]. [2023-09-19].https://arxiv.org/abs/1905.11946. [69] STAMOULIS D, DING R Z, WANG D, et al.Single-Path NAS: Designing Hardware-Efficient ConvNets in Less Than 4 Hours // Proc of the Joint European Conference on Machine Learning and Knowledge Discovery in Databases. Berlin, Germany: Springer, 2019: 481-497. [70] XIE S K, ZHENG H H, LIU C X, et al. SNAS: Stochastic Neural Architecture Search[C/OL].[2023-09-19]. https://arxiv.org/pdf/1812.09926.pdf. [71] DONG J D, CHENG A C, JUAN D C, et al.DPP-Net: Device-Aware Progressive Search for Pareto-Optimal Neural Architectures // Proc of the European Conference on Computer Vision. Berlin, Germany: Springer, 2018: 540-555. [72] RU B X, LYLE C, SCHUT L, et al. Revisiting the Train Loss: An Efficient Performance Estimator for Neural Architecture Search[C/OL].[2023-09-19]. https://arxiv.org/abs/2006.04492v1. [73] BENDER G, KINDERMANS P J, ZOPH B, et al.Understanding and Simplifying One-Shot Architecture Search // Proc of the 35th International Conference on Machine Learning. San Diego, USA: JMLR, 2018: 550-559. [74] DONG X Y, YANG Y.Searching for a Robust Neural Architecture in Four GPU Hours // Proc of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2019: 1761-1770. [75] WHITE C, NEISWANGER W, SAVANI Y.Bananas: Bayesian Optimization with Neural Architectures for Neural Architecture Search. Proceedings of the AAAI Conference on Artificial Intelligence, 2021, 35(12): 10293-10301. [76] DENG B Y, YAN J J, LIN D H.Peephole: Predicting Network Performance Before Training[C/OL]. [2023-09-19].https://arxiv.org/abs/1712.03351. [77] TANG Y H, WANG Y H, XU Y X, et al.A Semi-Supervised Asse-ssor of Neural Architectures // Proc of the IEEE/CVF Confe-rence on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2020: 1807-1816. [78] XU Y X, WANG Y H, HAN K, et al.RENAS: Relativistic Eva-luation of Neural Architecture Search // Proc of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2021: 4409-4418. [79] CHEN Y F, GUO Y, CHEN Q, et al.Contrastive Neural Architecture Search with Neural Architecture Comparators // Proc of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2021: 9491-9506. [80] KLEIN A, FALKNER S, SPRINGENBERG J T, et al. Learning Curve Prediction with Bayesian Neural Networks[C/OL].[2023-09-19]. https://openreview.net/pdf?id=S11KBYclx. [81] ZHANG X Y, HOU P F, ZHANG X Y, et al.Neural Architecture Search with Random Labels // Proc of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2021: 10902-10911. [82] HU Y M, LIANG Y D, GUO Z C, et al.Angle-Based Search Space Shrinking for Neural Architecture Search // Proc of the 16th European Conference on Computer Vision. Berlin, Germany: Springer, 2020: 119-134. [83] JEONG J, YU J, PARK G, et al. GeNAS: Neural Architecture Search with Better Generalization[C/OL].[2023-09-19]. https://arxiv.org/pdf/2305.08611.pdf. [84] CHEN X N, HSIEH C J.Stabilizing Differentiable Architecture Search via Perturbation-Based Regularization // Proc of the 37th International Conference on Machine Learning. San Diego, USA: JMLR, 2020: 1554-1565. [85] DYMAK S, LI M C, SOLTANOLKOTABI M.Generalization Gua-rantees for Neural Architecture Search with Train-Validation Split[C/OL]. [2023-09-19].https://arxiv.org/pdf/2104.14132v2.pdf. [86] MOK J, NA B, KIM J H, et al.Demystifying the Neural Tangent Kernel from a Practical Perspective: Can It Be Trusted for Neural Architecture Search Without Training? // Proc of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2022: 11851-11860. [87] CHEN W Y, GONG X Y, WANG Z Y.Neural Architecture Search on ImageNet in Four GPU Hours: A Theoretically Inspired Perspective[C/OL]. [2023-09-19].https://arxiv.org/pdf/2102.11535.pdf. [88] MELLOR J, TURNER J, STORKEY A, et al. Neural Architecture Search Without Training[C/OL].[2023-09-19]. https://arxiv.org/pdf/2006.04647.pdf. [89] LEE N, AJANTHAN T, TORR P H S. SNIP: Single-Shot Network Pruning Based on Connection Sensitivity[C/OL].[2023-09-19]. https://arxiv.org/abs/1810.02340 [90] WANG C Q, ZHANG G D, GROSSE R.Picking Winning Tickets Before Training by Preserving Gradient Flow[C/OL]. [2023-09-19].https://arxiv.org/pdf/2002.07376.pdf. [91] TANAKA H, KUNIN D, YAMINS D L K, et al. Pruning Neural Networks without Any Data by Iteratively Conserving Synaptic Flow // Proc of the 34th International Conference on Neural Information Processing Systems. Cambridge, USA: MIT Press, 2020: 6377-6389. [92] ZHANG Z H, JIA Z H.Gradsign: Model Performance Inference with Theoretical Insights[C/OL]. [2023-09-19].https://arxiv.org/pdf/2110.08616.pdf. [93] LOPES V, ALIREZAZADEH S, ALEXANDRE L A.EPE-NAS: Efficient Performance Estimation Without Training for Neural Architecture Search // Proc of the International Conference on Artificial Neural Networks. Berlin, Germany: Springer, 2021: 552-563. [94] LIU W, ANGUELOV D, ERHAN D, et al.SSD: Single Shot Multi-Box Detector // Proc of the European Conference on Computer Vision. Berlin, Germany: Springer, 2016: 21-37. [95] LIN G S, LIU F Y, MILAN A, et al.RefineNet: Multi-Path Refinement Networks for Dense Prediction. IEEE Transactions on Pa-ttern Analysis and Machine Intelligence, 2020, 42(5): 1228-1242.
2023, Vol. 36 
