Continual Zero-Shot Learning Algorithm Based on Latent Vectors Alignment
ZHONG Xiaorong1, HU Xiao2, DING Jiayu1
1. School of Electronics and Communication Engineering, Guangzhou University, Guangzhou 510006; 2. School of Mechanical and Electrical Engineering, Guangzhou University, Guangzhou 510006
Abstract Continual zero-shot learning aims to accumulate the knowledge of seen classes and utilize the knowledge for unseen classes recognition. However, catastrophic forgetting can easily occur in continual learning. Therefore, a continual zero-shot learning algorithm based on latent vectors alignment is proposed. Based on the cross and distribution aligned variational auto-encoder network, the visual latent vectors of current tasks and learned tasks are aligned to enhance the similarity of latent space of different tasks. Selective retraining is adopted to improve the discrimination ability of the current task model for learned tasks. For different tasks, the independent classifiers are trained with visual-hidden vectors of the seen classes and semantic-hidden vectors of the unseen classes to achieve zero-shot image classification. Extensive experiments on four standard datasets show that the proposed algorithm completes the continual zero-shot recognition task effectively and alleviates the catastrophic forgetting.
Fund:National Natural Science Foundation of China(No.62076075)
Corresponding Authors:
HU Xiao, Ph.D., professor. His research interests include machine learning, machine vision and video behavior analysis.
About author:: ZHONG Xiaorong, master student. Her research interests include computer vision and zero shot learning. DING Jiayu, master student. His research interests include artificial intelligence, computer vision and zero shot learning.
[1] LAMPERT C H, NICKISCH H, HARMELING S.Attribute-Based Classification for Zero-Shot Visual Object Categorization. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2014, 36(3): 453-465. [2] WEI W, ZHENG V W, YU H, et al. A Survey of Zero-Shot Learning: Settings. A Survey of Zero-Shot Learning: Settings, Methods, Applications. ACM Transactions on Intelligent Systems and Technology, 2019, 10(2): 13.1-13.37. [3] ZHANG Z M, SALIGRAMA V.Zero-Shot Learning via Joint Latent Similarity Embedding // Proc of the IEEE Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2016: 6034-6042. [4] MA P R, HU X.A Variational Autoencoder with Deep Embedding Model for Generalized Zero-Shot Learning // Proc of the 34th AAAI Conference on Artificial Intelligence. Palo Alto, USA: AAAI Press, 2020: 11733-11740. [5] DING J Y, HU X, ZHONG X R.A Semantic Encoding Out-of-Distribution Classifier for Generalized Zero-Shot Learning. IEEE Signal Processing Letters, 2021, 28: 1395-1399. [6] SCHÖNFELD E, EBRAHIMI S, SINHA S, et al. Generalized Zero-Shot and Few-Shot Learning via Aligned Variational Autoencoders // Proc of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2019: 8239-8247. [7] SCHMIDHUBER J.Deep Learning in Neural Networks: An Overview. Neural Networks, 2015, 61: 85-117. [8] KINGMA D P, WELLING M. Auto-Encoding Variational Bayes[C/OL]. [2021-06-10]. https://arxiv.org/pdf/1312.6114.pdf. [9] GOODFELLOW I J, POUGET-ABADIE J, MIRZA M, et al. Ge-nerative Adversarial Networks[C/OL]. [2021-06-10]. https://arxiv.org/pdf/1406.2661.pdf. [10] WEI K, DENG C, YANG X.Lifelong Zero-Shot Learning // Proc of the 29th International Joint Conference on Artificial Intelligence and Seventeenth Pacific Rim International Conference on Artificial Intelligence. San Francisco, USA: Morgan Kaufmann, 2020: 551-557. [11] DE LANGE M, ALJUNDI R, MASANA M, et al. A Continual Learning Survey: Defying Forgetting in Classification Tasks[C/OL].[2021-06-10]. https://arxiv.org/pdf/1909.08383.pdf. [12] CHAUDHRY A, DOKANIA P K, AJANTHAN T, et al. Riema-nnian Walk for Incremental Learning: Understanding Forgetting and Intransigence // Proc of the European Conference on Computer Vision. Berlin, Germany: Springer, 2018: 556-572. [13] CACCIA M, RODRIGUEZ P, OSTAPENKO O, et al. Online Fast Adaptation and Knowledge Accumulation(OSAKA): A New Approach to Continual Learning[C/OL].[2021-06-10]. https://arxiv.org/pdf/2003.05856.pdf. [14] HAYES T L, CAHILL N D, KANAN C.Memory Efficient Experience Replay for Streaming Learning // Proc of the International Conference on Robotics and Automation. Washington, USA: IEEE, 2019: 9769-9776. [15] RUSU A A, RABINOWITZ N C, DESJARDINS G, et al. Progre-ssive Neural Networks[C/OL].[2021-06-10]. https://arxiv.org/pdf/1606.04671.pdf. [16] LOPZE-PAZ D, RANZATO M.Gradient Episodic Memory for Con-tinual Learning // Proc of the 31st International Conference on Neural Information Processing Systems. Cambridge, USA: The MIT Press, 2017: 6470-6479. [17] PARISI G I, KEMKER R, PART J L, et al. Continual Lifelong Learning with Neural Networks: A Review. Neural Networks, 2019, 113: 54-71. [18] CHAUDHRY A, RANZATO M, ROHRABACH M, et al. Efficient Lifelong Learning with A-GEM[C/OL].[2021-06-10]. https://arxiv.org/pdf/1812.00420v1.pdf. [19] CHANDAN G, SETHUPANTHY P, ASHISH M, et al. Online Lifelong Generalized Zero-Shot Learning[C/OL].[2021-06-10]. https://arxiv.org/pdf/2103.10741.pdf. [20] GHOSH S.Dynamic VAEs with Generative Replay for Continual Zero-Shot Learning[C/OL]. [2021-06-10].https://arxiv.org/pdf/2104.12468.pdf. [21] KIRKPATRICK J, PASCANU R, RABINOWITZ N, et al. Overcoming Catastrophic Forgetting in Neural Networks. Proceedings of the National Academy of Sciences of the United States of America, 2017, 114(13): 3521-3526. [22] LIU X L, MASANA M, HERRANZ L, et al. Rotate Your Networks: Better Weight Consolidation and Less Catastrophic Forge-tting // Proc of the 24th International Conference on Pattern Recog-nition. Washington, USA: IEEE, 2018: 2262-2268. [23] CHEN S M, WANG W J, XIA B H, et al. FREE: Feature Refinement for Generalized Zero-Shot Learning[C/OL].[2021-06-10]. https://arxiv.org/pdf/2107.13807.pdf. [24] VYAS M R, VENKATESWARA H, PANCHANATHAN S.Leve-raging Seen and Unseen Semantic Relationships for Generative Zero-Shot Learning // Proc of the European Conference on Computer Vision. Berlin, Germany: Springer, 2020: 70-86. [25] 林克正,李昊天,白婧轩,等.语义自编码结合关系网络的零样本图像识别算法.模式识别与人工智能, 2019, 32(3): 214-224. (LIN K Z, LI H T, BAI J X, et al. Zero-Shot Image Recognition Algorithm via Semantic Auto-Encoder Combining Relation Network. Pattern Recognition and Artificial Intelligence, 2019, 32(3): 214-224.) [26] KESHARI R, SINGH R, VATSA M.Generalized Zero-Shot Lear-ning via Over-Complete Distribution // Proc of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2020: 13297-13305. [27] FARHADI A, ENDRES I, HOIEM D, et al. Describing Objects by Their Attributes // Proc of the IEEE Conference on Computer Vision. Describing Objects by Their Attributes // Proc of the IEEE Conference on Computer Vision and Pattern Recognition. Washington, USA:IEEE, 2009, I: 1778-1785. [28] PATTERSON G, HAYS J.SUN Attribute Database: Discovering, Annotating, and Recognizing Scene Attributes // Proc of the IEEE Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2012: 2751-2758.
2021, Vol. 34 
