Multi-field Features Representation Based Colorization of Grayscale Images
LI Hong'an1, ZHENG Qiaoxue1, MA Tian1, ZHANG Jing1, LI Zhanli1, KANG Baosheng2
1.College of Computer Science and Technology, Xi'an University of Science and Technology, Xi'an 710054; 2.School of Information Science and Technology, Northwest University, Xi'an 710127
Abstract Image colorization improves image quality by predicting color information of gray-scale images. Although the grayscale images can be colored automatically by deep learning methods, the colorization quality of targets with different scales in the images is not satifactory. Especially, the existing colorizing methods is confronted with problems of color overflow, mis-coloring and inconsistent image colors, while dealing with complex objects and small target objects. To address these problems, a method for image colorization of multi-field features representation is proposed in the paper. Firstly, the multi-field feature representation block(MFRB) is designed and combined with the upgraded U-Net to acquire multi-field feature representation U-Net. Then, a grayscale image is input into the U-Net and the color image is obtained by adversarial training with PatchGAN. Finally, the VGG-19 network is employed to compute the perceptual loss of pictures at different scales to enhance the general consistency of the image colorization results. Experimental results on six distinct datasets demonstrate that the proposed method successfully enhances the quality of colorized images and creates color images with richer colors and more consistent tones. The results of the proposed method outperform the main colorization algorithms in both quantitative assessment and subjective perception.
Fund:Supported by National Natural Science Foundation of China(No.61902311), Natural Science Basic Research Plan of Shaanxi Province(No.2022JM-508,2022JM-317)
Corresponding Authors:LI Hong'an, Ph.D., associate professor. His research interests include computer gra-phics and visual media computing.
About author:: About Author:
ZHENG Qiaoxue, master student. His research interests include visual media computing and artificial intelligence.
MA Tian, Ph.D., associate professor. His research interests include 3D simulation, image processing and data visualization.
ZHANG Jing, Ph.D., lecturer. Her research interests include graphic image processing and intelligent information proce-ssing.
LI Zhanli, Ph.D., professor. His research interests include intelligent information processing, visual computing and visualization.
KANG Baosheng, Ph.D., professor. His research interests include computer graphics and image processing.
LI Hong'an,ZHENG Qiaoxue,MA Tian等. Multi-field Features Representation Based Colorization of Grayscale Images[J]. Pattern Recognition and Artificial Intelligence, 2022, 35(7): 637-648.
[1] KHAN M U G, GOTOH Y, NIDA N. Medical Image Colorization for Better Visualization and Segmentation // Proc of the Annual Conference on Medical Image Understanding and Analysis. Berlin, Germany: Springer, 2017: 571-580. [2] LIZUKA S, SIMO-SERRA E, ISHIKAWA H. Let There Be Color! Joint End-to-End Learning of Global and Local Image Priors for Automatic Image Colorization with Simultaneous Classification. ACM Transactions on Graphics, 2016, 35(4). DOI: 10.1145/2897824.2925974. [3] ZHANG R, ISOLA P, EFROS A A. Colorful Image Colorization // Proc of the European Conference on Computer Vision. Berlin, Germany: Springer, 2016: 649-666. [4] DESHPANDE A, LU J J, YEH M C, et al. Learning Diverse Image Colorization // Proc of the IEEE Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2017: 2877-2885. [5] HE M M, CHEN D D, LIAO J, et al. Deep Exemplar-Based Colorization. ACM Transactions on Graphics, 2018, 37(4). DOI: 10.1145/3197517.3201365. [6] LEVIN A, LISCHINSKI D, WEISS Y. Colorization Using Optimization. ACM Transactions on Graphics, 2004, 23(3): 689-694. [7] LIU Y F, XU K, YAN L Q. Adaptive BRDF-Oriented Multiple Importance Sampling of Many Lights. Computer Graphics Forum, 2019, 38(4): 123-133. [8] HEU J H, HYUN D Y, KIM C S, et al. Image and Video Colorization Based on Prioritized Source Propagation // Proc of the 16th IEEE International Conference on Image Processing. Washington, USA: IEEE, 2009: 465-468. [9] WANG Y L, LIU Y F, XU K. An Improved Geometric Approach for Palette-Based Image Decomposition and Recoloring. Computer Graphics Forum, 2019, 38(7): 11-22. [10] 曹丽琴,商永星,刘婷婷,等.局部自适应的灰度图像彩色化.中国图象图形学报, 2019, 24(8): 1249-1257. (CAO L Q, SHANG Y X, LIU T T, et al. Novel Image Colorization of a Local Adaptive Weighted Average Filter. Journal of Image and Graphics, 2019, 24(8): 1249-1257.) [11] 李洪安,张敏,杜卓明,等.一种基于分块特征的交互式图像色彩编辑方法.红外与激光工程, 2019, 48(12): 293-298. (LI H A, ZHANG M, DU Z M, et al. Interactive Image Color Editing Method Based on Block Feature. Infrared and Laser Engineering, 2019, 48(12): 293-298.) [12] LI F, NG M K. Image Colorization by Using Graph Bi-laplacian. Advances in Computational Mathematics, 2019, 45(3): 1521-1549. [13] ZHANG Q, XIAO C X, SUN H Q, et al. Palette-Based Image Recoloring Using Color Decomposition Optimization. IEEE Transactions on Image Processing, 2017, 26(4): 1952-1964. [14] LI B, LAI Y K, JOHN M, et al. Automatic Example-Based Image Colorization Using Location-Aware Cross-Scale Matching. IEEE Tran-sactions on Image Processing, 2019, 28(9): 4606-4619. [15] HU S M, LIANG D, YANG G Y, et al. Jittor: A Novel Deep Learning Framework with Meta-Operators and Unified Graph Execution. Science China Information Sciences, 2020, 63(12). DOI: 10.1007/s11432-020-3097-4. [16] ZHANG S H, LI R L, DONG X, et al. Pose2Seg: Detection Free Human Instance Segmentation // Proc of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2019: 889-898. [17] LIANG Y, WANG X T, ZHANG S H, et al. PhotoRecomposer: Interactive Photo Recomposition by Cropping. IEEE Transactions on Visualization and Computer Graphics, 2018, 24(10): 2728-2742. [18] LI H A, ZHANG M, YU Z H, et al. An Improved pix2pix Model Based on Gabor Filter for Robust Color Image Rendering. Mathematical Biosciences and Engineering, 2022, 19(1): 86-101. [19] 李洪安,郑峭雪,张婧,等.结合Pix2Pix生成对抗网络的灰度图像着色方法.计算机辅助设计与图形学学报, 2021, 33(6): 929-938. (LI H A, ZHENG Q X, ZHANG J, et al. Pix2Pix-Based Grays-cale Image Coloring Method. Journal of Computer-Aided Design and Computer Graphics, 2021, 33(6): 929-938.) [20] SANGKLOY P, LU J W, FANG C, et al. Scribbler: Controlling Deep Image Synthesis with Sketch and Color // Proc of the IEEE Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2017: 6836-6845. [21] YOO S, BAHNG H, CHUNG S, et al. Coloring with Limited Data: Few-Shot Colorization via Memory Augmented Networks // Proc of the IEEE/CVF Conference on Computer Vision and Pattern Re-cognition. Washington, USA: IEEE, 2019: 11275-11284. [22] JOHARI M M, BEHROOZI H. Context-Aware Colorization of Gray-Scale Images Utilizing a Cycle-Consistent Generative Adversarial Network Architecture. Neurocomputing, 2020, 407: 94-104. [23] LIN T Y, DOLLÁR P, GIRSHICK R, et al. Feature Pyramid Networks for Object Detection // Proc of the IEEE Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2017: 936-944. [24] ZHAO H S, SHI J P, QI X J, et al. Pyramid Scene Parsing Network // Proc of the IEEE Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2017: 6230-6239. [25] CHEN C F, FAN Q F, MALLINAR N, et al. Big-Little Net: An Efficient Multi-scale Feature Representation for Visual and Speech Recognition[C/OL].[2022-04-24]. https://arxiv.org/pdf/1807.03848.pdf. [26] MENG Y, LIN C C, PANDA R, et al. AR-Net: Adaptive Frame Resolution for Efficient Action Recognition // Proc of the 16th European Conference on Computer Vision. Berlin, Germany: Springer, 2020: 86-104. [27] JOHNSON J, ALAHI A, LI F F. Perceptual Losses for Real-Time Style Transfer and Super-Resolution // Proc of the European Conference on Computer Vision. Berlin, Germany: Springer, 2016: 694-711. [28] RAD M S, BOZORGTABAR B, MARTI U V, et al. SROBB: Targeted Perceptual Loss for Single Image Super-Resolution // Proc of the IEEE/CVF International Conference on Computer Vision. Washington, USA: IEEE, 2019: 2710-2719. [29] CHEN L F, YANG Z, MA J J, et al. Driving Scene Perception Network: Real-Time Joint Detection, Depth Estimation and Semantic Segmentation // Proc of the IEEE Winter Conference on Applications of Computer Vision. Washington, USA: IEEE, 2018: 1283-1291. [30] XIAO X, LIAN S, LUO Z M, et al. Weighted Res-UNet for High-Quality Retina Vessel Segmentation // Proc of the 9th International Conference on Information Technology in Medicine and Education. Washington, USA: IEEE, 2018: 327-331. [31] GOODFELLOW I J, POUGET-ABADIE J, MIRZA M, et al. Generative Adversarial Nets // Proc of the 27th International Conference on Neural Information Processing Systems. Cambridge, USA: The MIT Press, 2014: 2672-2680. [32] NAZERI K, NG E, EBRAHIMI M. Image Colorization Using Generative Adversarial Networks // Proc of the International Conference on Articulated Motion and Deformable Objects. Berlin, Germany: Springer, 2018: 85-94. [33] VITORIA P, RAAD L, BALLESTER C. ChromaGAN: Adversarial Picture Colorization with Semantic Class Distribution // Proc of the IEEE Winter Conference on Applications of Computer Vision. Washington, USA: IEEE, 2020: 2434-2443. [34] ISOLA P, ZHU J Y, ZHOU T H, et al. Image-to-Image Translation with Conditional Adversarial Networks // Proc of the IEEE Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2017: 5967-5976. [35] GUAN S, KHAN A A, SIKDAR S, et al. Fully Dense UNet for 2D Sparse Photoacoustic Tomography Artifact Removal. IEEE Journal of Biomedical and Health Informatics, 2019, 24(2): 568-576. [36] IBTEHAZ N, RAHMAN M S. MultiResUNet: Rethinking the U-Net Architecture for Multimodal Biomedical Image Segmentation. Neural Networks, 2020, 121: 74-87. [37] GATYS L A, ECKER A S, BETHGE M.A Neural Algorithm of Artistic Style[C/OL]. [2022-04-24].https://arxiv.org/pdf/1508.06576.pdf. [38] GATYS L A, ECKER A S, BETHGE M. Image Style Transfer Using Convolutional Neural Networks // Proc of the IEEE Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2016: 2414-2423. [39] ANWAR S, TAHIR M, LI C Y, et al. Image Colorization: A Survey and Dataset[C/OL].[2022-04-24]. https://arxiv.org/pdf/2008.10774.pdf. [40] ZHANG R, ISOLA P, EFROS A A, et al. The Unreasonable Effectiveness of Deep Features as a Perceptual Metric // Proc of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2018: 586-595.
2022, Vol. 35 
