Abstract Enhancing low-light images with high quality is a highly challenging task due to the features of low-light images such as brightness, color, and details in the information of different scales. Existing deep learning-based methods fail to fully utilize multi-scale features and fuse multi-scale features to comprehensively enhance the brightness, color and details of the images. To address these problems, a Transformer-based multi-scale optimization network for low-light image enhancement is proposed. Firstly, the Transformer-based multi-task enhancement module is designed. Through multi-task training, the Transformer-based enhancement module gains the ability to globally model brightness, color, and details. Therefore, it can initially cope with various degradation challenges commonly found in low-light images, such as insufficient brightness, color deviation, blurred details and severe noises. Then, the architecture combining global and local multi-scale features is designed to progressively optimize the features at different scales. Finally, a multi-scale feature fusion module and an adaptive enhancement module are proposed. They learn and fuse the information association among different scales, while adaptively enhancing images in various local multi-scale branches. Extensive experiments on six public datasets, including paired or unpaired images, show that the proposed method can effectively solve the problems of multiple degradation types, such as brightness, color, details and noise in low-light images.
Fund:General Program of National Natural Science Foundation of China(No.61972097)
Corresponding Authors:
CHEN Yuzhong, Ph.D., professor. His research interests include computational intelligence and data mi-ning.
About author:: About Author:NIU Yuzhen, Ph.D., professor. Her research interests include computer vision and artificial intelligence.LIN Xiaofeng, master student. Her research interests include deep learning and image enhancement.XU Huangbiao, master student. His research interests include computer vision, and image and video understanding.LI Yuezhou, Ph.D. candidate. His research interests include image and video understanding, and image restoration.
[1] LI C Y, GUO C L, HAN L H, et al. Low-Light Image and Video Enhancement Using Deep Learning: A Survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021, 44(12): 9396-9416. [2] LÜ F F, LI Y, LU F.Attention Guided Low-Light Image Enhancement with a Large Scale Low-Light Simulation Dataset. International Journal of Computer Vision, 2021, 129(7): 2175-2193. [3] ARICI T, DIKBAS S, ALTUNBASAK Y.A Histogram Modification Framework and Its Application for Image Contrast Enhancement. IEEE Transactions on Image Processing, 2009, 18(9): 1921-1935. [4] CELIK T, TJAHJADI T.Contextual and Variational Contrast En-hancement. IEEE Transactions on Image Processing, 2011, 20(12): 3431-3441. [5] XU H T, ZHAI G T, WU X L, et al. Generalized Equalization Model for Image Enhancement. IEEE Transactions on Multimedia, 2013, 16(1): 68-82. [6] LEE C, LEE C, KIM C S.Contrast Enhancement Based on Layered Difference Representation of 2D Histograms. IEEE Transactions on Image Processing, 2013, 22(12): 5372-5384. [7] FU X Y, ZENG D L, HUANG Y, et al. A Fusion-Based Enhancing Method for Weakly Illuminated Images. Signal Processing, 2016, 129: 82-96. [8] FU X Y, ZENG D L, HUANG Y, et al. A Weighted Variational Model for Simultaneous Reflectance and Illumination Estimation//Proc of the IEEE Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2016: 2782-2790. [9] GUO X J, LI Y, LING H B.LIME: Low-Light Image Enhancement via Illumination Map Estimation. IEEE Transactions on Image Processing, 2016, 26(2): 982-993. [10] LI M D, LIU J Y, YANG W H, et al. Structure-Revealing Low-Light Image Enhancement via Robust Retinex Model. IEEE Transactions on Image Processing, 2018, 27(6): 2828-2841. [11] WEI C, WANG W J, YANG W H, et al. Deep Retinex Decomposition for Low-Light Enhancement[C/OL].[2023-03-16]. https://arxiv.org/pdf/1808.04560.pdf. [12] ZHANG Y H, ZHANG J W, GUO X J. Kindling the Darkness: A Practical Low-Light Image Enhancer//Proc of the 27th ACM International Conference on Multimedia. New York, USA: ACM, 2019: 1632-1640. [13] LIU R S, MA L, ZHANG J A, et al. Retinex-Inspired Unrolling with Cooperative Prior Architecture Search for low-Light Image Enhancement//Proc of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2021: 10556-10565. [14] JIANG Y F, GONG X Y, LIU D, et al. EnlightenGAN: Deep Light Enhancement without Paired Supervision. IEEE Transactions on Image Processing, 2021, 30: 2340-2349. [15] GUO C L, LI C Y, GUO J C, et al. Zero-Reference Deep Curve Estimation for Low-Light Image Enhancement//Proc of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2020: 1777-1786. [16] LI C Y, GUO C L, LOY C C.Learning to Enhance Low-Light Image via Zero-Reference Deep Curve Estimation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022, 44(8): 4225-4238. [17] CHEN H T, WANG Y H, GUO T Y, et al. Pre-trained Image Processing Transformer//Proc of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2021: 12294-12305. [18] WANG Z D, CUN X D, BAO J M, et al. Uformer: A General U-Shaped Transformer for Image Restoration//Proc of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2022: 17662-17672. [19] 武英. 基于双直方图均衡的自适应图像增强算法.计算机工程, 2011, 37(4): 244-245. (WU Y.Adaptive Image Enhancement Algorithm Based on Bi-histogram Equalization. Computer Engineering, 2011, 37(4): 244-245) [20] 李乐鹏,孙水发,夏冲,等.直方图均衡技术综述.计算机系统应用, 2014, 23(3): 1-8. (LI L P, SUN S F, XIA C, et al. Survey of Histogram Equaliza-tion Technology. Computer Systems and Applications, 2014, 23(3):1-8.) [21] LEE H G, YANG S, SIM J Y. Color Preserving Contrast Enhancement for Low Light Level Images Based on Retinex//Proc of the Asia-Pacific Signal and Information Processing Association Annual Summit and Conference. Washington, USA: IEEE, 2015: 884-887. [22] JOBSON D J, RAHMAN Z, WOODELL G A.A Multiscale Re-tinex for Bridging the Gap between Color Images and the Human Observation of Scenes. IEEE Transactions on Image Processing, 1997, 6(7): 965-976. [23] JOBSON D J, RAHMAN Z, WOODELL G A.A Multiscale Re-tinex for Bridging the Gap between Color Images and the Human Observation of Scenes. IEEE Transactions on Image Processing, 1997, 6(7): 965-976. [24] 欧嘉敏,胡晓,杨佳信.改进Retinex-Net 的低光照图像增强算法.模式识别与人工智能, 2021, 34(1): 77-86. (OU J M, HU X, YANG J X.Low-Light Image Enhancement Algorithm Based on Improved Retinex-Net. Pattern Recognition and Artificial Intelligence, 2021, 34(1): 77-86.) [25] 江泽涛,覃露露.一种基于U-Net生成对抗网络的低照度图像增强方法.电子学报, 2020, 48(2): 258-264. (JIANG Z T, QIN L L.Low-Light Image Enhancement Method Based on U-Net Generative Adversarial Network. Acta Electronica Sinica, 2020, 48(2): 258-264. [26] 尚晓可,安南,尚敬捷,等.结合视觉显著性与注意力机制的低光照图像增强.模式识别与人工智能, 2022, 35(7): 602-613. (SHANG X K, AN N, SHANG J J, et al. Combining Visual Saliency and Attention Mechanism for Low-Light Image Enhancement. Pattern Recognition and Artificial Intelligence, 2022, 35(7): 602-613.) [27] KIM H, CHOI S M, KIM C S, et al. Representative Color Transform for Image Enhancement//Proc of the IEEE/CVF Internatio-nal Conference on Computer Vision. Washington, USA: IEEE, 2021: 4439-4448. [28] ZHANG Z, ZHENG H, HONG R C, et al. Deep Color Consistent Network for Low-Light Image Enhancement//Proc of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2022: 1889-1898. [29] 江泽涛,伍旭,张少钦.一种基于MR-VAE的低照度图像增强方法.计算机学报, 2020, 43(7): 1328-1339. (JIANG Z T, WU X, ZHANG S Q.Low-Illumination Image Enhancement Base on MR-VAE. Chinese Journal of Computers, 2020, 43(7): 1328-1339.) [30] WANG Y, CAO Y, ZHA Z J, et al. Progressive Retinex: Mutually Reinforced Illumination-Noise Perception Network for Low-Light Image Enhancement//Proc of the 27th ACM International Confe-rence on Multimedia. New York, USA: ACM, 2019: 2015-2023. [31] DOSOVITSKIY A, BEYER L, KOLESNIKOV A, et al. An Image Is Worth 16×16 Words: Transformers for Image Recognition at Scale[C/OL].[2023-03-16]. https://arxiv.org/pdf/2010.11929.pdf. [32] JI J Y, LUO Y P, SUN X S, et al. Improving Image Captioning by Leveraging Intra-and Inter-Layer Global Representation in Transformer Network. Proceedings of the AAAI Conference on Artificial Intelligence, 2021, 35(2): 1655-1663. [33] CARION N, MASSA F, SYNNAEVE G, et al. End-to-End Object Detection with Transformers//Proc of the 16th European Confe-rence on Computer Vision. Berlin, Germany: Springer, 2020: 213-229. [34] XIE E Z, WANG W H, YU Z D, et al. SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers[C/OL].[2023-03-16]. https://arxiv.org/pdf/2105.15203.pdf. [35] PARMAR N, VASWANI A, USZKOREIT J, et al.Image Transformer[C/OL].[2023-03-16]. https://arxiv.org/pdf/1802.05751.pdf. [36] HU J, SHEN L, SUN G. Squeeze-and-Excitation Networks//Proc of the IEEE/CVF Conference on Computer Vision and Pattern Re-cognition. Washington, USA: IEEE, 2018: 7132-7141. [37] HOU Q B, ZHOU D Q, FENG J S. Coordinate Attention for Efficient Mobile Network Design//Proc of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2021: 13708-13717. [38] WOO S, PARK J, LEE J Y, et al. CBAM: Convolutional Block Attention Module//Proc of the European Conference on Computer Vision. Berlin, Germany: Springer, 2018: 3-19. [39] 陈晓雷,卢禹冰,曹宝宁,等.轻量化高精度双通道注意力机制模块.计算机科学与探索, 2023, 17(4): 857-867. (CHEN X L, LU Y B, CAO B N, et al. Lightweight and High-Precision Dual-Channel Attention Mechanism Module. Journal of Frontiers of Computer Science and Technology, 2023, 17(4): 857-867.) [40] HUI Z, GAO X B, YANG Y C, et al. Lightweight Image Superresolution with Information Multi-distillation Network//Proc of the 27th ACM International Conference on Multimedia. New York, USA: ACM, 2019: 2024-2032. [41] LI J Q, LI J C, FANG F M, et al. Luminance-Aware Pyramid Network for Low-Light Image Enhancement. IEEE Transactions on Multimedia, 2020, 23: 3153-3165. [42] DENTON E, CHINTALA S, SZLAM A, et al. Deep Generative Image Models Using a Laplacian Pyramid of Adversarial Networks//Proc of the 28th International Conference on Neural Information Processing Systems. New York, USA: ACM, 2015: 1486-1494. [43] WENG R L, LU J W, TAN Y P, et al. Learning Cascaded Deep Autoencoder Networks for Face Alignment. IEEE Transactions on Multimedia, 2016, 18(10): 2066-2078. [44] NIU Y Z, LIN Z H, LIU W X, et al. Progressive Moire Removal and Texture Complementation for Image Demoireing. IEEE Transactions on Circuits and Systems for Video Technology, 2023. DOI: 10.1109/TCSVT.2023.3237810. [45] SUN K, XIAO B, LIU D, et al. Deep High-Resolution Representation Learning for Human Pose Estimation//Proc of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2019: 5686-5696. [46] HE K M, ZHANG X Y, REN S Q, et al. Deep Residual Learning for Image Recognition//Proc of the IEEE Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2016: 770-778. [47] LIU Y H, OTT M, GOYAL N, et al. RoBERTa: A Robustly Optimized BERT Pretraining Approach[C/OL].[2023-03-16]. https://arxiv.org/pdf/1907.11692.pdf. [48] IOFFE S, SZEGEDY C. Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift//Proc of the 32nd International Conference on Machine Learning. San Diego, USA: JMLR, 2015: 448-456. [49] KIM B, LEE S, KIM N, et al. Learning Color Representations for Low-Light Image Enhancement//Proc of the IEEE/CVF Winter Conference on Applications of Computer Vision. Washington, USA: IEEE, 2022: 904-912. [50] CHEN C, CHEN Q F, XU J, et al. Learning to See in the Dark//Proc of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2018: 3291-3300. [51] JIANG H Y, ZHENG Y Q. Learning to See Moving Objects in the Dark//Proc of the IEEE/CVF International Conference on Computer Vision. Washington, USA: IEEE, 2019: 7323-7332. [52] LEE C, LEE C F, LEE Y Y, et al. Power-Constrained Contrast Enhancement for Emissive Displays Based on Histogram Equalization. IEEE Transactions on Image Processing, 2011, 21(1): 80-93. [53] CHEN C, CHEN Q F, DO M, et al. Seeing Motion in the Dark//Proc of the IEEE/CVF International Conference on Computer Vision. Washington, USA: IEEE, 2019: 3184-3193. [54] HAI J, XUAN Z, HAN S C, et al. R2RNET: Low-Light Image Enhancement via Real-Low to Real-Normal Network. Journal of Visual Communication and Image Representation, 2023, 90. DOI: 10.1016/j.jvcir.2022.103712. [55] ZAMIR S W, ARORA A, KHAN S, et al. Learning Enriched Features for Real Image Restoration and Enhancement//Proc of the European Conference on Computer Vision. Berlin, Germany: Springer, 2020: 492-511. [56] YANG W H, WANG S Q, FANG Y M, et al. From Fidelity to Perceptual Quality: A Semi-Supervised Approach for Low-Light Image Enhancement//Proc of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2020: 3060-3069. [57] ZHANG Y H, GUO X J, MA J Y, et al. Beyond Brightening Low-Light Images. International Journal of Computer Vision, 2021, 129(4): 1013-1037.
2023, Vol. 36 
