面向高动态范围成像的内容恢复和鬼影抑制网络

doi:10.16451/j.cnki.issn1003-6059.202404003

Abstract
Figure/Table
References (46)
Related Citation (15)

Download: PDF (4208 KB) HTML (1 KB)
Export: BibTeX | EndNote (RIS)

Abstract

High dynamic range(HDR) imaging based on multi-exposure fusion aims to generate high-quality HDR images by integrating the information from multiple low dynamic range(LDR) images. However, HDR imaging is faced with two major challenges, ghosting artifact suppression in motion regions and lost information restoration in over-saturated areas. To comprehensively address the challenges of restoring missing content from reference images and suppressing ghosting artifacts in motion regions, a missing content restoration and ghosting suppression network for high dynamic range imaging is proposed in this paper. In terms of content restoration, a predictive filtering-based content restoration block is introduced. The filtering kernel predicted by the content restoration block is employed to filter reference image features, integrating key information from both reference images and non-reference images to provide richer information for effective reconstruction of missing content. To suppress ghosting artifacts in motion regions and fully exploit complementary information from non-reference images, deformable convolutions are introduced to align features from non-reference images with those from the reference image. Additionally, to enhance the HDR image reconstruction capability of the network, a three-branch image reconstruction module is constructed, including a main branch and two auxiliary branches. The auxiliary branches assist the main branch with better preserved details during the generation of HDR results. Experimental results demonstrate superior performance of the proposed network.

Key words： High Dynamic Range Imaging Ghosting Suppression Predictive Filtering Content Restoration

Received: 11 December 2023

ZTFLH:

TP 391.41

Fund:

National Natural Science Foundation of China(No.62161015), Yunnan Fundamental Research Projects(No.202301AV070004)

Corresponding Authors: LI Huafeng, Ph.D., professor. His research interests include image processing and computer vision.

About author:: YANG Zhenmei, Master student. Her research interests include pattern recognition and image processing. ZHANG Yafei, Ph.D., associate profe-ssor. Her research interests include image processing and pattern recognition.

	Service

	E-mail this article
	Add to my bookshelf
	Add to citation manager
	E-mail Alert
	RSS
	Articles by authors
	YANG Zhenmei
	LI Huafeng
	ZHANG Yafei

Cite this article:

YANG Zhenmei,LI Huafeng,ZHANG Yafei. Missing Content Restoration and Ghosting Suppression Network for High Dynamic Range Imaging[J]. Pattern Recognition and Artificial Intelligence, 2024, 37(4): 313-327.

URL:

http://manu46.magtech.com.cn/Jweb_prai/EN/10.16451/j.cnki.issn1003-6059.202404003 OR http://manu46.magtech.com.cn/Jweb_prai/EN/Y2024/V37/I4/313

[1] MA K D, ZENG K, WANG Z. Perceptual Quality Assessment for Multi-exposure Image Fusion. IEEE Transactions on Image Proce-ssing, 2015, 24(11): 3345-3356.
[2] PRABHAKAR K R, SRIKAR V S, BABU R V. Deepfuse: A Deep Unsupervised Approach for Exposure Fusion with Extreme Exposure Image Pairs // Proc of the IEEE International Conference on Computer Vision. Washington, USA: IEEE, 2017: 4724-4732.
[3] DENG X, DRAGOTTI P L. Deep Convolutional Neural Network for Multi-modal Image Restoration and Fusion. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020, 43(10): 3333-3348.
[4] DENG X, ZHANG Y T, XU M, et al. Deep Coupled Feedback Network for Joint Exposure Fusion and Image Super-Resolution. IEEE Transactions on Image Processing, 2021, 30: 3098-3112.
[5] WARD G. Fast, Robust Image Registration for Compositing High Dynamic Range Photographs from Hand-Held Exposures. Journal of Graphics Tools, 2003, 8(2): 17-30.
[6] ZIMMER H, BRUHN A, WEICKERT J. Freehand HDR Imaging of Moving Scenes with Simultaneous Resolution Enhancement. Computer Graphics Forum, 2011, 30(2): 405-414.
[7] SEN P, KALANTARI N K, YAESOUBI M, et al. Robust Patch-Based HDR Reconstruction of Dynamic Scenes. ACM Transactions on Graphics, 2012, 31(6). DOI: 10.1145/2366145.2366222.
[8] HU J, GALLO O, PULLI K, et al. HDR Deghosting: How to Deal with Saturation? // Proc of the IEEE Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2013: 1163-1170.
[9] RAMAN S, CHAUDHURI S. Reconstruction of High Contrast Images for Dynamic Scenes. The Visual Computer, 2011, 27: 1099-1114.
[10] YAN Q S, SUN J Q, LI H S, et al. High Dynamic Range Imaging by Sparse Representation. Neurocomputing, 2017, 269: 160-169.
[11] MA K D, LI H, YONG H W, et al. Robust Multi-exposure Image Fusion: A Structural Patch Decomposition Approach. IEEE Transactions on Image Processing, 2017, 26(5): 2519-2532.
[12] LI H, MA K D, YONG H W, et al. Fast Multi-scale Structural Patch Decomposition for Multi-exposure Image Fusion. IEEE Transac-tions on Image Processing, 2020, 29: 5805-5816.
[13] BAKER S, SCHARSTEIN D, LEWIS J P, et al. A Database and Evaluation Methodology for Optical Flow. International Journal of Computer Vision, 2011, 92. DOI: 10.1007/s11263-010-0390-2.
[14] KALANTARI N K, RAMAMOORTHI R. Deep High Dynamic Range Imaging of Dynamic Scenes. ACM Transactions on Graphics, 2017, 36(4). DOI: 10.1145/3072959.3073609.
[15] WU S Z, XU J R, TAI Y W, et al. Deep High Dynamic Range Imaging with Large Foreground Motions // Proc of the European Conference on Computer Vision. Berlin, Germany: Springer, 2018: 120-135.
[16] YAN Q S, GONG D, SHI Q F, et al. Attention-Guided Network for Ghost-Free High Dynamic Range Imaging // Proc of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2019: 1751-1760.
[17] DENG Y P, LIU Q, IKENAGA T. Selective Kernel and Motion-Emphasized Loss Based Attention-Guided Network for HDR Imaging of Dynamic Scenes // Proc of the 25th International Confe-rence on Pattern Recognition. Washington, USA: IEEE, 2021: 8976-8983.
[18] YAN Q S, ZHANG L, LIU Y, et al. Deep HDR Imaging via a Non-local Network. IEEE Transactions on Image Processing, 2020, 29: 4308-4322.
[19] PU Z Y, GUO P Y, ASIF M S, et al. Robust High Dynamic Range(HDR) Imaging with Complex Motion and Parallax // Proc of the Asian Conference on Computer Vision. Berlin, Germany: Sprin-ger, 2020: 134-149
[20] LIU Z, LIN W J, LI X P, et al. ADNet: Attention-Guided Deformable Convolutional Network for High Dynamic Range Imaging // Proc of the IEEE/CVF Conference on Computer Vision and Pa-ttern Recognition. Washington, USA: IEEE, 2021: 463-470.
[21] TAN X, CHEN H, XU K, et al. High Dynamic Range Imaging for Dynamic Scenes with Large-Scale Motions and Severe Saturation. IEEE Transactions on Instrumentation and Measurement, 2022, 715. DOI: 10.1109/TIM.2022.3144205.
[22] PRABHAKAR K R, AGRAWAL S, SINGH D K, et al. Towards Practical and Efficient High-Resolution HDR Deghosting with CNN // Proc of the 16th European Conference on Computer Vision. Berlin, Germany: Springer, 2020: 497-513.
[23] CHEN J, YANG Z F, CHAN T N, et al. Attention-Guided Progressive Neural Texture Fusion for High Dynamic Range Image Restoration. IEEE Transactions on Image Processing, 2022, 31: 2661-2672.
[24] CHUNG H, CHO N I. High Dynamic Range Imaging of Dynamic Scenes with Saturation Compensation but without Explicit Motion Compensation // Proc of the IEEE/CVF Winter Conference on App-lications of Computer Vision. Washington, USA: IEEE, 2022: 61-71.
[25] DOSOVITSKIY A, BEYER L, KOLESNIKOV A, et al. An Image Is Worth 16×16 Words: Transformers for Image Recognition at Scale[C/OL].[2024-01-09]. https://arxiv.org/pdf/2010.11929.pdf.
[26] 薛峰,洪自坤,李书杰,等.基于Vision Transformer的中文唇语识别.模式识别与人工智能, 2022, 35(12): 1111-1121.
(XUE F, HONG Z K, LI S J, et al. Chinese Lipreading Network Based on Vision Transformer. Pattern Recognition and Artificial Intelligence, 2022, 35(12): 1111-1121.)
[27] 牛玉贞,林晓锋,许煌标,等.基于Transformer的多尺度优化低照度图像增强网络.模式识别与人工智能, 2023, 36(6): 511-529.
(NIU Y Z, LIN X F, XU H B, et al. Transformer-Based Multi-scale Optimization Network for Low-Light Image Enhancement. Pattern Recognition and Artificial Intelligence, 2023, 36(6): 511-529.)
[28] 王科平,左鑫浩,杨艺,等.基于伪全局Swin Transformer的遥感图像识别算法.模式识别与人工智能, 2023, 36(9): 818-831.
(WANG K P, ZUO X H, YANG Y, et al. Remote Sensing Image Recognition Algorithm Based on Pseudo Global Swin Transformer. Pattern Recognition and Artificial Intelligence, 2023, 36(9): 818-831.)
[29] SONG J W, PARK Y I, KONG K, et al. Selective TransHDR: Transformer-Based Selective HDR Imaging Using Ghost Region Mask // Proc of the European Conference on Computer Vision. Berlin, Germany: Springer, 2022: 288-304.
[30] LIU Z, WANG Y L, ZENG B, et al. Ghost-Free High Dynamic Range Imaging with Context-Aware Transformer // Proc of the European Conference on Computer Vision. Berlin, Germany: Sprin-ger, 2022: 344-360.
[31] YAN Q S, CHEN W Y, ZHANG S, et al. A Unified HDR Imaging Method with Pixel and Patch Level // Proc of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2023: 22211-22220.
[32] YAN Q S, ZHANG S, CHEN W Y, et al. SMAE: Few-Shot Learning for HDR Deghosting with Saturation-Aware Masked Autoencoders // Proc of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2023: 5775-5784.
[33] CHEN R F, ZHENG B L, ZHANG H, et al. Improving Dynamic HDR Imaging with Fusion Transformer. Proceedings of the AAAI Conference on Artificial Intelligence, 2023, 37(1): 340-349.
[34] TANG L F, HUANG H, ZHANG Y F, et al. Structure-Embedded Ghosting Artifact Suppression Network for High Dynamic Range Ima-ge Reconstruction. Knowledge-Based Systems, 2023, 263. DOI: 10.1016/j.knosys.2023.110278.
[35] GUO Q, SUN J Y, JUEFEI-XU F, et al. EfficientDeRain: Lear-ning Pixel-Wise Dilation Filtering for High-Efficiency Single-Image Deraining. Proceedings of the AAAI Conference on Artificial Inte-lligence, 2021, 35(2): 1487-1495.
[36] GUO Q, LI X G, JUEFEI-XU F, et al. JPGNet: Joint Predictive Filtering and Generative Network for Image Inpainting // Proc of the 29th ACM International Conference on Multimedia. New York, USA: ACM, 2021: 386-394.
[37] LI X G, GUO Q, LIN D, et al. MISF: Multi-level Interactive Siamese Filtering for High-Fidelity Image Inpainting // Proc of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2022: 1859-1868.
[38] TURSUN O T, AKYUZ A O, ERDEM A, et al. An Objective Deghosting Quality Metric for HDR Images. Computer Graphics Forum, 2016, 35(2): 139-152.
[39] MANTIUK R, KIM K J, REMPEL A G, et al. HDR-VDP-2: A Calibrated Visual Metric for Visibility and Quality Predictions in all Luminance Conditions. ACM Transactions on Graphics, 2011, 30(4). DOI: 10.1145/2010324.1964935.
[40] GU K, WANG S Q, ZHAI G T, et al. Blind Quality Assessment of Tone-Mapped Images via Analysis of Information, Naturalness, and Structure. IEEE Transactions on Multimedia, 2016, 18(3): 432-443.
[41] FANG Y M, ZHU H W, MA K D, et al. Perceptual Evaluation for Multi-exposure Image Fusion of Dynamic Scenes. IEEE Transactions on Image Processing, 2019, 29: 1127-1138.
[42] KINGMA D P, BA J L. Adam: A Method for Stochastic Optimization[C/OL]. [2024-01-09]. https://arxiv.org/pdf/1412.6980.pdf.
[43] GOYAL P, DOLLÁR P, GIRSHICK R, et al. Accurate, Large Mini-batch SGD: Training ImageNet in 1 Hour[C/OL].[2024-01-09]. https://arxiv.org/pdf/1706.02677.pdf.
[44] NIU Y Z, WU J B, LIU W X, et al. HDR-GAN: HDR Image Reconstruction from Multi-exposed LDR Images with Large Motions. IEEE Transactions on Image Processing, 2021, 30: 3885-3896.
[45] YUAN Y, WU J Q, JING Z L, et al. Ghost-Free High Dynamic Range Imaging via Hybrid CNN-Transformer and Structure Tensor[C/OL].[2024-01-09]. https://arxiv.org/pdf/2212.00595.pdf.
[46] TEL S, WU Z W, ZHANG Y L, et al. Alignment-Free HDR Deghosting with Semantics Consistent Transformer // Proc of the IEEE/CVF International Conference on Computer Vision. Wa-shington, USA: IEEE, 2023: 12790-12799.