面向高动态范围成像的内容恢复和鬼影抑制网络

doi:10.16451/j.cnki.issn1003-6059.202404003

摘要
图/表
参考文献(46)
相关文章 (15)

全文: PDF (4208 KB) HTML (1 KB)
输出: BibTeX | EndNote (RIS)

摘要

基于多曝光融合的高动态范围(High Dynamic Range, HDR)成像旨在整合多幅低动态范围图像(Low Dynamic Range, LDR)的信息,以生成高质量的HDR图像.然而,抑制运动区域的鬼影信息和恢复过饱和区域的缺失信息仍是HDR成像面临的两大挑战.为了综合考虑对参考图像缺失内容的恢复和运动区域鬼影的抑制,文中提出面向HDR成像的内容恢复和鬼影抑制网络.在内容恢复方面,引入基于预测滤波的内容恢复块.由内容恢复块预测到的滤波核对参考图像特征进行滤波,整合参考图像和非参考图像中的关键信息,为有效进行缺失内容的重建提供更丰富的信息.为了抑制运动区域的鬼影信息并充分利用非参考图像中的互补信息,引入可变形卷积,将非参考图像特征与参考图像特征对齐.此外,为了提升网络的HDR图像重建能力,构建三支路图像重建模块,包括一条主支路和两条辅助支路,辅助支路协助主支路在HDR结果的生成过程中更好地保留细节.实验表明,文中网络在主观视觉和客观指标上均具有较优表现.

	服务

	把本文推荐给朋友
	加入我的书架
	加入引用管理器
	E-mail Alert
	RSS
	作者相关文章
	杨珍妹
	李华锋
	张亚飞

关键词 ：高动态范围成像, 鬼影抑制, 预测滤波, 内容恢复

Abstract：

High dynamic range(HDR) imaging based on multi-exposure fusion aims to generate high-quality HDR images by integrating the information from multiple low dynamic range(LDR) images. However, HDR imaging is faced with two major challenges, ghosting artifact suppression in motion regions and lost information restoration in over-saturated areas. To comprehensively address the challenges of restoring missing content from reference images and suppressing ghosting artifacts in motion regions, a missing content restoration and ghosting suppression network for high dynamic range imaging is proposed in this paper. In terms of content restoration, a predictive filtering-based content restoration block is introduced. The filtering kernel predicted by the content restoration block is employed to filter reference image features, integrating key information from both reference images and non-reference images to provide richer information for effective reconstruction of missing content. To suppress ghosting artifacts in motion regions and fully exploit complementary information from non-reference images, deformable convolutions are introduced to align features from non-reference images with those from the reference image. Additionally, to enhance the HDR image reconstruction capability of the network, a three-branch image reconstruction module is constructed, including a main branch and two auxiliary branches. The auxiliary branches assist the main branch with better preserved details during the generation of HDR results. Experimental results demonstrate superior performance of the proposed network.

Key words： High Dynamic Range Imaging Ghosting Suppression Predictive Filtering Content Restoration

收稿日期: 2023-12-11

ZTFLH:

TP 391.41

基金资助:

国家自然科学基金项目(No.62161015)、云南省基础研究项目(No.202301AV070004)资助

通讯作者: 李华锋, 博士, 教授,主要研究方向为图像处理、计算机视觉.E-mail:hfchina99@163.com.

作者简介: 杨珍妹,硕士研究生,主要研究方向为模式识别、图像处理.E-mail:yangtongxue123_km@163.com. 张亚飞,博士,副教授,主要研究方向为图像处理、模式识别.E-mail:zyfeimail@163.com.

引用本文:

杨珍妹, 李华锋, 张亚飞. 面向高动态范围成像的内容恢复和鬼影抑制网络[J]. 模式识别与人工智能, 2024, 37(4): 313-327. YANG Zhenmei, LI Huafeng, ZHANG Yafei. Missing Content Restoration and Ghosting Suppression Network for High Dynamic Range Imaging. Pattern Recognition and Artificial Intelligence, 2024, 37(4): 313-327.

链接本文:

http://manu46.magtech.com.cn/Jweb_prai/CN/10.16451/j.cnki.issn1003-6059.202404003 或 http://manu46.magtech.com.cn/Jweb_prai/CN/Y2024/V37/I4/313

[1] MA K D, ZENG K, WANG Z. Perceptual Quality Assessment for Multi-exposure Image Fusion. IEEE Transactions on Image Proce-ssing, 2015, 24(11): 3345-3356.
[2] PRABHAKAR K R, SRIKAR V S, BABU R V. Deepfuse: A Deep Unsupervised Approach for Exposure Fusion with Extreme Exposure Image Pairs // Proc of the IEEE International Conference on Computer Vision. Washington, USA: IEEE, 2017: 4724-4732.
[3] DENG X, DRAGOTTI P L. Deep Convolutional Neural Network for Multi-modal Image Restoration and Fusion. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020, 43(10): 3333-3348.
[4] DENG X, ZHANG Y T, XU M, et al. Deep Coupled Feedback Network for Joint Exposure Fusion and Image Super-Resolution. IEEE Transactions on Image Processing, 2021, 30: 3098-3112.
[5] WARD G. Fast, Robust Image Registration for Compositing High Dynamic Range Photographs from Hand-Held Exposures. Journal of Graphics Tools, 2003, 8(2): 17-30.
[6] ZIMMER H, BRUHN A, WEICKERT J. Freehand HDR Imaging of Moving Scenes with Simultaneous Resolution Enhancement. Computer Graphics Forum, 2011, 30(2): 405-414.
[7] SEN P, KALANTARI N K, YAESOUBI M, et al. Robust Patch-Based HDR Reconstruction of Dynamic Scenes. ACM Transactions on Graphics, 2012, 31(6). DOI: 10.1145/2366145.2366222.
[8] HU J, GALLO O, PULLI K, et al. HDR Deghosting: How to Deal with Saturation? // Proc of the IEEE Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2013: 1163-1170.
[9] RAMAN S, CHAUDHURI S. Reconstruction of High Contrast Images for Dynamic Scenes. The Visual Computer, 2011, 27: 1099-1114.
[10] YAN Q S, SUN J Q, LI H S, et al. High Dynamic Range Imaging by Sparse Representation. Neurocomputing, 2017, 269: 160-169.
[11] MA K D, LI H, YONG H W, et al. Robust Multi-exposure Image Fusion: A Structural Patch Decomposition Approach. IEEE Transactions on Image Processing, 2017, 26(5): 2519-2532.
[12] LI H, MA K D, YONG H W, et al. Fast Multi-scale Structural Patch Decomposition for Multi-exposure Image Fusion. IEEE Transac-tions on Image Processing, 2020, 29: 5805-5816.
[13] BAKER S, SCHARSTEIN D, LEWIS J P, et al. A Database and Evaluation Methodology for Optical Flow. International Journal of Computer Vision, 2011, 92. DOI: 10.1007/s11263-010-0390-2.
[14] KALANTARI N K, RAMAMOORTHI R. Deep High Dynamic Range Imaging of Dynamic Scenes. ACM Transactions on Graphics, 2017, 36(4). DOI: 10.1145/3072959.3073609.
[15] WU S Z, XU J R, TAI Y W, et al. Deep High Dynamic Range Imaging with Large Foreground Motions // Proc of the European Conference on Computer Vision. Berlin, Germany: Springer, 2018: 120-135.
[16] YAN Q S, GONG D, SHI Q F, et al. Attention-Guided Network for Ghost-Free High Dynamic Range Imaging // Proc of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2019: 1751-1760.
[17] DENG Y P, LIU Q, IKENAGA T. Selective Kernel and Motion-Emphasized Loss Based Attention-Guided Network for HDR Imaging of Dynamic Scenes // Proc of the 25th International Confe-rence on Pattern Recognition. Washington, USA: IEEE, 2021: 8976-8983.
[18] YAN Q S, ZHANG L, LIU Y, et al. Deep HDR Imaging via a Non-local Network. IEEE Transactions on Image Processing, 2020, 29: 4308-4322.
[19] PU Z Y, GUO P Y, ASIF M S, et al. Robust High Dynamic Range(HDR) Imaging with Complex Motion and Parallax // Proc of the Asian Conference on Computer Vision. Berlin, Germany: Sprin-ger, 2020: 134-149
[20] LIU Z, LIN W J, LI X P, et al. ADNet: Attention-Guided Deformable Convolutional Network for High Dynamic Range Imaging // Proc of the IEEE/CVF Conference on Computer Vision and Pa-ttern Recognition. Washington, USA: IEEE, 2021: 463-470.
[21] TAN X, CHEN H, XU K, et al. High Dynamic Range Imaging for Dynamic Scenes with Large-Scale Motions and Severe Saturation. IEEE Transactions on Instrumentation and Measurement, 2022, 715. DOI: 10.1109/TIM.2022.3144205.
[22] PRABHAKAR K R, AGRAWAL S, SINGH D K, et al. Towards Practical and Efficient High-Resolution HDR Deghosting with CNN // Proc of the 16th European Conference on Computer Vision. Berlin, Germany: Springer, 2020: 497-513.
[23] CHEN J, YANG Z F, CHAN T N, et al. Attention-Guided Progressive Neural Texture Fusion for High Dynamic Range Image Restoration. IEEE Transactions on Image Processing, 2022, 31: 2661-2672.
[24] CHUNG H, CHO N I. High Dynamic Range Imaging of Dynamic Scenes with Saturation Compensation but without Explicit Motion Compensation // Proc of the IEEE/CVF Winter Conference on App-lications of Computer Vision. Washington, USA: IEEE, 2022: 61-71.
[25] DOSOVITSKIY A, BEYER L, KOLESNIKOV A, et al. An Image Is Worth 16×16 Words: Transformers for Image Recognition at Scale[C/OL].[2024-01-09]. https://arxiv.org/pdf/2010.11929.pdf.
[26] 薛峰,洪自坤,李书杰,等.基于Vision Transformer的中文唇语识别.模式识别与人工智能, 2022, 35(12): 1111-1121.
(XUE F, HONG Z K, LI S J, et al. Chinese Lipreading Network Based on Vision Transformer. Pattern Recognition and Artificial Intelligence, 2022, 35(12): 1111-1121.)
[27] 牛玉贞,林晓锋,许煌标,等.基于Transformer的多尺度优化低照度图像增强网络.模式识别与人工智能, 2023, 36(6): 511-529.
(NIU Y Z, LIN X F, XU H B, et al. Transformer-Based Multi-scale Optimization Network for Low-Light Image Enhancement. Pattern Recognition and Artificial Intelligence, 2023, 36(6): 511-529.)
[28] 王科平,左鑫浩,杨艺,等.基于伪全局Swin Transformer的遥感图像识别算法.模式识别与人工智能, 2023, 36(9): 818-831.
(WANG K P, ZUO X H, YANG Y, et al. Remote Sensing Image Recognition Algorithm Based on Pseudo Global Swin Transformer. Pattern Recognition and Artificial Intelligence, 2023, 36(9): 818-831.)
[29] SONG J W, PARK Y I, KONG K, et al. Selective TransHDR: Transformer-Based Selective HDR Imaging Using Ghost Region Mask // Proc of the European Conference on Computer Vision. Berlin, Germany: Springer, 2022: 288-304.
[30] LIU Z, WANG Y L, ZENG B, et al. Ghost-Free High Dynamic Range Imaging with Context-Aware Transformer // Proc of the European Conference on Computer Vision. Berlin, Germany: Sprin-ger, 2022: 344-360.
[31] YAN Q S, CHEN W Y, ZHANG S, et al. A Unified HDR Imaging Method with Pixel and Patch Level // Proc of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2023: 22211-22220.
[32] YAN Q S, ZHANG S, CHEN W Y, et al. SMAE: Few-Shot Learning for HDR Deghosting with Saturation-Aware Masked Autoencoders // Proc of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2023: 5775-5784.
[33] CHEN R F, ZHENG B L, ZHANG H, et al. Improving Dynamic HDR Imaging with Fusion Transformer. Proceedings of the AAAI Conference on Artificial Intelligence, 2023, 37(1): 340-349.
[34] TANG L F, HUANG H, ZHANG Y F, et al. Structure-Embedded Ghosting Artifact Suppression Network for High Dynamic Range Ima-ge Reconstruction. Knowledge-Based Systems, 2023, 263. DOI: 10.1016/j.knosys.2023.110278.
[35] GUO Q, SUN J Y, JUEFEI-XU F, et al. EfficientDeRain: Lear-ning Pixel-Wise Dilation Filtering for High-Efficiency Single-Image Deraining. Proceedings of the AAAI Conference on Artificial Inte-lligence, 2021, 35(2): 1487-1495.
[36] GUO Q, LI X G, JUEFEI-XU F, et al. JPGNet: Joint Predictive Filtering and Generative Network for Image Inpainting // Proc of the 29th ACM International Conference on Multimedia. New York, USA: ACM, 2021: 386-394.
[37] LI X G, GUO Q, LIN D, et al. MISF: Multi-level Interactive Siamese Filtering for High-Fidelity Image Inpainting // Proc of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2022: 1859-1868.
[38] TURSUN O T, AKYUZ A O, ERDEM A, et al. An Objective Deghosting Quality Metric for HDR Images. Computer Graphics Forum, 2016, 35(2): 139-152.
[39] MANTIUK R, KIM K J, REMPEL A G, et al. HDR-VDP-2: A Calibrated Visual Metric for Visibility and Quality Predictions in all Luminance Conditions. ACM Transactions on Graphics, 2011, 30(4). DOI: 10.1145/2010324.1964935.
[40] GU K, WANG S Q, ZHAI G T, et al. Blind Quality Assessment of Tone-Mapped Images via Analysis of Information, Naturalness, and Structure. IEEE Transactions on Multimedia, 2016, 18(3): 432-443.
[41] FANG Y M, ZHU H W, MA K D, et al. Perceptual Evaluation for Multi-exposure Image Fusion of Dynamic Scenes. IEEE Transactions on Image Processing, 2019, 29: 1127-1138.
[42] KINGMA D P, BA J L. Adam: A Method for Stochastic Optimization[C/OL]. [2024-01-09]. https://arxiv.org/pdf/1412.6980.pdf.
[43] GOYAL P, DOLLÁR P, GIRSHICK R, et al. Accurate, Large Mini-batch SGD: Training ImageNet in 1 Hour[C/OL].[2024-01-09]. https://arxiv.org/pdf/1706.02677.pdf.
[44] NIU Y Z, WU J B, LIU W X, et al. HDR-GAN: HDR Image Reconstruction from Multi-exposed LDR Images with Large Motions. IEEE Transactions on Image Processing, 2021, 30: 3885-3896.
[45] YUAN Y, WU J Q, JING Z L, et al. Ghost-Free High Dynamic Range Imaging via Hybrid CNN-Transformer and Structure Tensor[C/OL].[2024-01-09]. https://arxiv.org/pdf/2212.00595.pdf.
[46] TEL S, WU Z W, ZHANG Y L, et al. Alignment-Free HDR Deghosting with Semantics Consistent Transformer // Proc of the IEEE/CVF International Conference on Computer Vision. Wa-shington, USA: IEEE, 2023: 12790-12799.