Abstract Video super-resolution reconstruction methods based on deep learning are often faced with the problems of long time consumption or low accuracy. A video super-resolution reconstruction method based on deep residual network is proposed. It reconstructs videos with high accuracy quickly and meets the real-time requirements for low-resolution videos. Firstly, the adaptive key frame discrimination subnet is utilized to adaptively identify key frames from the video. Then, the reconstruction results of the key frames are obtained by the high precision reconstruction subnet. For non-key frames, the reconstruction results are directly gained based on the features obtained by fusing the features of the corresponding key frame and the motion estimation features between the non-key frame and the adjacent key frame. Experiments on open datasets show that videos are fast reconstructed by the proposed method with high accuracy and robustness.
Fund:Supported by Natural Science Foundation of Beijing(No.4173072)
Corresponding Authors:
FU Lihua, Ph.D., associate professor. Her research interests include intelligent information processing, image processing and computer vision.
About author:: SUN Xiaowei, master student. His research interests include computer image processing and super resolution.ZHAO Yu, master student. His research interests include computer image processing and video target segmentation.LI Zonggang, master student. His research interests include computer image processing and super resolution.HUANG Jialiang, master student. His research interests include computer image processing and super resolution.WANG Luyuan, master student. His research interests include computer image processing and video target tracking.
[1] LI X, ORCHARD M T. New Edge Directed Interpolation. IEEE Transactions on Image Processing, 2001, 10(10): 1521-1527. [2] ZHANG L, WU X L. An Edge-Guided Image Interpolation Algorithm via Directional Filtering and Data Fusion. IEEE Transactions on Image Processing, 2006, 15(8): 2226-2238. [3] SUN J, XU Z B, SHUM R Y. Image Super-Resolution Using Gra-dient Profile Prior // Proc of the IEEE Conference on Computer Vision and Pattern Recognition. Washington, USA:IEEE, 2008. DOI: 10.1109/CVPR.2008.4587659. [4] TAI Y W, LIU S C, BROWN M S, et al. Super Resolution Using Edge Prior and Single Image Detail Synthesis // Proc of the IEEE International Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2010: 2400-2407. [5] DONG C, LOY C C, HE K M, et al. Learning a Deep Convolutional Network for Image Super-Resolution // Proc of the European Conference on Computer Vision. Berlin, Germany: Springer, 2014: 184-199. [6] DONG C, LOY C C, TANG X O. Accelerating the Super-Resolution Convolutional Neural Network // Proc of the European Confe-rence on Computer Vision. Berlin, Germany: Springer, 2016: 391-407. [7] TONG T, LI G, LIU X J, et al. Image Super-Resolution Using Dense Skip Connections // Proc of the IEEE International Confe-rence on Computer Vision. Washington, USA: IEEE, 2017: 4809-4817. [8] LEDIG C, THEIS C, HUSZAR F, et al. Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network // Proc of the IEEE Conference on Computer Vision and Pattern Re-cognition. Washington, USA: IEEE, 2017: 105-114. [9] LIM B, SON S, KIM H, et al. Enhanced Deep Residual Networks for Single Image Super-Resolution // Proc of the IEEE Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2017: 1132-1140. [10] YU J H, FAN Y C, YANG J C, et al. Wide Activation for Efficient and Accurate Image Super-Resolution // Proc of the IEEE Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2018: 2621-2624. [11] WANG Z, BOVIK A C, SHEIKH H R, et al. Image Quality Assessment: From Error Visibility to Structural Similarity. IEEE Transactions on Image Processing, 2004, 13(4): 600-612. [12] CABALLERO J, LEDIG C, AITKEN A, et al. Real-Time Video Super-Resolution with Spatio-Temporal Networks and Motion Compensation // Proc of the IEEE Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2017:2848-2857. [13] KAPPELER A, YOO S, DAI Q Q, et al. Video Super-Resolution with Convolutional Neural Networks. IEEE Transactions on Computational Imaging, 2016, 2(2): 109-122. [14] HUANG Y, WANG W, WANG L. Video Super-Resolution via Bidirectional Recurrent Convolutional Networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2018, 40(4): 1015-1028. [15] ZHENG H T, JI M Q, WANG H Q, et al. CrossNet: An End-to-End Reference-Based Super Resolution Network Using Cross-Scale Warping // Proc of the European Conference on Computer Vision. Berlin, Germany: Springer, 2018: 88-104. [16] ZHU X Z, XIONG Y W, DAI J F, et al. Deep Feature Flow for Video Recognition // Proc of the IEEE Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2017:2349-2358. [17] LI Y L, SHI J P, LIN D H. Low-Latency Video Semantic Segmentation // Proc of the IEEE Conference on Computer Vision and Pa-ttern Recognition. Washington, USA: IEEE, 2018: 5997-6005. [18] SHI W Z, CABALLERO J, HUSZÁR F, et al. Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-pixel Convolutional Neural Network // Proc of the IEEE Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2016: 1874-1883. [19] PERAZZI F, PONT-TUSET J, MCWILLIAMS B, et al. A Benchmark Dataset and Evaluation Methodology for Video Object Segmentation // Proc of the IEEE Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2016: 724-732. [20] KINGMA D P, BA J L. ADAM: A Method for Stochastic Optimization[C/OL]. [2019-03-15]. https://arxiv.org/pdf/1412.6980.pdf. [21] AGUSTSSON E, TIMOFTE R. NTIRE 2017 Challenge on Single Image Super-Resolution: Dataset and Study // Proc of the IEEE Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2017: 1122-1131. [22] KIM J, LEE J K, LEE K M. Accurate Image Super-Resolution Using Very Deep Convolutional Networks // Proc of the IEEE Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2016: 1646-1654.
2019, Vol. 32 
