Image super-resolution via multi-path recursive convolutional network

Shen Mingyu; Yu Pengfei; Wang Ronggui; Yang Juan; Xue Lixia

doi:10.12086/oee.2019.180489

Article navigation > Opto-Electronic Engineering > 2019 Vol. 46 > No. 11 > 180489

Next Article Previous Article

Shen Mingyu, Yu Pengfei, Wang Ronggui, et al. Image super-resolution via multi-path recursive convolutional network[J]. Opto-Electronic Engineering, 2019, 46(11): 180489. doi: 10.12086/oee.2019.180489

Citation:

Shen Mingyu, Yu Pengfei, Wang Ronggui, et al. Image super-resolution via multi-path recursive convolutional network[J]. Opto-Electronic Engineering, 2019, 46(11): 180489. doi: 10.12086/oee.2019.180489

Image super-resolution via multi-path recursive convolutional network

School of Computer and Information, Hefei University of Technology, Hefei, Anhui 230009, China

Fund Project: Supported by National Natural Science Foundation of China (61672202)

More Information

Corresponding author: Yang Juan, E-mail: yangjuan6985@163.com

Received Date 17 September 2018

Revised Date 28 December 2018

Published Date 01 November 2019

Abstract

Abstract

Convolutional neural network (CNN) has recently achieved a great success for single image super-resolution (SISR). However, most deep CNN-based super-resolution models use chained stacking to build the network, which results in the fact that the relationship between layers is weak and does not make full use of hierarchical features. In this paper, a multi-path recursive convolutional network (MRCN) is designed to address these problems in SISR. By using multi-path structure to strengthen the relationship between layers, our network can effectively utilize features and extract rich high-frequency components. At the same time, we also use recursive structure to alleviate training difficulty. In addition, by introducing the operation of feature fusion into the model, our network can make full use of the features extracted from each layer in the reconstruction process and select the effective features adaptively. Extensive experiments on benchmarks datasets have shown that MRCN has a significant performance improvement against existing methods.
- convolutional neural network /
- super-resolution /
- hierarchical features /
- multi-path /
- feature fusion

FullText(HTML)

References

[1]	Li X, Orchard M T. New edge-directed interpolation[J]. IEEE Transactions on Image Processing, 2001, 10(10): 1521-1527. doi: 10.1109/83.951537 CrossRef Google Scholar
[2]	Zhang L, Wu X L. An edge-guided image interpolation algorithm via directional filtering and data fusion[J]. IEEE Transactions on Image Processing, 2006, 15(8): 2226-2238. doi: 10.1109/TIP.2006.877407 CrossRef Google Scholar
[3]	Dai S Y, Han M, Xu W, et al. SoftCuts: a soft edge smoothness prior for color image super-resolution[J]. IEEE Transactions on Image Processing, 2009, 18(5): 969-981. doi: 10.1109/TIP.2009.2012908 CrossRef Google Scholar
[4]	Sun J, Xu Z B, Shum H Y. Image super-resolution using gradient profile prior[C]//Proceedings of 2008 IEEE Conference on Computer Vision and Pattern Recognition, 2008: 1-8. Google Scholar
[5]	吴从中, 胡长胜, 张明君, 等.有监督多类字典学习的单幅图像超分辨率重建[J].光电工程, 2016, 43(11): 69-75. doi: 10.3969/j.issn.1003-501X.2016.11.011 CrossRef Google Scholar Wu C Z, Hu C S, Zhang M J, et al. Single image super-resolution reconstruction via supervised multi-dictionary learning[J]. Opto-Electronic Engineering, 2016, 43(11): 69-75. doi: 10.3969/j.issn.1003-501X.2016.11.011 CrossRef Google Scholar
[6]	汪荣贵, 汪庆辉, 杨娟, 等.融合特征分类和独立字典训练的超分辨率重建[J].光电工程, 2018, 45(1): 170542. doi: 10.12086/oee.2018.170542 CrossRef Google Scholar Wang R G, Wang Q H, Yang J, et al. Image super-resolution reconstruction by fusing feature classification and independent dictionary training[J]. Opto-Electronic Engineering, 2018, 45(1): 170542. doi: 10.12086/oee.2018.170542 CrossRef Google Scholar
[7]	Timofte R, De Smet V, Van Gool L. Anchored neighborhood regression for fast example-based super-resolution[C]// Proceedings of 2013 IEEE International Conference on Computer Vision, 2013: 1920-1927. Google Scholar
[8]	Timofte R, De Smet V, Van Gool L. A+: adjusted anchored neighborhood regression for fast super-resolution[M]//Cremers D, Reid I, Saito H, et al. Computer Vision--ACCV 2014. Cham: Springer, 2014: 111-126. Google Scholar
[9]	Dong C, Loy C C, He K M, et al. Learning a deep convolutional network for image super-resolution[C]//Proceedings of the 13th European Conference on Computer Vision, 2014: 184-199. Google Scholar
[10]	Dong C, Loy C C, He K M, et al. Image super-resolution using deep convolutional networks[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2016, 38(2): 295-307. doi: 10.1109/TPAMI.2015.2439281 CrossRef Google Scholar
[11]	Yang J C, Wright J, Huang T S, et al. Image super-resolution via sparse representation[J]. IEEE Transactions on Image Processing, 2010, 19(11): 2861-2873. doi: 10.1109/TIP.2010.2050625 CrossRef Google Scholar
[12]	Kim J, Lee J K, Lee K M. Accurate image super-resolution using very deep convolutional networks[C]//Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition, 2016: 1646-1654. Google Scholar
[13]	Kim J, Lee J K, Lee K M. Deeply-recursive convolutional network for image super-resolution[C]//Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition, 2016: 1637-1645. Google Scholar
[14]	Zhang K, Zuo W M, Gu S H, et al. Learning deep CNN Denoiser prior for image restoration[C]//Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition, 2017: 2808-2817. Google Scholar
[15]	Shi W Z, Jiang F, Zhao D B. Single image super-resolution with dilated convolution based multi-scale information learning inception module[C]//Proceedings of 2017 IEEE International Conference on Image Processing, 2017: 977-981. Google Scholar
[16]	Ledig C, Theis L, Huszár F, et al. Photo-realistic single image super-resolution using a generative adversarial network[C]//Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition, 2017: 105-114. Google Scholar
[17]	Tai Y, Yang J, Liu X M. Image super-resolution via deep recursive residual network[C]//Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition, 2017: 2790-2798. Google Scholar
[18]	Mao X J, Shen C H, Yang Y B. Image restoration using very deep convolutional encoder-decoder networks with symmetric skip connections[C]//Proceedings of the 30th International Conference on Neural Information Processing Systems, 2016: 2810-2818. Google Scholar
[19]	Tai Y, Yang J, Liu X M, et al. MemNet: a persistent memory network for image restoration[C]//Proceedings of 2017 IEEE International Conference on Computer Vision, 2017: 4549-4557. Google Scholar
[20]	Dong C, Loy C C, Tang X O. Accelerating the super-resolution convolutional neural network[C]//Proceedings of the 14th European Conference on Computer Vision, 2016: 391-407. Google Scholar
[21]	Shi W Z, Caballero J, Huszár F, et al. Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network[C]//Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition, 2016: 1874-1883. Google Scholar
[22]	Lai W S, Huang J B, Ahuja N, et al. Deep Laplacian pyramid networks for fast and accurate super-resolution[C]// Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition, 2017: 5835-5843. Google Scholar
[23]	Huang G, Liu Z, van der Maaten L, et al. Densely connected convolutional networks[C]//Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition, 2017: 2261-2269. Google Scholar
[24]	Chen Y P, Li J A, Xiao H X, et al. Dual path networks[C]//Proceedings of the 31st International Conference on Neural Information Processing Systems, 2017: 4470-4478. Google Scholar
[25]	Hu J, Shen L, Albanie S, et al. Squeeze-and-excitation networks[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017, doi: 10.1109/TPAMI.2019.2913372. CrossRef Google Scholar
[26]	Timofte R, Agustsson E, van Gool L, et al. NTIRE 2017 challenge on single image super-resolution: methods and results[C]//Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops, 2017: 1110-1121. Google Scholar
[27]	Bevilacqua M, Roumy A, Guillemot C, et al. Low-complexity single-image super-resolution based on nonnegative neighbor embedding[C]//Proceedings of the British Machine Vision Conference, 2012: 135.1-135.10. Google Scholar
[28]	Zeyde R, Elad M, Protter M. On single image scale-up using sparse-representations[C]//Proceedings of the 7th International Conference on Curves and Surfaces, 2010: 711-730. Google Scholar
[29]	Huang J B, Singh A, Ahuja N. Single image super-resolution from transformed self-exemplars[C]//Proceedings of 2015 IEEE Conference on Computer Vision and Pattern Recognition, 2015: 5197-5206. Google Scholar
[30]	Martin D, Fowlkes C, Tal D, et al. A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics[C]//Proceedings of the 8th IEEE International Conference on Computer Vision, 2001: 416-423. Google Scholar
[31]	He K M, Zhang X Y, Ren S Q, et al. Delving deep into rectifiers: surpassing human-level performance on ImageNet classification[C]//Proceedings of 2015 IEEE International Conference on Computer Vision, 2015: 1026-1034. Google Scholar
[32]	Zhao H, Gallo O, Frosio I, et al. Loss functions for neural networks for image processing[OL]. arXiv: 1511.08861[cs.CV], 2015. Google Scholar
[33]	Bruhn A, Weickert J, Schnörr C. Lucas/Kanade meets Horn/Schunck: combining local and global optic flow methods[J]. International Journal of Computer Vision, 2005, 61(3): 211-231. doi: 10.1023/b:visi.0000045324.43199.43 CrossRef Google Scholar

Overview

Overview

Overview: Single image super-resolution is widely used in security monitoring, satellite remote sensing imagery, and medical image processing. It aims at restoring a high-resolution image from corresponding degraded low resolution LR-image. Recently, Dong et al. first discovered that convolutional neural networks can accomplish super-resolution by end-to-end manner, opening the door for deep learning in the field of super-resolution. And a series of new network model were proposed. Although these models have achieved good performance, the existing problems cannot be ignored. First, with the increase of network depth, many models fail to take into account the effect of hierarchical features on super-resolution, and the extracted features of each layer can only be learned once, which cannot be fully utilized. Second, the many models use pre-processing methods to get the target size, which not only increases the computational complexity, but also destroys the information carried by the original image. In response to this problem, ESPCN based on subpixel convolution and FSRCNN based on deconvolution are proposed. However, their structure is too simple to complete the exact mapping. Third, most methods use the mean square error (MSE) to optimize the model, resulting in overly smooth images.

To solve these problems, we propose a multipath recursive network (MRCN). We use multi-path structure to extract features and improve the ability of non-linear mapping, which accelerates the transfer of feature and gradient in the network. Then we use recursive methods to reduce network parameters. Finally, all the features were merged to complete super-resolution. Compared with other models, our network mainly has the following differences. First, different from the traditional single-chain structure, our network adopts a multi-path structure, which enables the extracted features of each layer to be learned multiple times, improving feature richness, and the reconstructed image contains more high-frequency information. Second, most models use the last layer of the network to complete reconstruction, while our network uses all the features extracted from the network to complete reconstruction together. At the same time, we use the nature of SENet to select the effective features of these features adaptively and suppress the useless features. Third, we use the Charbonnier loss function to alleviate the problem that the reconstructed images are too smooth due to MSE, and the performance of the network can be improved. A large number of experiments on the benchmark set show that our method is superior to the existing methods in reconstruction performance.