变分贝叶斯框架中的多帧图像盲超分辨

闵雷, 杨平, 许冰, 等. 变分贝叶斯框架中的多帧图像盲超分辨[J]. 光电工程, 2019, 46(6): 180149. doi: 10.12086/oee.2019.180149
引用本文: 闵雷, 杨平, 许冰, 等. 变分贝叶斯框架中的多帧图像盲超分辨[J]. 光电工程, 2019, 46(6): 180149. doi: 10.12086/oee.2019.180149
Min Lei, Yang Ping, Xu Bing, et al. Multi-image blind super-resolution in variational Bayesian framework[J]. Opto-Electronic Engineering, 2019, 46(6): 180149. doi: 10.12086/oee.2019.180149
Citation: Min Lei, Yang Ping, Xu Bing, et al. Multi-image blind super-resolution in variational Bayesian framework[J]. Opto-Electronic Engineering, 2019, 46(6): 180149. doi: 10.12086/oee.2019.180149

变分贝叶斯框架中的多帧图像盲超分辨

  • 基金项目:
    中国科学院创新基金项目(CXJJ-16M208);四川省杰出青年基金项目(2012JQ0012);中国科学院卓越科学家项目
详细信息
    作者简介:
    通讯作者: 杨平(1980-),男,博士,研究员,主要从事自适应光学,光场信号获取与处理,激光光束净化等的研究。E-mail:pingyang2516@163.com
  • 中图分类号: TN911.73

Multi-image blind super-resolution in variational Bayesian framework

  • Fund Project: Supported by the National Innovation Fund of Chinese Academy of Sciences (CXJJ-16M208), the Preeminent Youth Fund of Sichuan Province, China (2012JQ0012), and the Outstanding Youth Science Fund of Chinese Academy of Sciences
More Information
  • 多帧图像超分辨理论是通过融合多帧低分辨率图像的信息来重构高空间分辨率图像。准确估计低分辨率图像的模糊核是进行有效信息融合的先决条件。传统超分辨方法通常假设模糊核已知且采用固定的高斯滤波模糊核,并且模型参数需要费时的手动调整。本文在变分贝叶斯框架下获得相应的超分辨算法,该算法对高分辨率图像、模糊核和模型参数同时进行最优估计。对比实验表明,模糊核自适应估计的盲超分辨方法总体性能优于现有的变分贝叶斯框架下的图像超分辨方法,特别是在高信噪比场景,推荐方法优势更加明显。

  • Overview: Images with high spatial resolution are usually desirable in real applications. The most direct approaches to increase the spatial resolution are: 1) increasing the bandwidth of the optics; 2) increasing the sampling frequency of the image sensor. However, these hardware-based approaches usually increase the volume or cost of the imaging system. Different from the hardware-based approaches, we use digital signal processing algorithm to increase the spatial resolution by a set of low-resolution (LR) images of the same scene. This approach is termed as multi-frame image super-resolution (SR). Variational Bayesian framework has been used to derive the SR algorithms for its model flexibility and parameter adaptivity. For multi-frame image SR, the accurate estimation of blur kernel of LR image is prerequisite for high-efficiency SR reconstruction. However, in variational Bayesian SR framework, all the methods assume a known and fixed blur kernel for LR images. We propose a blind SR method containing blur kernel self-adaptive estimation. First, the desired high-resolution (HR) image and the blur kernel are modeled in the imaging degradation model. Next, the total variation model is used to model the HR image and the blur kernel, and the Gamma distribution is used to model the corresponding parameters. Finally, the variational Bayesian inference based on Kullback-Leibler divergence and majorization-minimization approach is utilized to derive the SR algorithm. For the proposed method, the HR image, the blur kernel and the model parameters are estimated simultaneously and automatically. Experiments demonstrate that the proposed method outperforms the state-of-art methods. For the experiments on simulated data, the performance of the resolution enhancement method is quantitatively measured by the peak signal-to-noise ratio (PSNR) and structural similarity measure (SSIM). For typical ground truth HR image and blur kernel setup, the proposed method has the highest PSNR and SSIM and improves the PSNR by at least 1 dB~5 dB. For the visual effect, the proposed method has better blur removing performance. For the real data experiments using resolution chart LR images, the proposed method has better performance in preserving image details, suppressing noise and removing artifacts. The comparison experiments demonstrate the advantage of the proposed method. Especially, for the high signal-to-noise ratio (SNR) scenarios, the accuracy of blur kernel estimation dominates the performance and the proposed method can improve the performance dramatically. By the visual effect, the proposed method has better trade-off in preserving image details and removing noise and artifacts.

  • 加载中
  • 图 1  仿真实验数据真实高分辨率图像。(a) Lena;(b) EIA

    Figure 1.  The ground truth high-resolution image for simulated experiment data. (a) Lena; (b) EIA

    图 2  EIA图像35 dB高斯噪声,模糊核设置1,增强结果。(a)第一幅低分辨率图像;(b) BBC;(c) SRCNN;(d) SAR;(e) TV;(f) NS;(g)本文方法

    Figure 2.  The enhancement result on EIA for 35 dB and blur kernel case 1. (a) First LR image; (b) BBC; (c) SRCNN; (d) SAR; (e) TV; (f) NS; (g) Proposed

    图 3  Lena图像35 dB高斯噪声,模糊核设置2,增强结果。(a)第一幅低分辨率图像;(b) BBC;(c) SRCNN;(d) SAR;(e) TV;(f) NS;(g)本文方法

    Figure 3.  The enhancement result on Lena for 35 dB and blur kernel case 2. (a) First LR image; (b) BBC; (c) SRCNN; (d) SAR; (e) TV; (f) NS; (g) Proposed

    图 4  分辨率板图像增强结果。(a)第一幅低分辨率图像;(b) BBC;(c) SRCNN;(d) SAR;(e) TV;(f) NS;(g)本文方法

    Figure 4.  The enhancement result on resolution chart image. (a) First LR image; (b) BBC; (c) SRCNN; (d) SAR; (e) TV; (f) NS; (g) Proposed

    表 1  模糊核SAR和总变差先验模型时推荐方法PSNR(dB)值比较

    Table 1.  Comparison of PSNR(dB) derived by SAR and TV prior blur kernel

    模糊核模型 Case1 Case2
    15 dB 35 dB 55 dB 15 dB 35 dB 55 dB
    SAR先验 30.76 34.86 36.96 30.92 35.50 40.12
    总变差先验 30.77 35.82 40.20 30.93 37.99 45.74
    下载: 导出CSV

    表 2  模糊核SAR和总变差先验模型时推荐方法SSIM值比较

    Table 2.  Comparison of SSIM derived by SAR and TV prior blur kernel

    模糊核模型 Case1 Case2
    15 dB 35 dB 55 dB 15 dB 35 dB 55 dB
    SAR先验 0.9893 0.9959 0.9971 0.9899 0.9965 0.9971
    总变差先验 0.9895 0.9967 0.9988 0.9899 0.9980 0.9997
    下载: 导出CSV

    表 3  模糊核设置1时,各图像分辨率增强方法PSNR(dB)值比较

    Table 3.  Comparison of PSNR(dB) derived by several image resolution enhancement methods for blur kernel case 1

    Image SNR/dB BBC SRCNN SAR TV NS Proposed
    Lena 15 25.50 24.08 29.85 30.76 30.76 30.77
    35 26.84 27.03 33.60 34.45 33.96 35.82
    55 28.86 27.07 34.04 34.94 34.29 40.20
    EIA 15 17.12 17.23 23.18 24.04 23.77 24.16
    35 17.54 18.29 25.29 26.96 25.40 30.50
    55 17.55 18.31 25.36 27.01 25.47 34.73
    下载: 导出CSV

    表 4  模糊核设置1时,各图像分辨率增强方法SSIM值比较

    Table 4.  Comparison of SSIM derived by several image resolution enhancement methods for blur kernel case 1

    Image SNR/dB BBC SRCNN SAR TV NS Proposed
    Lena 15 0.9641 0.9515 0.9870 0.9895 0.9895 0.9895
    35 0.9734 0.9748 0.9946 0.9951 0.9947 0.9967
    55 0.9735 0.9750 0.9951 0.9960 0.9954 0.9988
    EIA 15 0.8456 0.8680 0.9685 0.9746 0.9736 0.9752
    35 0.8598 0.8955 0.9815 0.9873 0.9824 0.9943
    55 0.8601 0.8959 0.9818 0.9874 0.9826 0.9978
    下载: 导出CSV

    表 5  模糊核设置2时,各图像分辨率增强方法PSNR(dB)值比较

    Table 5.  Comparison of PSNR(dB) derived by several image resolution enhancement methods for blur kernel case 2

    Image SNR/dB BBC SRCNN SAR TV NS Proposed
    Lena 15 25.78 23.63 29.95 30.91 30.80 30.93
    35 27.22 26.25 33.12 33.99 33.58 37.99
    55 27.24 26.29 33.54 34.17 34.11 45.74
    EIA 15 17.43 16.23 22.98 24.02 22.64 24.04
    35 17.88 16.97 25.00 25.81 25.57 32.72
    55 17.89 16.98 25.06 25.92 25.76 38.04
    下载: 导出CSV

    表 6  模糊核设置2时,各图像分辨率增强方法SSIM值比较

    Table 6.  Comparison of SSIM derived by several image resolution enhancement methods for blur kernel case 2

    Image SNR/dB BBC SRCNN SAR TV NS Proposed
    Lena 15 0.9668 0.9476 0.9874 0.9899 0.9897 0.9899
    35 0.9759 0.9707 0.9940 0.9950 0.9946 0.9980
    55 0.9760 0.9710 0.9945 0.9952 0.9952 0.9997
    EIA 15 0.8656 0.8513 0.9675 0.9733 0.9663 0.9734
    35 0.8790 0.8738 0.9799 0.9836 0.9828 0.9966
    55 0.8793 0.8741 0.9802 0.9840 0.9836 0.9990
    下载: 导出CSV
  • [1]

    郭迎辉, 蒲明博, 马晓亮, 等.电磁超构材料色散调控研究进展[J].光电工程, 2017, 44(1): 3-22. doi: 10.3969/j.issn.1003-501X.2017.01.001

    Guo Y H, Pu M B, Ma X L, et al. Advances of dispersion-engineered metamaterials[J]. Opto-Electronic Engineering, 2017, 44(1): 3-22. doi: 10.3969/j.issn.1003-501X.2017.01.001

    [2]

    Dong C, Loy C C, He K M, et al. Image super-resolution using deep convolutional networks[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2016, 38(2): 295-307. doi: 10.1109/TPAMI.2015.2439281

    [3]

    Ledig C, Theis L, Huszár F, et al. Photo-realistic single image super-resolution using a generative adversarial network[C]//Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, 2017: 105-114.

    [4]

    汪荣贵, 汪庆辉, 杨娟, 等.融合特征分类和独立字典训练的超分辨率重建[J].光电工程, 2018, 45(1): 170542. doi: 10.12086/oee.2018.170542

    Wang R G, Wang Q H, Yang J, et al. Image super-resolution reconstruction by fusing feature classification and independent dictionary training[J]. Opto-Electronic Engineering, 2018, 45(1): 170542. doi: 10.12086/oee.2018.170542

    [5]

    Nasrollahi K, Moeslund T B. Super-resolution: a comprehensive survey[J]. Machine Vision and Applications, 2014, 25(6): 1423-1468. doi: 10.1007/s00138-014-0623-4

    [6]

    Yue L W, Shen H F, Li J, et al. Image super-resolution: the techniques, applications, and future[J]. Signal Processing, 2016, 128: 389-408. doi: 10.1016/j.sigpro.2016.05.002

    [7]

    Tsai R Y, Huang T S. Multi-frame image restoration and registration[M]//Huang T S. Advances in Computer Vision and Image Processing. Greenwich: JAI Press, 1984: 317-339.

    [8]

    Shen H F, Peng L, Yue L W, et al. Adaptive norm selection for regularized image restoration and super-resolution[J]. IEEE Transactions on Cybernetics, 2016, 46(6): 1388-1399. doi: 10.1109/TCYB.2015.2446755

    [9]

    Zeng X Y, Yang L H. A robust multiframe super-resolution algorithm based on half-quadratic estimation with modified BTV regularization[J]. Digital Signal Processing, 2013, 23(1): 98-109. doi: 10.1016/j.dsp.2012.06.013

    [10]

    Huang S Y, Sun J, Yang Y, et al. Multi-frame super-resolution reconstruction based on gradient vector flow hybrid field[J]. IEEE Access, 2017, 5: 21669-21683. doi: 10.1109/ACCESS.2017.2757239

    [11]

    Molina R, Vega M, Abad J, et al. Parameter estimation in Bayesian high-resolution image reconstruction with multisensors[J]. IEEE Transactions on Image Processing, 2003, 12(12): 1655-1667. doi: 10.1109/TIP.2003.818117

    [12]

    Babacan S D, Molina R, Katsaggelos A K. Variational Bayesian super resolution[J]. IEEE Transactions on Image Processing, 2011, 20(4): 984-999. doi: 10.1109/TIP.2010.2080278

    [13]

    Villena S, Vega M, Molina R, et al. A non-stationary image prior combination in super-resolution[J]. Digital Signal Processing, 2014, 32: 1-10. doi: 10.1016/j.dsp.2014.05.017

    [14]

    Liu C, Sun D Q. On Bayesian adaptive video super resolution[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2014, 36(2): 346-360. doi: 10.1109/TPAMI.2013.127

    [15]

    Ng M K, Shen H F, Lam E Y, et al. A total variation regularization based super-resolution reconstruction algorithm for digital video[J]. EURASIP Journal on Advances in Signal Processing, 2007, 2007: 074585. doi: 10.1155/2007/74585

    [16]

    Beal M J. Variational algorithms for approximate Bayesian inference[D]. London, UK: University College London, 2003.

    [17]

    Bioucas-Dias J M, Figueiredo M A T, Oliveira J P. Total variation-based image deconvolution: a majorization-minimization approach[C]//Proceedings of 2006 IEEE International Conference on Acoustics Speech and Signal Processing Proceedings, Toulouse, 2006: 861-864.

    [18]

    Dong C, Loy C C, He K M, et al. Image super-resolution using deep convolutional networks[EB/OL]. http://mmlab.ie.cuhk.edu.hk/projects/SRCNN.html.

    [19]

    Lucas B, Kanade T. An iterative image registration technique with an application to stereo vision[C]//Proceedings of the 7th International Joint Conference on Artificial Intelligence, San Francisco, 1981: 674-679.

    [20]

    Wang Z, Bovik A C. Mean squared error: love it or leave it? A new look at signal fidelity measures[J]. IEEE Signal Processing Magazine, 2009, 26(1): 98-117. doi: 10.1109/MSP.2008.930649

    [21]

    Wang Z, Bovik A C, Sheikh H R, et al. Image quality assessment: from error visibility to structural similarity[J]. IEEE Transactions on Image Processing, 2004, 13(4): 600-612. doi: 10.1109/TIP.2003.819861

  • 加载中

(4)

(6)

计量
  • 文章访问数:  6645
  • PDF下载数:  2864
  • 施引文献:  0
出版历程
收稿日期:  2018-04-03
修回日期:  2018-05-31
刊出日期:  2019-06-01

目录

/

返回文章
返回