Image restoration of mobile phone under-screen imaging based on deconvolution

Wang Xiaona; Huang Yuran; Kuang Cuifang; Li Haifeng; Liu Xu

doi:10.12086/oee.2021.200423

Article navigation > Opto-Electronic Engineering > 2021 Vol. 48 > No. 6 > 200423

Next Article Previous Article

Wang X N, Huang Y R, Kuang C F, et al. Image restoration of mobile phone under-screen imaging based on deconvolution[J]. Opto-Electron Eng, 2021, 48(6): 200423. doi: 10.12086/oee.2021.200423

Citation:

Wang X N, Huang Y R, Kuang C F, et al. Image restoration of mobile phone under-screen imaging based on deconvolution[J]. Opto-Electron Eng, 2021, 48(6): 200423. doi: 10.12086/oee.2021.200423

Image restoration of mobile phone under-screen imaging based on deconvolution

1.
State Key Laboratory of Modern Optical Instrumentation, College of Optical Science and Engineering, Zhejiang University, Hangzhou, Zhejiang 310027, China
2.
Ningbo Research Institute, Zhejiang University, Ningbo, Zhejiang 315100, China

Fund Project: Research and Development Project of Major Scientific Research Instruments of National Natural Science Foundation of China (61827825)

More Information

^*Corresponding author: Kuang Cuifang, E-mail: cfkuang@zju.edu.cn

Received Date 16 November 2020

Revised Date 21 January 2021

Published Date 01 June 2021

Abstract

Abstract

With the development of full-screen mobile phones, the need for under-screen imaging of mobile phones has emerged. However, the diffraction caused by the wiring and other opaque parts will affect the image quality of the under-screen image. In this article, under-screen image is restored from the perspective of image restoration. The point spread function (PSF) of the mobile phone imaging system is obtained through actual measurement, and the image is deconvolved using the measured PSF. In this article, traditional deconvolution method has been improved, in which the color space of the image is converted and different channels are processed separately. Compared with the traditional deconvolution method, the results of the sub-channel deconvolution method have improved structural similarity (SSIM), peak signal-to-noise ratio (PSNR) and other indicators, and the required running time is shorter. After sub-channel deconvolution, the non-local averaging algorithm is used for denoising, which further improves the quality of the under-screen image.
- deconvolution /
- denoising /
- under-screen imaging /
- image restoration

FullText(HTML)

References

[1]	Holmes T J. Blind deconvolution of quantum-limited incoherent imagery: maximum-likelihood approach[J]. J Opt Soc Am A, 1992, 9(7): 1052-1061. doi: 10.1364/JOSAA.9.001052 CrossRef Google Scholar
[2]	Fish D A, Brinicombe A M, Pike E R, et al. Blind deconvolution by means of the Richardson-Lucy algorithm[J]. J Opt Soc Am A, 1995, 12(1): 58-65. doi: 10.1364/JOSAA.12.000058 CrossRef Google Scholar
[3]	Tugnait J K. Identification and deconvolution of multichannel linear non-Gaussian processes using higher order statistics and inverse filter criteria[J]. IEEE Trans Signal Process, 1997, 45(3): 658-672. doi: 10.1109/78.558482 CrossRef Google Scholar
[4]	Wiener N. Extrapolation, Interpolation, and Smoothing of Stationary Time Series: with Engineering Applications[M]. Cambridge: MIT Press, 1964: 1043-1054. Google Scholar
[5]	Richardson W H. Bayesian-based iterative method of image restoration[J]. J Opt Soc Am, 1972, 62(1): 55-59. doi: 10.1364/JOSA.62.000055 CrossRef Google Scholar
[6]	Lucy L B. An iterative technique for the rectification of observed distributions[J]. Astron J, 1974, 79: 745. doi: 10.1086/111605 CrossRef Google Scholar
[7]	Bayes. An essay towards solving a problem in the doctrine of chances[J]. Resonance, 2003, 8(4): 80-88. doi: 10.1007/BF02883540 CrossRef Google Scholar
[8]	Ingaramo M, York A G, Hoogendoorn E, et al. Richardson-lucy deconvolution as a general tool for combining images with complementary strengths[J]. ChemPhysChem, 2014, 15(4): 794-800. doi: 10.1002/cphc.201300831 CrossRef Google Scholar
[9]	Preibisch S, Amat F, Stamataki E, et al. Efficient bayesian-based multiview deconvolution[J]. Nat Methods, 2014, 11(6): 645-648. doi: 10.1038/nmeth.2929 CrossRef Google Scholar
[10]	Guo M, Li Y, Su Y J, et al. Rapid image deconvolution and multiview fusion for optical microscopy[J]. Nat Biotechnol, 2020, 38(11): 1337-1346. doi: 10.1038/s41587-020-0560-x CrossRef Google Scholar
[11]	Dines K, Kak A. Constrained least squares filtering[J]. IEEE Trans Acoust, Speech, Signal Process, 1977, 25(4): 346-350. doi: 10.1109/TASSP.1977.1162965 CrossRef Google Scholar
[12]	Ronneberger O, Fischer P, Brox T. U-Net: convolutional networks for biomedical image segmentation[C]//Proceedings of the 18th International Conference on Medical Image Computing and Computer-Assisted Intervention, Munich, Germany, 2015: 234-241. Google Scholar
[13]	Dong C, Loy C C, He K M, et al. Learning a deep convolutional network for image super-resolution[C]//Proceedings of the 13th European Conference on Computer Vision, Zurich, Switzerland, 2014: 184-199. Google Scholar
[14]	Dong C, Loy C C, He K M, et al. Image super-resolution using deep convolutional networks[J]. IEEE Trans Pattern Anal Mach Intell, 2016, 38(2): 295-307. doi: 10.1109/TPAMI.2015.2439281 CrossRef Google Scholar
[15]	Goodfellow I J, Pouget-abadie J, Mirza M, et al. Generative adversarial networks[EB/OL]. (2014-06-10)[2020-08-07]. https://arxiv.org/abs/1406.2661. Google Scholar
[16]	Ledig C, Theis L, Huszár F, et al. Photo-realistic single image super-resolution using a generative adversarial network[C]//Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, USA, 2017: 105-114. Google Scholar
[17]	Peng Y F, Sun Q L, Dun X, et al. Learned large field-of-view imaging with thin-plate optics[J]. ACM Trans Graph, 2019, 38(6): 219. Google Scholar
[18]	Dabov K, Foi A, Katkovnik V, et al. Image denoising by sparse 3-D transform-domain collaborative filtering[J]. IEEE Trans Image Process, 2007, 16(8): 2080-2095. doi: 10.1109/TIP.2007.901238 CrossRef Google Scholar
[19]	Buades A, Coll B, Morel J M. A non-local algorithm for image denoising[C]//Proceedings of 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, San Diego, USA, 2005. Google Scholar
[20]	冈萨雷斯R C, 伍兹R E, 艾丁斯S L. 数字图像处理[M]. 阮秋琦, 阮宇智, 译. 2版. 北京: 电子工业出版社, 2003: 42-46. Google Scholar Gonzalez R C, Woods R E, Eddins S L. Digital Image Processing[M]. Ruan Q Q, Ruan Y Z, trans. 2nd ed. Beijing: Publishing House of Electronics Industry, 2003: 42-46. Google Scholar
[21]	Wang J, Guo Y W, Ying W T, et al. Fast non-local algorithm for image denoising[C]//Proceedings of 2006 International Conference on Image Processing, Atlanta, USA, 2016: 1429-1432. Google Scholar
[22]	Froment J. Parameter-free fast pixelwise non-local means denoising[J]. Image Process Line, 2014, 4: 300-326 doi: 10.5201/ipol.2014.120 CrossRef Google Scholar
[23]	Lin Z, Song E M. A fast non-local means algorithm based on integral image and reconstructed similar kernel[J]. Proc SPIE, 2018, 10609: 106091L. Google Scholar
[24]	Wang Z, Bovik A C, Sheikh H R, et al. Image quality assessment: from error visibility to structural similarity[J]. IEEE Trans Image Process, 2004, 13(4): 600-612. doi: 10.1109/TIP.2003.819861 CrossRef Google Scholar
[25]	Jiang J, Huang J N, Zhang G J. An accelerated motion blurred star restoration based on single image[J]. IEEE Sens J, 2017, 17(5): 1306-1315. doi: 10.1109/JSEN.2016.2645861 CrossRef Google Scholar
[26]	He T, Hu J, Huang H Q. Hybrid high-order nonlocal gradient sparsity regularization for Poisson image deconvolution[J]. Appl Opt, 2018, 57(35): 10243-10256. doi: 10.1364/AO.57.010243 CrossRef Google Scholar
[27]	MacAdam D L. Projective transformations of I. C. I. color specifications[J]. J Opt Soc Am, 1937, 27(8): 294-299. doi: 10.1364/JOSA.27.000294 CrossRef Google Scholar

Overview

Overview

Overview: With the development of full-screen mobile phones, the need for under-screen imaging of mobile phones has emerged. However, the diffraction caused by the wiring and other opaque parts will affect the image quality of the under-screen image. In this article, under-screen image is restored from the perspective of image restoration. The point spread function (PSF) of the mobile phone imaging system is obtained through actual measurement, and the image is deconvolved using the measured PSF. Traditional deconvolution method has been improved in this article. In deconvolution process, the traditional and common way is to divide the image into different color channels, use the PSF of the corresponding channel to deconvolve each channel, and finally synthesize the color image. Because the high noise of the image sensor of the mobile phone has a destructive effect on the deconvolution algorithm, one need to reduce the noise of the image. The traditional denoising method will destroy the high-frequency information and cause serious ringing effect on the image restored by deconvolution. In this article, we propose a new solution to this problem: convert the blurred image from RGB color space to YCrCb space, where Y represents brightness information; Cr and Cb represent hue and saturation, respectively. Image clarity is mainly affected by brightness information, so only the Y channel needs to be deconvolved, whose noise level is lower than any of the RGB channels. In order to further reduce the influence of noise, Cr and Cb channels are processed by Gaussian filtering to reduce noise. Finally, the processed image is converted back to RGB color space to form a traditional color image. Compared with traditional deconvolution method, the results of the sub-channel deconvolution method have improved structural similarity (SSIM), peak signal-to-noise ratio (PSNR) and other indicators, and the required running time is shortened by almost three times. After sub-channel deconvolution, the sharpness of the image has been greatly improved. In order to further improve the quality of under-screen image, the non-local averaging algorithm is used to denoise the image after the sub-channel deconvolution, which finds similar image blocks in the same image to average, and redundant information in the image is used to remove noise. Furthermore, the integral image method is used to shorten the running time to meet the real-time requirements in mobile phone photography. The visual perception of the image has been better improved, and both PSNR and SSIM have been further improved after denoising.