E-mail Alert
RSS
| Citation: |
Tian X, Li RZ, Peng T et al. Multi-prior physics-enhanced neural network enables pixel super-resolution and twin-image-free phase retrieval from single-shot hologram. Opto-Electron Adv 7, 240060 (2024). doi: 10.29026/oea.2024.240060
|
Multi-prior physics-enhanced neural network enables pixel super-resolution and twin-image-free phase retrieval from single-shot hologram
-
Abstract
Digital in-line holographic microscopy (DIHM) is a widely used interference technique for real-time reconstruction of living cells’ morphological information with large space-bandwidth product and compact setup. However, the need for a larger pixel size of detector to improve imaging photosensitivity, field-of-view, and signal-to-noise ratio often leads to the loss of sub-pixel information and limited pixel resolution. Additionally, the twin-image appearing in the reconstruction severely degrades the quality of the reconstructed image. The deep learning (DL) approach has emerged as a powerful tool for phase retrieval in DIHM, effectively addressing these challenges. However, most DL-based strategies are data-driven or end-to-end net approaches, suffering from excessive data dependency and limited generalization ability. Herein, a novel multi-prior physics-enhanced neural network with pixel super-resolution (MPPN-PSR) for phase retrieval of DIHM is proposed. It encapsulates the physical model prior, sparsity prior and deep image prior in an untrained deep neural network. The effectiveness and feasibility of MPPN-PSR are demonstrated by comparing it with other traditional and learning-based phase retrieval methods. With the capabilities of pixel super-resolution, twin-image elimination and high-throughput jointly from a single-shot intensity measurement, the proposed DIHM approach is expected to be widely adopted in biomedical workflow and industrial measurement. -
-
References
[1] Kemper B, von Bally G. Digital holographic microscopy for live cell applications and technical inspection. Appl Opt 47, A52–A61 (2008). doi: 10.1364/AO.47.000A52 [2] Schnars U, Jüptner WPO. Digital recording and numerical reconstruction of holograms. Meas Sci Technol 13, R85–R101 (2002). doi: 10.1088/0957-0233/13/9/201 [3] Garcia-Sucerquia J, Xu WB, Jericho SK et al. Digital in-line holographic microscopy. Appl Opt 45, 836–850 (2006). doi: 10.1364/AO.45.000836 [4] Zhou J, Jin YB, Lu LP et al. Deep learning-enabled pixel-super-resolved quantitative phase microscopy from single-shot aliased intensity measurement. Laser Photonics Rev 18, 2300488 (2024). doi: 10.1002/lpor.202300488 [5] de Almeida JL, Comunello E, Sobieranski A et al. Twin-image suppression in digital in-line holography based on wave-front filtering. Pattern Anal Appl 24, 907–914 (2021). doi: 10.1007/s10044-020-00949-7 [6] Bai C, Peng T, Min JW et al. Dual-wavelength in-line digital holography with untrained deep neural networks. Photonics Res 9, 2501–2510 (2021). doi: 10.1364/PRJ.441054 [7] Zhang JL, Sun JS, Chen Q et al. Adaptive pixel-super-resolved lensfree in-line digital holography for wide-field on-chip microscopy. Sci Rep 7, 11777 (2017). doi: 10.1038/s41598-017-11715-x [8] Luo W, Zhang YB, Feizi A et al. Pixel super-resolution using wavelength scanning. Light Sci Appl 5, e16060 (2016). [9] Pellizzari CJ, Spencer MF, Bouman CA. Coherent plug-and-play: digital holographic imaging through atmospheric turbulence using model-based iterative reconstruction and convolutional neural networks. IEEE Trans Comput Imag 6, 1607–1621 (2020). doi: 10.1109/TCI.2020.3042948 [10] Chang XY, Bian LH, Gao YH et al. Plug-and-play pixel super-resolution phase retrieval for digital holography. Opt Lett 47, 2658–2661 (2022). doi: 10.1364/OL.458117 [11] Bao P, Situ GH, Pedrini G et al. Lensless phase microscopy using phase retrieval with multiple illumination wavelengths. Appl Opt 51, 5486–5494 (2012). doi: 10.1364/AO.51.005486 [12] Luo W, Greenbaum A, Zhang YB et al. Synthetic aperture-based on-chip microscopy. Light Sci Appl 4, e261 (2015). doi: 10.1038/lsa.2015.34 [13] Yamaguchi I, Zhang T. Phase-shifting digital holography. Opt Lett 22, 1268–1270 (1997). doi: 10.1364/OL.22.001268 [14] Song J, Swisher CL, Im H et al. Sparsity-based pixel super resolution for lens-free digital in-line holography. Sci Rep 6, 24681 (2016). doi: 10.1038/srep24681 [15] Raupach SMF. Cascaded adaptive-mask algorithm for twin-image removal and its application to digital holograms of ice crystals. Appl Opt 48, 287–301 (2009). doi: 10.1364/AO.48.000287 [16] Zhang WH, Cao LC, Brady DJ et al. Twin-image-free holography: A compressive sensing approach. Phys Rev Lett 121, 093902 (2018). doi: 10.1103/PhysRevLett.121.093902 [17] Gao YH, Cao LC. Generalized optimization framework for pixel super-resolution imaging in digital holography. Opt Express 29, 28805–28823 (2021). doi: 10.1364/OE.434449 [18] Wang H, Lyu M, Situ GH. eHoloNet: A learning-based end-to-end approach for in-line digital holographic reconstruction. Opt Express 26, 22603–22614 (2018). doi: 10.1364/OE.26.022603 [19] Rivenson Y, Zhang YB, Günaydın H et al. Phase recovery and holographic image reconstruction using deep learning in neural networks. Light Sci Appl 7, 17141 (2018). [20] Lempitsky V, Vedaldi A, Ulyanov D. Deep image prior. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition 9446–9454 (IEEE, 2018); http://doi.org/10.1109/CVPR.2018.00984. [21] Wang F, Bian YM, Wang HC et al. Phase imaging with an untrained neural network. Light Sci Appl 9, 77 (2020). doi: 10.1038/s41377-020-0302-3 [22] Han F, Mu TK, Li HY et al. Deep image prior plus sparsity prior: Toward single-shot full-stokes spectropolarimetric imaging with a multiple-order retarder. Adv Photonics 2, 036009 (2023). [23] Galande AS, Thapa V, Gurram HPR et al. Untrained deep network powered with explicit denoiser for phase recovery in inline holography. Appl Phys Lett 122, 133701 (2023). doi: 10.1063/5.0144795 [24] Niknam F, Qazvini H, Latifi H. Holographic optical field recovery using a regularized untrained deep decoder network. Sci Rep 11, 10903 (2021). doi: 10.1038/s41598-021-90312-5 [25] Mait JN, Euliss GW, Athale RA. Computational imaging. Adv Opt Photonics 10, 409–483 (2018). doi: 10.1364/AOP.10.000409 [26] Zhao WS, Zhao SQ, Li LJ et al. Sparse deconvolution improves the resolution of live-cell super-resolution fluorescence microscopy. Nat Biotechnol 40, 606–617 (2022). doi: 10.1038/s41587-021-01092-2 [27] Zhao H, Gallo O, Frosio I et al. Loss functions for image restoration with neural networks. IEEE Trans Comput Imag 3, 47–57 (2017). doi: 10.1109/TCI.2016.2644865 [28] Ravishankar S, Ye JC, Fessler JA. Image reconstruction: from sparsity to data-adaptive methods and machine learning. Proc IEEE 108, 86–109 (2020). doi: 10.1109/JPROC.2019.2936204 [29] Ronneberger O, Fischer P, Brox T. U-net: convolutional networks for biomedical image segmentation. In Proceedings of the 18th International Conference on Medical Image Computing and Computer-Assisted Intervention 234–241 (Springer, 2015); http://doi.org/10.1007/978-3-319-24574-4_28. [30] Schanz D, Gesemann S, Schröder A et al. Non-uniform optical transfer functions in particle imaging: Calibration and application to tomographic reconstruction. Meas Sci Technol 24, 024009 (2013). doi: 10.1088/0957-0233/24/2/024009 [31] Bai C, Liu C, Jia H et al. Compressed blind deconvolution and denoising for complementary beam subtraction light-sheet fluorescence microscopy. IEEE Trans Biomed Eng 66, 2979–2989 (2019). doi: 10.1109/TBME.2019.2899583 [32] Crete F, Dolmiere T, Ladret P et al. The blur effect: Perception and estimation with a new no-reference perceptual blur metric. In Proceedings of the SPIE 6492, Human Vision and Electronic Imaging XII 64920I (SPIE, 2007); http://doi.org/10.1117/12.702790. [33] Polyanskiy MN. Refractiveindex. Info database of optical constants. Sci Data 11, 94 (2024). doi: 10.1038/s41597-023-02898-2 [34] Luke SM, Vukusic P, Hallam B. Measuring and modelling optical scattering and the colour quality of white pierid butterfly scales. Opt Express 17, 14729–14743 (2009). doi: 10.1364/OE.17.014729 [35] Zhang K, Liang JY, Van Gool L et al. Designing a practical degradation model for deep blind image super-resolution. In 2021 IEEE/CVF International Conference on Computer Vision (ICCV) 4771–4780 (IEEE, 2021); http://doi.org/10.1109/ICCV48922.2021.00475. -
Access History
Figures(14)
Tables(3)
Article Metrics
Export File
Citation
Tian X, Li RZ, Peng T et al. Multi-prior physics-enhanced neural network enables pixel super-resolution and twin-image-free phase retrieval from single-shot hologram. Opto-Electron Adv 7, 240060 (2024). doi: 10.29026/oea.2024.240060
Format
Content
DownLoad:
-
Figure 1.
An overview of the proposed reconstruction method. (a) Diagrams of classical solutions based on CS and TV regularization framework. (b) DIP-based reconstruction method combined with physical model. (c) Pipeline of MPPN-PSR for hologram reconstruction. A measured hologram I of a phase object Ψ is the input to the neural networks. The output of the neural networks is taken as the estimated phase
$ {\boldsymbol{G}}_{{\mathrm{MPPN}}}^{p} $ $ {\boldsymbol{G}}_{{\mathrm{MPPN}}}^{p} $ $ {\boldsymbol{G}}_{{\mathrm{MPPN}}}^{p} $ -
Figure 2.
Schematic of the experimental setup of the DIHM.
-
Figure 3.
Simulated phase objects (960 × 960 pixels) and corresponding single-shot DIHM imaging results (320 × 320 pixels), in which the spatial down-sampling rate was set as θ = 3. The red scale bar measures 500 μm.
-
Figure 4.
(a) Phase retrieval results of different methods on simulated holograms (three times down-sampling) via back propagation, TwIST-TV-PSR, PnP-TV-PSR, PnP-FFDNet-PSR, PN-PSR, and MPPN-PSR, respectively. The autocorrelation functions of (b) resolution chart and (c) cell with different methods. The profiles along the blue lines are also investigated while the red scale bar measures 500 μm.
-
Figure 5.
MPPN-PSR results of the holograms simulated with different down-sampling rates in terms of (a) resolution chart and (b) cell, respectively. (c) the examination of ratio β, which serves as a key proportional coefficient to balance the ℓ-1 norm and ℓ-2 norm terms. (d) The evolution of the MSE with an increasing number of epochs. The profiles along the blue lines are also investigated while the red scale bar measures 500 μm.
-
Figure 6.
Reconstruction results of different methods at different noise levels, the red scale bar measures 500 μm. (a) Reconstruction results of resolution chart, the down-sampling rate θ = 2. (b) Reconstruction results of cell, the down-sampling rate θ = 3.
-
Figure 7.
Quantitative analysis of the effect of noise on the reconstruction results using different methods. (a) and (b) are the trends of PSNR and SSIM indices of reconstruction results of resolution chart when the down-sampling rate θ = 2, while (c) and (d) are those of cell when the down-sampling rate θ = 3.
-
Figure 8.
The practicality and persuasiveness of the MPPN-PSR method in real-world applications was evaluated by standard imaging and pixel binning modes of the camera. The pixel pitch varied from (a) 6.5 μm to (b) 13 μm, representing the resolution has been reduced to 1/4 of the original. The blue and green boxes respectively select the ROI of the hologram and results reconstructed by different methods.
-
Figure 9.
Experimental images of the phase step (a1–a7) and PMMA beads (b1–b7) processed with back propagation, TwIST-TV-PSR, PnP-FFDNet-PSR, PN-PSR, and MPPN-PSR methods, respectively. The down-sampling rate is θ = 3, consistent with all subsequent experiments. The cross-section phase profiles (along the blue lines) were also measured in insets and the corresponding optical thickness maps are shown. The reconstruction size is 1200×1200 pixels, corresponding to the FOV of 130×130 µm2. The red scale bar measures 20 µm.
-
Figure 10.
Imaging results of (a) butterfly wing and (b) fish ovary with different methods, including the reconstructed phase maps and the magnified views of selected regions. The reconstruction size is 1536 × 1536 pixels, i.e. θ = 3, corresponding to the FOV of 166×166 µm2. The red scale bar measures 25 µm.
-
Figure 11.
Imaging results of (a) TOMM20 antibody and (b) frog intestine by different methods. The reconstruction size is 2700 × 2700 pixels, i.e. θ=3, corresponding to the FOV of 293×293 µm2. The red scale bar measures 40 µm.
-
Figure 12.
The autocorrelation functions of (a) butterfly wing and (b) fish ovary with different methods.
-
Figure 13.
The experimental result of MPPN-PSR to reconstruct the full-FOV high-resolution phase image of a quantitative phase target. (a) The full-FOV LR defocused hologram, and the phase image reconstructed by TwIST-TV-PSR and PN-PSR. (b) The full-FOV PSR phase image reconstructed by MPPN-PSR from a single frame of LR hologram. (c) The LR bright field images of three ROIs. (d) The phase images of the three ROIs reconstructed by MPPN-PSR, TwIST-TV-PSR and PN-PSR, respectively. (e) The results of cascading non-PSR reconstruction with outstanding pixel super-resolution networks, i.e. BSRGAN and BSRNet, have also been demonstrated, although this approach represents over-smooth and artifacts. The profiles along the blue and orange-colored lines are also investigated.
-
Figure 14.
(a) The full-FOV PSR phase reconstruction of the TOMM20 antibody by MPPN-PSR, (b) and the comparison of PSR phase images of two ROIs. (c) The corresponding optical thickness maps are shown as well. (d) BSRGAN and BSRNet networks introduce unsatisfactory artifacts with non PSR reconstruction as well. The purple area represents the FOV that a 40× objective lens can bring.
