
Citation: | Tian X, Li RZ, Peng T et al. Multi-prior physics-enhanced neural network enables pixel super-resolution and twin-image-free phase retrieval from single-shot hologram. Opto-Electron Adv 7, 240060 (2024). doi: 10.29026/oea.2024.240060 |
Secret sharing distributes the secret image to different shareholders and decodes the secret images by conjoining sufficient shares together. It is considered as a trustworthy cryptographic method and has been widely used in the fields of privacy protection and intellectual property rights1, 2. Due to the parallel information processing ability and multi-dimensional multiplexing capability, optical encryption has attracted more attention in the last decade3-19. As an essential security element, optical holography, where the secret images are embedded into the pre-encoded phase-only holograms, is an effective way to realize optical secret sharing20-27. Nevertheless, the bulky optical components and limited modulation ability hinder optical holography to build up a compact and multifunctional secret sharing system28. To break these bottlenecks, cascaded metasurface has further been used for secret sharing. In this method, the secret information can only be decrypted when the metasurfaces are accurately combined, thus ensuring the security of the secret sharing system29-32. However, owing to the inherent limitations of metasurface, it is difficult to create secret sharing platforms massively with merit of dynamic tunability.
Due to the multi-dimensional modulation capacity, voltage sensitivity and high diffraction efficiency, liquid crystals (LCs) with anisotropic molecule architectures have evolved as a promising candidate for information encryption33-37. For example, LC holograms based on chirality invertible superstructures have been proposed to enable a high-dimensional multiplexing information encryption platform38. However, so far, constrained by traditional algorithm, most LC holographic devices are one-layer configuration, which only allow for the access to different information channels, but not the physical segmenting of these information channels among different shareholders. In this context, cascaded LC holograms provide the possibility for the physical separation of the secret information and have great potential for creating a multi-dimensional multiplexing, dynamic adjustable and high diffraction efficiency secret sharing platform.
In this work, we experimentally demonstrate a hybrid-multiplexing LC optical secret sharing framework. The secret information is decomposed and dispensed into two co-constrained LC holograms, each of which can reconstruct an authentication image in the specific location. Moreover, when these two LC holograms are stacked together, six independent new holographic images can be decrypted with different secret keys, including polarization of the incident light and the distance between the cascaded LC holograms. We utilize the ANN to complete the inverse deign of the sophisticated multi-confined and cascaded multilayer issues. This cascaded LC scheme breaks the limitations of traditional optical secret sharing, and blazes a trail in broad areas of optical visual cryptography, dynamic holographic display, and ultra-high-capacity information storage.
The optical secret sharing framework based on the cascaded LC holograms is illustrated in Fig. 1(a). By optimizing the corresponding phase distribution of each LC hologram through ANN method, the incident light with specific polarization state is modulated by the cascaded LC holograms and the secret information is decrypted at the predetermined distance. The LC element consists of two indium-tin-oxide glass outside, two photoalignment layers inside and one nematic LC layer in the middle. The modulation function of the LC can be changed by setting different external voltages, as displayed in the zoom-in view in Fig. 1(a). If the incident light is left circular polarization (LCP), and only the LC hologram 1 is in the ‘on’ state (Uon), the authentication image ‘2’ will be decrypted at the predetermined location and recorded by the charge-coupled device (CCD). In this situation, the incident light is modulated only by LC hologram 1 and equivalent to directly passing through LC hologram 2 without phase modulation. When the working state of the two LC holograms is inverse, the authentication image ‘4’ will be decrypted at the same position. Keeping the position of LC hologram 1 unchanged and moving the LC hologram 2, six secret images will be decrypted when the corresponding polarized light and distance are used as the secret keys, as shown in Fig. 1(b). Specifically, the polarization state of incident light includes LCP and right circular polarization (RCP). In addition, the initial distance between the cascaded LC holograms is d1=4 cm, and the moving step length of the LC hologram 2 along the axial direction is 1.5 cm (d2=5.5 cm, d3=7 cm). The peak signal-to-noise ratio (PSNR) is adopted to evaluate the quality of the holographic reconstructed images. To obtain the value range of the distance with good effectiveness, the relationship of the average PSNR of the two circular-polarization-multiplexing images with the distance has been firstly analyzed (
According to the above-mentioned approach, we structure a neural network with multi-dimensional and co-constrained input fields as shown in Fig. 2. The polarization states of the incident light, the imposed external voltage and the moving distance between the two LC holograms are presumed as the input parameters. The angular spectrum diffraction theory builds up the full connection of all layers. The forward propagation model of the wavefronts in the free space after modulated by the layers can be described as
E(x,y)=F−1{F[ES(x,y)]⋅Hd(u,v)}, | (1) |
where F and F–1 denote the Fourier transform and the inverse Fourier transform, respectively. ES(x, y) represents the product of input field O(x0, y0) and the transmission coefficient of the neurons g(φs). The transfer function Hd (u, v) is expressed as
Hd(u,v)=exp[ikd√1−λ2u2−λ2v2], | (2) |
where u and v are spatial frequencies, and k = 2π/λ is the wave number. The distance between the cascaded LC holograms is d = d0+nΔd, where d0 is the initial distance between the cascaded LC holograms and Δd is the moving step length. According to the above framework, different input fields can be mapped to the corresponding secret image in the output plane by using the error back-propagation algorithm to update the learnable parameters g(φs). The aim of the error back-propagation algorithm is to minimize the loss function (mean squared error, MSE) between the actual and ideal output intensity distribution. The phase distribution of the LC holograms can be obtained when the loss function gradient with regard to phase distribution converges. Owing to the spin-orbital interplay of light in anisotropic medium, the consecutive phase modulation in LC through PB phase is feasible. In practice, all LC unit-cells possess orientation angle in the plane as well as out of plane, called LC director. The Pancharatnam–Berry (PB) phase modulation is twofold of the orientation angle in-plane of each unit-cell. In this way, different in-plane orientation angles ranged in (−90°, 90°) can correspondingly realize the PB phase modulation of (0, 2π).
Through the effective and convenient network, the design of the multi-multiplexing optical holographic secret sharing framework based on the cascaded LC holograms can be accomplished. We verify the performance and practicality of the proposed secret sharing framework using the experimental equipment displayed in Fig. 3(a). In order to import and transmit the expected circular polarization state of the incident light, a linear polarizer (OPPF1-VIS, JCOPTIX, China) and a quarter-wave plate (TWP20Q, JCOPTIX, China) are placed in front of as well as behind the cascaded LC holograms as circular polarizer and analyzer. Two LC holograms are laid on two three-dimensional translation stages to achieve the precise alignment and moving step length, meanwhile the initial distance between the two LC holograms is set as 4 cm. All the decrypted holographic images are collected by a CCD in the same imaging plane and the wavelength of the illumination light is 633 nm (HNL20L, Thorlabs, America). Each LC hologram has an effective size of 2.12 mm×2.12 mm and contains 768×768 pixels. The zoom-in view of Fig. 3(a) individually shows the simulated phase distribution and the polarized micrograph of the fabricated LC hologram samples.
In implementation, the anisotropy will alter when an external electric field is imposed to change the LC director from in-plane to out-of-plane. The variation of anisotropic refractive index bestowed the LC director with the ability of electrically adjustable phase retardation. Therefore, the diffraction efficiency can be dynamically controlled by imposing different external voltages. The tendencies of the diffraction efficiency versus voltage for two LC holograms at working wavelength of 633 nm are depicted in Fig. 3(b), respectively. The optimum on-state voltage for LC hologram 1 and LC hologram 2 is 5.4 V and 5.8 V respectively, and the corresponding diffraction efficiency can reach 61.5% and 62.4%. Nevertheless, the two LC holograms are off-state when the voltages are set as 3.6 V and 3.8 V, respectively. Here, the efficiency, or polarization conversion efficiency, at the activation voltage is determined by the diffraction efficiency of the phase pattern and the transmittance of the LC device. As such, the quality of holographic images from single LC hologram can be improved after selecting the appropriate polarization states. However, for secret sharing, the noise cannot be separated thoroughly from the final secret images through polarization filtering because the polarization states remain unchanged in the processes. Benefitting from the sensitivity of external voltage stimuli, LC holograms acquire the dynamic modulation ability, which provides the possibility to unlock information without detaching any shares and enables the optical secret sharing platform more compact and stable. According to above characteristics, we can conveniently acquire the authentication information and the secret images through the optical secret sharing framework by user-friendly adjusting the external voltage.
Numerous encoding information can transmit in different users with high security through the optical secret sharing framework as displayed in Fig. 4. In the optical secret sharing framework, different secrets are shared and customized keys are assigned to two different users. By using the identity keys, user 1 and user 2 can acquire the authentication images ‘2’ and ‘4’, as shown in Fig. 4(a). Utilizing different operator keys from other information recipient, i.e., the different polarization states and separation distance of two LC samples, the operator images including multiplication (×), division (÷), exponentiation (∧), integer rounding (%), calculating residue (//) and radical sign (√), can be experimentally reconstructed as shown in Fig. 4(b). Combining the authentication images with the operator images to carry out the second decryption, the eventual secret information is worked out. Here, the spatial position of the LC holograms along the propagation direction was adopted as the pre-negotiated calculation order of the secrets in our proposed scheme. The PSNR of experimental results in Fig. 4(b) can be calculated as 16.11 dB (‘2’), 16.77 dB (‘4’), 15.51 dB (‘×’), 15.8 dB (‘÷’), 15.06 dB (‘∧’), 13.58 dB (‘%’), 14.62 dB (‘//’), and 13.91 dB (‘√’), which are close to the theoretical results in Fig. 2. To explain the insufficient image quality, the optimization process in our scheme can mathematically be seen as a multiple loss optimization problem, wherein the loss function is co-constrained by all the mean square error (MSE) for reconstructing multiple images (Supplementary Section 1). As such, the increase of the multiplexing images results in more MSE terms in loss functions, which can be physically interpreted as the noises (or crosstalk) which results in the decrease of the PSNR values (
The proposed multi-dimensional multiplexing optical secret sharing framework allows for the simultaneous and ultra-high security transmission of multiple messages, which overcomes the drawbacks of traditional holographic encryption methods and has lots of superiorities than earlier solutions. Firstly, the secret images are hidden into different shares (LC holograms) and the terminal secret images can only be decrypted through the cascaded shares. Even if one of the shares is stolen, the secret image cannot be retrieved and only the authentication image as camouflage will be revealed, which greatly improves the security of the secret sharing platform. The crack difficulty of the framework was also analyzed when one hologram is fixed and the effective pixel number of the other hologram is ranging (Supplementary Section 2,
In this work, we propose and experimentally demonstrate the multi-dimensional multiplexing optical secret sharing framework with cascaded LC holograms. We create a dependable and practical neural network of error back-propagation based on the angular spectrum diffraction theory to design the framework. The multi-dimensional inputs of our network in the encryption process, such as polarization of light, distance between the LC holograms and the applied external electrical voltage, enhance the security of the secret information. The two authentication images and six operator images can be respectively decrypted if all the secret keys are correct, and the final secret information will be revealed through the second decoding. The full-fledged manufacturing technology of LC devices allows our proposal to be more practicable and versatile. Therefore, with merits of convenient design, extensive fabrication and outstanding security, the multi-dimensional multiplexing optical secret sharing framework has great potential to be used in information storage, dynamic display, and multi-functional optical information processing.
We would like to acknowledge the support from the National Natural Science Foundation of China (No. 62005164, 62222507, 62175101, and 62005166), the Shanghai Natural Science Foundation (23ZR1443700), Shuguang Program of Shanghai Education Development Foundation and Shanghai Municipal Education Commission(23SG41), the Young Elite Scientist Sponsorship Program by CAST (No. 20220042), Science and Technology Commission of Shanghai Municipality (Grant No. 21DZ1100500), the Shanghai Municipal Science and Technology Major Project, and the Shanghai Frontiers Science Center Program (2021–2025 No. 20).
X. Y Fang proposed the original idea and conceived the experiment. K. Y. Li performed the experiment, assisted by B. L. Li and H. T. Luan. Y. M. Wang and P. Chen fabricated the sample and carried out the microimaging. D. P Pi and K. Y. Li wrote the original manuscript, revised by X. Y. Fang and P. Chen. M. Gu, Y. Q. Lu, P. Chen and X. Y. Fang provided the resource support, funding acquisition and supervised the project.
The authors declare no competing financial interests.
†These authors contributed equally to this work
[1] | Kemper B, von Bally G. Digital holographic microscopy for live cell applications and technical inspection. Appl Opt 47, A52–A61 (2008). doi: 10.1364/AO.47.000A52 |
[2] | Schnars U, Jüptner WPO. Digital recording and numerical reconstruction of holograms. Meas Sci Technol 13, R85–R101 (2002). doi: 10.1088/0957-0233/13/9/201 |
[3] | Garcia-Sucerquia J, Xu WB, Jericho SK et al. Digital in-line holographic microscopy. Appl Opt 45, 836–850 (2006). doi: 10.1364/AO.45.000836 |
[4] | Zhou J, Jin YB, Lu LP et al. Deep learning-enabled pixel-super-resolved quantitative phase microscopy from single-shot aliased intensity measurement. Laser Photonics Rev 18, 2300488 (2024). doi: 10.1002/lpor.202300488 |
[5] | de Almeida JL, Comunello E, Sobieranski A et al. Twin-image suppression in digital in-line holography based on wave-front filtering. Pattern Anal Appl 24, 907–914 (2021). doi: 10.1007/s10044-020-00949-7 |
[6] | Bai C, Peng T, Min JW et al. Dual-wavelength in-line digital holography with untrained deep neural networks. Photonics Res 9, 2501–2510 (2021). doi: 10.1364/PRJ.441054 |
[7] | Zhang JL, Sun JS, Chen Q et al. Adaptive pixel-super-resolved lensfree in-line digital holography for wide-field on-chip microscopy. Sci Rep 7, 11777 (2017). doi: 10.1038/s41598-017-11715-x |
[8] | Luo W, Zhang YB, Feizi A et al. Pixel super-resolution using wavelength scanning. Light Sci Appl 5, e16060 (2016). |
[9] | Pellizzari CJ, Spencer MF, Bouman CA. Coherent plug-and-play: digital holographic imaging through atmospheric turbulence using model-based iterative reconstruction and convolutional neural networks. IEEE Trans Comput Imag 6, 1607–1621 (2020). doi: 10.1109/TCI.2020.3042948 |
[10] | Chang XY, Bian LH, Gao YH et al. Plug-and-play pixel super-resolution phase retrieval for digital holography. Opt Lett 47, 2658–2661 (2022). doi: 10.1364/OL.458117 |
[11] | Bao P, Situ GH, Pedrini G et al. Lensless phase microscopy using phase retrieval with multiple illumination wavelengths. Appl Opt 51, 5486–5494 (2012). doi: 10.1364/AO.51.005486 |
[12] | Luo W, Greenbaum A, Zhang YB et al. Synthetic aperture-based on-chip microscopy. Light Sci Appl 4, e261 (2015). doi: 10.1038/lsa.2015.34 |
[13] | Yamaguchi I, Zhang T. Phase-shifting digital holography. Opt Lett 22, 1268–1270 (1997). doi: 10.1364/OL.22.001268 |
[14] | Song J, Swisher CL, Im H et al. Sparsity-based pixel super resolution for lens-free digital in-line holography. Sci Rep 6, 24681 (2016). doi: 10.1038/srep24681 |
[15] | Raupach SMF. Cascaded adaptive-mask algorithm for twin-image removal and its application to digital holograms of ice crystals. Appl Opt 48, 287–301 (2009). doi: 10.1364/AO.48.000287 |
[16] | Zhang WH, Cao LC, Brady DJ et al. Twin-image-free holography: A compressive sensing approach. Phys Rev Lett 121, 093902 (2018). doi: 10.1103/PhysRevLett.121.093902 |
[17] | Gao YH, Cao LC. Generalized optimization framework for pixel super-resolution imaging in digital holography. Opt Express 29, 28805–28823 (2021). doi: 10.1364/OE.434449 |
[18] | Wang H, Lyu M, Situ GH. eHoloNet: A learning-based end-to-end approach for in-line digital holographic reconstruction. Opt Express 26, 22603–22614 (2018). doi: 10.1364/OE.26.022603 |
[19] | Rivenson Y, Zhang YB, Günaydın H et al. Phase recovery and holographic image reconstruction using deep learning in neural networks. Light Sci Appl 7, 17141 (2018). |
[20] | Lempitsky V, Vedaldi A, Ulyanov D. Deep image prior. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition 9446–9454 (IEEE, 2018); http://doi.org/10.1109/CVPR.2018.00984. |
[21] | Wang F, Bian YM, Wang HC et al. Phase imaging with an untrained neural network. Light Sci Appl 9, 77 (2020). doi: 10.1038/s41377-020-0302-3 |
[22] | Han F, Mu TK, Li HY et al. Deep image prior plus sparsity prior: Toward single-shot full-stokes spectropolarimetric imaging with a multiple-order retarder. Adv Photonics 2, 036009 (2023). |
[23] | Galande AS, Thapa V, Gurram HPR et al. Untrained deep network powered with explicit denoiser for phase recovery in inline holography. Appl Phys Lett 122, 133701 (2023). doi: 10.1063/5.0144795 |
[24] | Niknam F, Qazvini H, Latifi H. Holographic optical field recovery using a regularized untrained deep decoder network. Sci Rep 11, 10903 (2021). doi: 10.1038/s41598-021-90312-5 |
[25] | Mait JN, Euliss GW, Athale RA. Computational imaging. Adv Opt Photonics 10, 409–483 (2018). doi: 10.1364/AOP.10.000409 |
[26] | Zhao WS, Zhao SQ, Li LJ et al. Sparse deconvolution improves the resolution of live-cell super-resolution fluorescence microscopy. Nat Biotechnol 40, 606–617 (2022). doi: 10.1038/s41587-021-01092-2 |
[27] | Zhao H, Gallo O, Frosio I et al. Loss functions for image restoration with neural networks. IEEE Trans Comput Imag 3, 47–57 (2017). doi: 10.1109/TCI.2016.2644865 |
[28] | Ravishankar S, Ye JC, Fessler JA. Image reconstruction: from sparsity to data-adaptive methods and machine learning. Proc IEEE 108, 86–109 (2020). doi: 10.1109/JPROC.2019.2936204 |
[29] | Ronneberger O, Fischer P, Brox T. U-net: convolutional networks for biomedical image segmentation. In Proceedings of the 18th International Conference on Medical Image Computing and Computer-Assisted Intervention 234–241 (Springer, 2015); http://doi.org/10.1007/978-3-319-24574-4_28. |
[30] | Schanz D, Gesemann S, Schröder A et al. Non-uniform optical transfer functions in particle imaging: Calibration and application to tomographic reconstruction. Meas Sci Technol 24, 024009 (2013). doi: 10.1088/0957-0233/24/2/024009 |
[31] | Bai C, Liu C, Jia H et al. Compressed blind deconvolution and denoising for complementary beam subtraction light-sheet fluorescence microscopy. IEEE Trans Biomed Eng 66, 2979–2989 (2019). doi: 10.1109/TBME.2019.2899583 |
[32] | Crete F, Dolmiere T, Ladret P et al. The blur effect: Perception and estimation with a new no-reference perceptual blur metric. In Proceedings of the SPIE 6492, Human Vision and Electronic Imaging XII 64920I (SPIE, 2007); http://doi.org/10.1117/12.702790. |
[33] | Polyanskiy MN. Refractiveindex. Info database of optical constants. Sci Data 11, 94 (2024). doi: 10.1038/s41597-023-02898-2 |
[34] | Luke SM, Vukusic P, Hallam B. Measuring and modelling optical scattering and the colour quality of white pierid butterfly scales. Opt Express 17, 14729–14743 (2009). doi: 10.1364/OE.17.014729 |
[35] | Zhang K, Liang JY, Van Gool L et al. Designing a practical degradation model for deep blind image super-resolution. In 2021 IEEE/CVF International Conference on Computer Vision (ICCV) 4771–4780 (IEEE, 2021); http://doi.org/10.1109/ICCV48922.2021.00475. |
An overview of the proposed reconstruction method. (a) Diagrams of classical solutions based on CS and TV regularization framework. (b) DIP-based reconstruction method combined with physical model. (c) Pipeline of MPPN-PSR for hologram reconstruction. A measured hologram I of a phase object Ψ is the input to the neural networks. The output of the neural networks is taken as the estimated phase
Schematic of the experimental setup of the DIHM.
Simulated phase objects (960 × 960 pixels) and corresponding single-shot DIHM imaging results (320 × 320 pixels), in which the spatial down-sampling rate was set as θ = 3. The red scale bar measures 500 μm.
(a) Phase retrieval results of different methods on simulated holograms (three times down-sampling) via back propagation, TwIST-TV-PSR, PnP-TV-PSR, PnP-FFDNet-PSR, PN-PSR, and MPPN-PSR, respectively. The autocorrelation functions of (b) resolution chart and (c) cell with different methods. The profiles along the blue lines are also investigated while the red scale bar measures 500 μm.
MPPN-PSR results of the holograms simulated with different down-sampling rates in terms of (a) resolution chart and (b) cell, respectively. (c) the examination of ratio β, which serves as a key proportional coefficient to balance the ℓ-1 norm and ℓ-2 norm terms. (d) The evolution of the MSE with an increasing number of epochs. The profiles along the blue lines are also investigated while the red scale bar measures 500 μm.
Reconstruction results of different methods at different noise levels, the red scale bar measures 500 μm. (a) Reconstruction results of resolution chart, the down-sampling rate θ = 2. (b) Reconstruction results of cell, the down-sampling rate θ = 3.
Quantitative analysis of the effect of noise on the reconstruction results using different methods. (a) and (b) are the trends of PSNR and SSIM indices of reconstruction results of resolution chart when the down-sampling rate θ = 2, while (c) and (d) are those of cell when the down-sampling rate θ = 3.
The practicality and persuasiveness of the MPPN-PSR method in real-world applications was evaluated by standard imaging and pixel binning modes of the camera. The pixel pitch varied from (a) 6.5 μm to (b) 13 μm, representing the resolution has been reduced to 1/4 of the original. The blue and green boxes respectively select the ROI of the hologram and results reconstructed by different methods.
Experimental images of the phase step (a1–a7) and PMMA beads (b1–b7) processed with back propagation, TwIST-TV-PSR, PnP-FFDNet-PSR, PN-PSR, and MPPN-PSR methods, respectively. The down-sampling rate is θ = 3, consistent with all subsequent experiments. The cross-section phase profiles (along the blue lines) were also measured in insets and the corresponding optical thickness maps are shown. The reconstruction size is 1200×1200 pixels, corresponding to the FOV of 130×130 µm2. The red scale bar measures 20 µm.
Imaging results of (a) butterfly wing and (b) fish ovary with different methods, including the reconstructed phase maps and the magnified views of selected regions. The reconstruction size is 1536 × 1536 pixels, i.e. θ = 3, corresponding to the FOV of 166×166 µm2. The red scale bar measures 25 µm.
Imaging results of (a) TOMM20 antibody and (b) frog intestine by different methods. The reconstruction size is 2700 × 2700 pixels, i.e. θ=3, corresponding to the FOV of 293×293 µm2. The red scale bar measures 40 µm.
The autocorrelation functions of (a) butterfly wing and (b) fish ovary with different methods.
The experimental result of MPPN-PSR to reconstruct the full-FOV high-resolution phase image of a quantitative phase target. (a) The full-FOV LR defocused hologram, and the phase image reconstructed by TwIST-TV-PSR and PN-PSR. (b) The full-FOV PSR phase image reconstructed by MPPN-PSR from a single frame of LR hologram. (c) The LR bright field images of three ROIs. (d) The phase images of the three ROIs reconstructed by MPPN-PSR, TwIST-TV-PSR and PN-PSR, respectively. (e) The results of cascading non-PSR reconstruction with outstanding pixel super-resolution networks, i.e. BSRGAN and BSRNet, have also been demonstrated, although this approach represents over-smooth and artifacts. The profiles along the blue and orange-colored lines are also investigated.
(a) The full-FOV PSR phase reconstruction of the TOMM20 antibody by MPPN-PSR, (b) and the comparison of PSR phase images of two ROIs. (c) The corresponding optical thickness maps are shown as well. (d) BSRGAN and BSRNet networks introduce unsatisfactory artifacts with non PSR reconstruction as well. The purple area represents the FOV that a 40× objective lens can bring.