E-mail Alert
RSS
| Citation: |
Liu KX, Wu JC, He ZH, Cao LC. 4K-DMDNet: diffraction model-driven network for 4K computer-generated holography. Opto-Electron Adv 6, 220135 (2023). doi: 10.29026/oea.2023.220135
|
Article Open Access
4K-DMDNet: diffraction model-driven network for 4K computer-generated holography
-
Abstract
Deep learning offers a novel opportunity to achieve both high-quality and high-speed computer-generated holography (CGH). Current data-driven deep learning algorithms face the challenge that the labeled training datasets limit the training performance and generalization. The model-driven deep learning introduces the diffraction model into the neural network. It eliminates the need for the labeled training dataset and has been extensively applied to hologram generation. However, the existing model-driven deep learning algorithms face the problem of insufficient constraints. In this study, we propose a model-driven neural network capable of high-fidelity 4K computer-generated hologram generation, called 4K Diffraction Model-driven Network (4K-DMDNet). The constraint of the reconstructed images in the frequency domain is strengthened. And a network structure that combines the residual method and sub-pixel convolution method is built, which effectively enhances the fitting ability of the network for inverse problems. The generalization of the 4K-DMDNet is demonstrated with binary, grayscale and 3D images. High-quality full-color optical reconstructions of the 4K holograms have been achieved at the wavelengths of 450 nm, 520 nm, and 638 nm. -
-
References
[1] Shi L, Li BC, Kim C, Kellnhofer P, Matusik W. Towards real-time photorealistic 3D holography with deep neural networks. Nature 591, 234–239 (2021). doi: 10.1038/s41586-020-03152-0 [2] Zhang CL, Zhang DF, Bian ZP. Dynamic full-color digital holographic 3D display on single DMD. Opto-Electron Adv 4, 200049 (2021). doi: 10.29026/oea.2021.200049 [3] He ZH, Sui XM, Jin GF, Cao LC. Progress in virtual reality and augmented reality based on holographic display. Appl Opt 58, A74–A81 (2019). doi: 10.1364/AO.58.000A74 [4] Zhao Y, Cao LC, Zhang H, Kong DZ, Jin GF. Accurate calculation of computer-generated holograms using angular-spectrum layer-oriented method. Opt Express 23, 25440–25449 (2015). doi: 10.1364/OE.23.025440 [5] Jiang Q, Jin GF, Cao LC. When metasurface meets hologram: principle and advances. Adv Opt Photonics 11, 518–576 (2019). doi: 10.1364/AOP.11.000518 [6] Huang LL, Chen XZ, Mühlenbernd H, Zhang H, Chen SM et al. Three-dimensional optical holography using a plasmonic metasurface. Nat Commun 4, 2808 (2013). doi: 10.1038/ncomms3808 [7] Guo JY, Wang T, Quan BG, Zhao H, Gu CZ et al. Polarization multiplexing for double images display. Opto-Electron Adv 2, 180029 (2019). doi: 10.29026/oea.2019.180029 [8] Gao H, Fan XH, Xiong W, Hong MH. Recent advances in optical dynamic meta-holography. Opto-Electron Adv 4, 210030 (2021). doi: 10.29026/oea.2021.210030 [9] Saha SK, Wang D, Nguyen VH, Chang Y, Oakdale JS et al. Scalable submicrometer additive manufacturing. Science 366, 105–109 (2019). doi: 10.1126/science.aax8760 [10] Gerchberg RW, Saxton WOA. A practical algorithm for the determination of phase from image and diffraction plane pictures. Optik 35, 237–246 (1972). [11] Tian SZ, Chen LZ and Zhang H. Optimized Fresnel phase hologram for ringing artifacts removal in lensless holographic projection. Appl Opt 61, B17–B24 (2022). [12] Chakravarthula P, Peng YF, Kollin J, Fuchs H, Heide F. Wirtinger holography for near-eye displays. ACM Trans Graph 38, 213 (2019). doi: 10.1145/3355089.3356539 [13] Zhang JZ, Pégard N, Zhong JS, Adesnik H, Waller L. 3D computer-generated holography by non-convex optimization. Optica 4, 1306–1313 (2017). doi: 10.1364/OPTICA.4.001306 [14] He KM, Zhang XY, Ren SQ, Sun J. Deep residual learning for image recognition. In Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition 770–778 (IEEE, 2016);http://doi.org/10.1109/CVPR.2016.90. [15] Liao MH, Zheng SS, Pan SX, Lu DJ, He WQ et al. Deep-learning-based ciphertext-only attack on optical double random phase encryption. Opto-Electron Adv 4, 200016 (2021). doi: 10.29026/oea.2021.200016 [16] Li YX, Qian JM, Feng SJ, Chen Q, Zuo C. Deep-learning-enabled dual-frequency composite fringe projection profilometry for single-shot absolute 3D shape measurement. Opto-Electron Adv 5, 210021 (2022). doi: 10.29026/oea.2022.210021 [17] Blinder D, Birnbaum T, Ito T, Shimobaba T. The state-of-the-art in computer generated holography for 3D display. Light Adv Manuf 3, 35 (2022). doi: 10.37188/lam.2022.035 [18] He ZH, Sui XM, Jin GF, Chu DP, Cao LC. Optimal quantization for amplitude and phase in computer-generated holography. Opt Express 29, 119–133 (2021). doi: 10.1364/OE.414160 [19] Sui XM, He ZH, Jin GF, Chu DP, Cao LC. Band-limited double-phase method for enhancing image sharpness in complex modulated computer-generated holograms. Opt Express 29, 2597–2612 (2021). doi: 10.1364/OE.414299 [20] Liu KX, He ZH, Cao LC. Double amplitude freedom Gerchberg–Saxton algorithm for generation of phase-only hologram with speckle suppression. Appl Phys Lett 120, 061103 (2022). doi: 10.1063/5.0080797 [21] Liu KX, He ZH, Cao LC. Pattern-adaptive error diffusion algorithm for improved phase-only hologram generation. Chin Opt Lett 19, 050501 (2021). doi: 10.3788/COL202119.050501 [22] Kang JW, Park BS, Kim JK, Kim DW, Seo YH. Deep-learning-based hologram generation using a generative model. Appl Opt 60, 7391–7399 (2021). doi: 10.1364/AO.427262 [23] Lee J, Jeong J, Cho J, Yoo D, Lee B et al. Deep neural network for multi-depth hologram generation and its training strategy. Opt Express 28, 27137–27154 (2020). doi: 10.1364/OE.402317 [24] Zheng HD, Hu JB, Zhou CJ, Wang XX. Computing 3D phase-type holograms based on deep learning method. Photonics 8, 280 (2021). doi: 10.3390/photonics8070280 [25] Liu SC, Chu DP. Deep learning for hologram generation. Opt Express 29, 27373–27395 (2021). doi: 10.1364/OE.418803 [26] Khan A, Zhang ZJ, Yu YJ, Khan MA, Yan KT et al. GAN-Holo: generative adversarial networks-based generated holography using deep learning. Complexity 2021, 6662161 (2021). doi: 10.1155/2021/6662161 [27] Horisaki R, Takagi R, Tanida J. Deep-learning-generated holography. Appl Opt 57, 3859–3863 (2018). doi: 10.1364/AO.57.003859 [28] Goi H, Komuro K, Nomura T. Deep-learning-based binary hologram. Appl Opt 59, 7103–7108 (2020). doi: 10.1364/AO.393500 [29] Chang CL, Wang D, Zhu DC, Li JM, Xia J et al. Deep-learning-based computer-generated hologram from a stereo image pair. Opt Lett 47, 1482–1485 (2022). doi: 10.1364/OL.453580 [30] Hossein Eybposh M, Caira NW, Atisa M, Chakravarthula P, Pégard NC. DeepCGH: 3D computer-generated holography using deep learning. Opt Express 28, 26636–26650 (2020). doi: 10.1364/OE.399624 [31] Horisaki R, Nishizaki Y, Kitaguchi K, Saito M, Tanida J. Three-dimensional deeply generated holography [Invited]. Appl Opt 60, A323–A328 (2021). doi: 10.1364/AO.404151 [32] Peng YF, Choi S, Padmanaban N, Wetzstein G. Neural holography with camera-in-the-loop training. ACM Trans Graph 39, 185 (2020). doi: 10.1145/3414685.3417802 [33] Gopakumar M, Kim J, Choi S, Peng YF, Wetzstein G. Unfiltered holography: optimizing high diffraction orders without optical filtering for compact holographic displays. Opt Lett 46, 5822–5825 (2021). doi: 10.1364/OL.442851 [34] Peng YF, Choi S, Kim J, Wetzstein G. Speckle-free holography with partially coherent light sources and camera-in-the-loop calibration. Sci Adv 7, eabg5040 (2021). doi: 10.1126/sciadv.abg5040 [35] Ishii Y, Shimobaba T, Blinder D, Birnbaum T, Schelkens P et al. Optimization of phase-only holograms calculated with scaled diffraction calculation through deep neural networks. Appl Phys B 128, 22 (2022). doi: 10.1007/s00340-022-07753-7 [36] Yu T, Zhang SJ, Chen W, Liu J, Zhang XY et al. Phase dual-resolution networks for a computer-generated hologram. Opt Express 30, 2378–2389 (2022). doi: 10.1364/OE.448996 [37] Sun XH, Mu XY, Xu C, Pang H, Deng QL et al. Dual-task convolutional neural network based on the combination of the U-Net and a diffraction propagation model for phase hologram design with suppressed speckle noise. Opt Express 30, 2646–2658 (2022). doi: 10.1364/OE.440956 [38] Wu JC, Liu KX, Sui XM, Cao LC. High-speed computer-generated holography using an autoencoder-based deep neural network. Opt Lett 46, 2908–2911 (2021). doi: 10.1364/OL.425485 [39] Situ GH. Deep holography. Light Adv Manuf 3, 13 (2022). doi: 10.37188/lam.2022.013 [40] Zuo C, Qian JM, Feng SJ, Yin W, Li YX et al. Deep learning in optical metrology: a review. Light Sci Appl 11, 39 (2022). doi: 10.1038/s41377-022-00714-x [41] Shi L, Li BC, Matusik W. End-to-end learning of 3D phase-only holograms for holographic display. Light Sci Appl 11, 247 (2022). doi: 10.1038/s41377-022-00894-6 [42] Ronneberger O, Fischer P, Brox T. U-Net: convolutional networks for biomedical image segmentation. In Proceedings of the 18th International Conference on Medical Image Computing and Computer-Assisted Intervention 234–241 (Springer, 2015);http://doi.org/10.1007/978-3-319-24574-4_28. [43] Shi WZ, Caballero J, Huszár F, Totz J, Aitken AP et al. Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition 1874–1883 (IEEE, 2016); http://doi.org/10.1109/CVPR.2016.207. [44] Dumoulin V, Shlens J, Kudlur M. A learned representation for artistic style. In Proceedings of the 5th International Conference on Learning Representations (IEEE, 2016). https://arxiv.org/abs/1610.07629 [45] Shimobaba T, Blinder D, Birnbaum T, Hoshi I, Shiomi H et al. Deep-learning computational holography: a review. Front Photonics 3, 854391 (2022). doi: 10.3389/fphot.2022.854391 [46] Kingma DP, Ba J. Adam: a method for stochastic optimization. In Proceedings of the 3rd International Conference on Learning Representations (2014). https://arxiv.org/abs/1412.6980 [47] Source code: https://github.com/THUHoloLab/4K-DMDNet [48] Wang JD, Sun K, Cheng TH, Jiang BR, Deng CR et al. Deep high-resolution representation learning for visual recognition. IEEE Trans Pattern Anal Mach Intell 43, 3349–3364 (2021). doi: 10.1109/TPAMI.2020.2983686 -
Supplementary Information
4K-DMDNet code 
-
Access History
Figures(9)
Article Metrics
Export File
Citation
Liu KX, Wu JC, He ZH, Cao LC. 4K-DMDNet: diffraction model-driven network for 4K computer-generated holography. Opto-Electron Adv 6, 220135 (2023). doi: 10.29026/oea.2023.220135
Format
Content
DownLoad:
-
Figure 1.
Training processes of (a) data-driven deep learning and (b) 4K-DMDNet, respectively.
-
Figure 2.
Generation and reconstruction process of 4K POHs by the 4K-DMDNet. The sub-pixel convolution method and oversampling method have played decisive roles to achieve it.
-
Figure 3.
(a) U-Net neural network architecture of 4K-DMDNet. (b) Upsampling block architecture. The figures between the brackets present the kernel size and the stride of the convolutional layer, respectively.
-
Figure 4.
(a–c) Schematic diagram of the transposed convolution, NN-resize convolution, and sub-pixel convolution with their corresponding numerical simulations.
-
Figure 5.
(a) Schematic diagram of the Fresnel diffraction model. (b) Fresnel diffraction model with oversampling method realized in the neural network layer manner. (c) Comparison between the numerical simulation and optical reconstruction with the undersampling problem. (d) Schematic diagram of the oversampling method.
-
Figure 6.
Contrast between the numerical simulations of POHs by (a) the GS algorithm, (b) Holo-Encoder, and (c) 4K-DMDNet. (d) Evaluation of algorithm runtime and image quality. The length of the bar represents the standard deviation of 100 samples (DIV2K_valid_HR).
-
Figure 7.
(a) Photograph of the experimental setup. (b) Schematic diagram of the time multiplexing method for full-color display. (c) Full-color 4K optical reconstruction by 4K-DMDNet and its detail views. (d) Optical reconstruction by GS algorithm. (e) Optical reconstruction by Holo-Encoder.
-
Figure 8.
(a) Object, (b) POH, and (c) optical reconstruction of the binary target.
-
Figure 9.
(a) All-in-focus image of the 3D scene. (b) Depth map of the 3D scene. (c) (d) and (e) Optical reconstructions of the 4K-DMDNet for 3D scene at the 28 cm, 30 cm and 32 cm, respectively. The enlarged views are presented at the right side.
