E-mail Alert
RSS
| Citation: |
Li YX, Qian JM, Feng SJ, Chen Q, Zuo C. Deep-learning-enabled dual-frequency composite fringe projection profilometry for single-shot absolute 3D shape measurement. Opto-Electron Adv 5, 210021 (2022). doi: 10.29026/oea.2022.210021
|
Deep-learning-enabled dual-frequency composite fringe projection profilometry for single-shot absolute 3D shape measurement
-
Abstract
Single-shot high-speed 3D imaging is important for reconstructions of dynamic objects. For fringe projection profilometry (FPP), however, it is still challenging to recover accurate 3D shapes of isolated objects by a single fringe image. In this paper, we demonstrate that the deep neural networks can be trained to directly recover the absolute phase from a unique fringe image that involves spatially multiplexed fringe patterns of different frequencies. The extracted phase is free from spectrum-aliasing problem which is hard to avoid for traditional spatial-multiplexing methods. Experiments on both static and dynamic scenes show that the proposed approach is robust to object motion and can obtain high-quality 3D reconstructions of isolated objects within a single fringe image. -
-
References
[1] Gorthi SS, Rastogi P. Fringe projection techniques: whither we are. Opt Lasers Eng 48, 133–140 (2010). doi: 10.1016/j.optlaseng.2009.09.001 [2] Zhang ZH, Towers CE, Towers DP. Time efficient color fringe projection system for 3D shape and color using optimum 3-frequency selection. Opt Express 14, 6444–6455 (2006). doi: 10.1364/OE.14.006444 [3] Su XY, Zhang QC. Dynamic 3-D shape measurement method: a review. Opt Lasers Eng 48, 191–204 (2010). doi: 10.1016/j.optlaseng.2009.03.012 [4] Tao TY, Chen Q, Da J, Feng SJ, Hu Y et al. Real-time 3-D shape measurement with composite phase-shifting fringes and multi-view system. Opt Express 24, 20253–20269 (2016). doi: 10.1364/OE.24.020253 [5] Feng SJ, Zhang L, Zuo C, Tao TY, Chen Q et al. High dynamic range 3D measurements with fringe projection profilometry: a review. Meas Sci Technol 29, 122001 (2018). doi: 10.1088/1361-6501/aae4fb [6] Feng SJ, Zuo C, Tao TY, Hu Y, Zhang ML et al. Robust dynamic 3-D measurements with motion-compensated phase-shifting profilometry. Opt Lasers Eng 103, 127–138 (2018). doi: 10.1016/j.optlaseng.2017.12.001 [7] Pan B, Xie HM, Wang ZY, Qian KM, Wang ZY. Study on subset size selection in digital image correlation for speckle patterns. Opt Express 16, 7037–7048 (2008). doi: 10.1364/OE.16.007037 [8] Hu Y, Chen Q, Feng SJ, Zuo C. Microscopic fringe projection profilometry: a review. Opt Lasers Eng 135 (2020). doi: 10.1016/j.optlaseng.2020.106192 [9] Tao TY, Chen Q, Feng SJ, Qian JM, Hu Y et al. High-speed real-time 3D shape measurement based on adaptive depth constraint. Opt Express 26, 22440–22456 (2018). doi: 10.1364/OE.26.022440 [10] Qian JM, Feng SJ, Tao TY, Hu Y, Liu K et al. High-resolution real-time 360° 3D model reconstruction of a handheld object with fringe projection profilometry. Opt Lett 44, 5751–5754 (2019). doi: 10.1364/OL.44.005751 [11] Qian JM, Feng SJ, Xu MZ, Tao TY, Shang YH et al. High-resolution real-time 360° 3D surface defect inspection with fringe projection profilometry. Opt Lasers Eng 137, 106382 (2021). doi: 10.1016/j.optlaseng.2020.106382 [12] Zuo C, Chen Q, Gu GH, Feng SJ, Feng FXY et al. High-speed three-dimensional shape measurement for dynamic scenes using bi-frequency tripolar pulse-width-modulation fringe projection. Opt Lasers Eng 51, 953–960 (2013). doi: 10.1016/j.optlaseng.2013.02.012 [13] Heist S, Lutzke P, Schmidt I, Dietrich P, Kühmstedt P et al. High-speed three-dimensional shape measurement using GOBO projection. Opt Lasers Eng 87, 90–96 (2016). doi: 10.1016/j.optlaseng.2016.02.017 [14] Heist S, Kühmstedt P, Tünnermann A, Notni G. Theoretical considerations on aperiodic sinusoidal fringes in comparison to phase-shifted sinusoidal fringes for high-speed three-dimensional shape measurement. Appl Opt 54, 10541–10551 (2015). doi: 10.1364/AO.54.010541 [15] Takeda M, Mutoh K. Fourier transform profilometry for the automatic measurement of 3-D object shapes. Appl Opt 22, 3977–3982 (1983). doi: 10.1364/AO.22.003977 [16] Su XY, Chen WJ. Fourier transform profilometry: : a review. Opt Lasers Eng 35, 263–284 (2001). doi: 10.1016/S0143-8166(01)00023-9 [17] Kemao Q. Two-dimensional windowed Fourier transform for fringe pattern analysis: principles, applications and implementations. Opt Lasers Eng 45, 304–317 (2007). doi: 10.1016/j.optlaseng.2005.10.012 [18] Huang L, Kemao Q, Pan B, Asundi AK. Comparison of Fourier transform, windowed Fourier transform, and wavelet transform methods for phase extraction from a single fringe pattern in fringe projection profilometry. Opt Lasers Eng 48, 141–148 (2010). doi: 10.1016/j.optlaseng.2009.04.003 [19] Zhang ZH, Jing Z, Wang ZH, Kuang DF. Comparison of Fourier transform, windowed Fourier transform, and wavelet transform methods for phase calculation at discontinuities in fringe projection profilometry. Opt Lasers Eng 50, 1152–1160 (2012). doi: 10.1016/j.optlaseng.2012.03.004 [20] Zuo C, Feng SJ, Huang L, Tao TY, Yin W et al. Phase shifting algorithms for fringe projection profilometry: a review. Opt Lasers Eng 109, 23–59 (2018). doi: 10.1016/j.optlaseng.2018.04.019 [21] Pan B, Kemao Q, Huang L, Asundi A. Phase error analysis and compensation for nonsinusoidal waveforms in phase-shifting digital fringe projection profilometry. Opt Lett 34, 416–418 (2009). doi: 10.1364/OL.34.000416 [22] Zuo C, Huang L, Zhang ML, Chen Q, Asundi A. Temporal phase unwrapping algorithms for fringe projection profilometry: a comparative review. Opt Lasers Eng 85, 84–103 (2016). doi: 10.1016/j.optlaseng.2016.04.022 [23] Liu K, Wang YC, Lau DL, Hao Q, Hassebrook LG. Dual-frequency pattern scheme for high-speed 3-D shape measurement. Opt Express 18, 5229–5244 (2010). doi: 10.1364/OE.18.005229 [24] Zuo C, Tao TY, Feng SJ, Huang L, Asundi A et al. Micro Fourier transform profilometry (µftp): 3D shape measurement at 10, 000 frames per second. Opt Lasers Eng 102, 70–91 (2018). doi: 10.1016/j.optlaseng.2017.10.013 [25] Takeda M, Gu Q, Kinoshita M, Takai H, Takahashi Y. Frequency-multiplex Fourier-transform profilometry: a single-shot three- dimensional shape measurement of objects with large height discontinuities and/or surface isolations. Appl Opt 36, 5347–5354 (1997). doi: 10.1364/AO.36.005347 [26] Zhong JG, Zhang YL. Absolute phase-measurement technique based on number theory in multifrequency grating projection profilometry. Appl Opt 40, 492–500 (2001). doi: 10.1364/AO.40.000492 [27] Guan C, Hassebrook LG, Lau DL. Composite structured light pattern for three-dimensional video. Opt Express 11, 406–417 (2003). doi: 10.1364/OE.11.000406 [28] Sansoni G, Redaelli E. A 3D vision system based on one-shot projection and phase demodulation for fast profilometry. Meas Sci Technol 16, 1109–1118 (2005). doi: 10.1088/0957-0233/16/5/009 [29] Yue HM, Su XY, Liu YZ. Fourier transform profilometry based on composite structured light pattern. Opt Laser Technol 39, 1170–1175 (2007). doi: 10.1016/j.optlastec.2006.08.014 [30] Chen WJ, Su XY, Cao Y, Xiang LQ, Zhang QC. Fourier transform profilometry based on a fringe pattern with two frequency components. Optik-Int J Light Electron Opt 119, 57–62 (2008). doi: 10.1016/j.ijleo.2006.05.024 [31] Zhang ZH. Review of single-shot 3D shape measurement by phase calculation-based fringe projection techniques. Opt Lasers Eng 50, 1097–1106 (2012). doi: 10.1016/j.optlaseng.2012.01.007 [32] García-Isáis C, Ochoa NA. One shot profilometry using a composite fringe pattern. Opt Lasers Eng 53, 25–30 (2014). doi: 10.1016/j.optlaseng.2013.08.006 [33] Feng SJ, Chen Q, Gu GH, Tao TY, Zhang L et al. Fringe pattern analysis using deep learning. Adv Photonics 1, 025001 (2019). [34] Yin W, Chen Q, Feng SJ, Tao TY, Huang L et al. Temporal phase unwrapping using deep learning. Sci Rep 9, 20175 (2019). doi: 10.1038/s41598-019-56222-3 [35] van der Jeught S, Dirckx JJJ. Deep neural networks for single shot structured light profilometry. Opt Express 27, 17091–17101 (2019). doi: 10.1364/OE.27.017091 [36] Qian JM, Feng SJ, Tao TY, Hu Y, Li YX et al. Deep-learning-enabled geometric constraints and phase unwrapping for single-shot absolute 3D shape measurement. APL Photonics 5, 046105 (2020). doi: 10.1063/5.0003217 [37] Feng SJ, Zuo C, Yin W, Gu GH, Chen Q. Micro deep learning profilometry for high-speed 3D surface imaging. Opt Lasers Eng 121, 416–427 (2019). doi: 10.1016/j.optlaseng.2019.04.020 [38] Qian JM, Feng SJ, Li YX, Tao TY, Han J et al. Single-shot absolute 3D shape measurement with deep-learning-based color fringe projection profilometry. Opt Lett 45, 1842–1845 (2020). doi: 10.1364/OL.388994 [39] Shi JS, Zhu XJ, Wang HY, Song LM, Guo QH. Label enhanced and patch based deep learning for phase retrieval from single frame fringe pattern in fringe projection 3D measurement. Opt Express 27, 28929–28943 (2019). doi: 10.1364/OE.27.028929 [40] Nguyen H, Wang YZ, Wang ZY. Single-shot 3D shape reconstruction using structured light and deep convolutional neural networks. Sensors 20, 3718 (2020). [41] Zheng Y, Wang SD, Li Q, Li BW. Fringe projection profilometry by conducting deep learning from its digital twin. Opt Express 28, 36568–36583 (2020). doi: 10.1364/OE.410428 [42] Zhang S. Absolute phase retrieval methods for digital fringe projection profilometry: a review. Opt Lasers Eng 107, 28–37 (2018). doi: 10.1016/j.optlaseng.2018.03.003 [43] Ghiglia DC, Pritt MD. Two-Dimensional Phase Unwrapping: Theory, Algorithms, and Software (Wiley-Interscience, New York, 1998). [44] Chollet F. Deep Learning with Python (Manning Publications, Shelter Island, 2018). [45] Qian JM, Tao TX, Feng SJ, Chen Q, Zuo C. Motion-artifact-free dynamic 3D shape measurement with hybrid Fourier-transform phase- shifting profilometry. Opt Express 27, 2713–2731 (2019). doi: 10.1364/OE.27.002713 [46] Lilienblum E, Michaelis B. Optical 3D surface reconstruction by a multi-period phase shift method. J Comput 2, 73–83 (2007). [47] Pribanić T, Mrvoš S, Salvi J. Efficient multiple phase shift patterns for dense 3D acquisition in structured light scanning. Image Vis Comput 28, 1255–1266 (2010). doi: 10.1016/j.imavis.2010.01.003 [48] Ding Y, Xi JT, Yu YG, Chicharo J. Recovering the absolute phase maps of two fringe patterns with selected frequencies. Opt Lett 36, 2518–2520 (2011). doi: 10.1364/OL.36.002518 [49] Ding Y, Xi JT, Yu YG, Cheng WQ, Wang S et al. Frequency selection in absolute phase maps recovery with two frequency projection fringes. Opt Express 20, 13238–13251 (2012). doi: 10.1364/OE.20.013238 [50] Yin W, Zuo C, Feng SJ, Tao TY, Hu Y et al. High-speed three-dimensional shape measurement using geometry- constraint-based number-theoretical phase unwrapping. Opt Lasers Eng 115, 21–31 (2019). doi: 10.1016/j.optlaseng.2018.11.006 [51] Zhang Z. A flexible new technique for camera calibration. IEEE Trans Pattern Anal Mach Intell 22, 1330–1334 (2000). doi: 10.1109/34.888718 [52] Zhang S, Huang PS. Novel method for structured light system calibration. Opt Eng 45, 083601 (2006). doi: 10.1117/1.2336196 [53] Huang L, Zhang QC, Asundi A. Camera calibration with active phase target: improvement on feature detection and optimization. Opt Lett 38, 1446–1448 (2013). doi: 10.1364/OL.38.001446 -
Supplementary Information
Supplementary Information visualization 1 
-
Access History
Figures(11)
Tables(1)
Article Metrics
Export File
Citation
Li YX, Qian JM, Feng SJ, Chen Q, Zuo C. Deep-learning-enabled dual-frequency composite fringe projection profilometry for single-shot absolute 3D shape measurement. Opto-Electron Adv 5, 210021 (2022). doi: 10.29026/oea.2022.210021
Format
Content
DownLoad:
-
Figure 1.
The process of generating training data. (a) The projection mode includes dual-frequency 12-step phase-shifting fringe projection patterns and dual-frequency composite fringe pattern. (b) A set of captured images and the corresponding labels contain numerator term, denominator term, and absolute phase map.
-
Figure 2.
Flowchart of our proposed approach. (a) Part of network training data sets. (b) Hardware system and the cross-section intensity distribution of the designed composite fringe pattern. (c) Test data and prediction results obtained by the training model.
-
Figure 3.
The U-Net network architecture.
-
Figure 4.
Comparison between one U-Net network and two U-Net networks (the proposed method). (a, d, g, j) The raw composite fringe images from four different measurement scenes. (b, e, h, k) The absolute phase result error between deep-learning-predicted value and the ground-truth value by using one U-Net network. (c, f, i, l) The absolute phase result error between deep-learning-predicted value and the ground-truth value by using two U-Net networks.
-
Figure 5.
Part of input training datasets. The surface shapes contain single complex surface, geometric surface and discontinuous surface, and the materials include plaster, plastic, and paper.
-
Figure 6.
Four sets of network test inputs in static scenarios (a–d) and their corresponding spectral cross-sectional intensity distributions (e–h).
-
Figure 7.
The prediction results of our proposed method (DCFPP) in four static scenarios. (a, e, i, m) The numerator
$ {\text{M}_{\text{dl}}} $ $ {\text{D}_{\text{dl}}} $ $ {\Phi_{\text{dl}}} $ $ {\Phi _{\text{dl}}} $ -
Figure 8.
Phase error comparison results of traditional dual-frequency composite FT method and the proposed DCFPP method. (a, e, i, m) The wrapped phase error calculated by traditional method. (b, f, j, n) The wrapped phase error predicted by the DCFPP. (c, g, k, o) The absolute phase error of traditional method. (d, h, l, p) The absolute phase error of the DCFPP.
-
Figure 9.
3D reconstruction results of end-to-end network, the proposed DCFPP and 12-step PS with number-theoretic method in four measurement scenes. (a, d, g, j) 3D reconstruction results of the end-to-end network. (b, e, h, k) 3D reconstruction results of DCFPP. (c, f, i, l) 3D reconstruction results obtained by 12-step PS with number-theoretic method (ground-truth).
-
Figure 10.
Measurement results of a dynamic scene. (a, c) The captured composite fringe images at two different moments. (b, d) The corresponding 3D results reconstructed by DCFPP. (Multimedia view: see Supplementary information
Visualization 1 for the whole measurement process of the rotating plaster statue) -
Figure 11.
Precision analysis of standard ceramic plate and a standard ceramic sphere. (a, d) 3D reconstruction results by DCFPP. (b, e) Error distribution. (c, f) RMS error.
