Physics-informed deep learning for fringe pattern analysis

Recently, deep learning has yielded transformative success across optics and photonics, especially in optical metrology. Deep neural networks (DNNs) with a fully convolutional architecture (e.g., U-Net and its derivatives) have been widely implemented in an end-to-end manner to accomplish various optical metrology tasks, such as fringe denoising, phase unwrapping, and fringe analysis. However, the task of training a DNN to accurately identify an image-to-image transform from massive input and output data pairs seems at best naïve, as the physical laws governing the image formation or other domain expertise pertaining to the measurement have not yet been fully exploited in current deep learning practice. To this end, we introduce a physics-informed deep learning method for fringe pattern analysis (PI-FPA) to overcome this limit by integrating a lightweight DNN with a learning-enhanced Fourier transform profilometry (LeFTP) module. By para-meterizing conventional phase retrieval methods, the LeFTP module embeds the prior knowledge in the network structure and the loss function to directly provide reliable phase results for new types of samples, while circumventing the requirement of collecting a large amount of high-quality data in supervised learning methods. Guided by the initial phase from LeFTP, the phase recovery ability of the lightweight DNN is enhanced to further improve the phase accuracy at a low computational cost compared with existing end-to-end networks. Experimental results demonstrate that PI-FPA enables more accurate and computationally efficient single-shot phase retrieval, exhibiting its excellent generalization to various unseen objects during training. The proposed PI-FPA presents that challenging issues in optical metrology can be potentially overcome through the synergy of physics-priors-based traditional tools and data-driven learning approaches, opening new avenues to achieve fast and accurate single-shot 3D imaging


Introduction
Optical metrology, as a general-purpose metrology technique that uses light as information carriers for non-contact and non-destructive measurement 1 , is fundamental to manufacturing, basic research, and engineering ap-plications.With the invention of the laser 2 and chargecoupled device (CCD) 3 , many optical metrology methods and instruments are employed in state-of-the-art manufacturing processes, precision positioning, and quality assessment because of their advantages in terms of accuracy, sensitivity, repeatability, and speed.In optical metrology, based on physical models of the image formation, the observed measurements (e.g., deformed fringe/speckle images) can be transformed into the desired physical properties of the objects (the profile, distance, strain, etc.).For many optical measurement techniques such as interferometry 4 , digital holography 5 , and fringe projection profilometry (FPP) 6,7 , the accuracy and efficiency of phase retrieval from the recorded fringe images are essential to reconstruct various underlying quantities dynamically.The most efficient method for phase measurement is recovering the phase distribution from a single fringe image, but as a typical case in optical metrology, it is an ill-posed inverse problem.The spatial phase-demodulation (SPD) methods can achieve singleframe fringe analysis by imposing some prior assumptions on the recovered phase (spatially smooth, limited spectral extension, piecewise constant, etc.) 8−10 , but at the cost of accuracy and resolution.Since optical metrology experiments are generally carried out in highly customized systems and stringent environments, phase-shifting (PS) methods can provide a deterministic and straightforward solution to the phase retrieval problem by additionally capturing multiple fringe patterns 11 .PS methods have obvious advantages in terms of speed, accuracy, and repeatability, which have brought up many high-end optical metrology instruments.However, when the optical system is under harsh measurement conditions or the state of the object changes dynamically, PS methods will be severely limited and cannot provide accurate phase recovery results for dynamic measurements.Despite extensive research efforts for decades, how to achieve phase measurement with the highest possible accuracy from the minimum number (preferably single shot) of fringe patterns remains one of the most challenging problems in optical metrology.
With the explosive growth of available data and computing resources, deep learning, as a "data-driven " machine learning technique, has achieved impressive success in numerous fields, such as computer vision and computational imaging 12 .Deep learning pervades almost all aspects of optical metrology 13 , and provides solutions to many challenging problems, such as fringe denoising 14,15 , fringe analysis 16 , and digital holographic reconstruction 17−19 .Feng et al. 16 proposed a deep learning method for fringe pattern analysis that establishes an inverse mapping between single-frame fringe and the label phase obtained using 12-step PS method.The trained network can directly estimate the sine and cosine components of fringes, enabling single-shot phase reconstruction with higher accuracy than SPD methods.Recently, phase retrieval methods based on deep learning have been applied to ultrafast 3D imaging (speed up to 20 kHz) 20 , phase measuring deflectometry 21 , and singleframe absolute 3D measurement 22 by adopting diverse deep neural networks (DNNs) with a fully convolutional architecture 23,24 or combining the predictions of multiple networks with ensemble learning 25 .However, these deep learning approaches focus mainly on training a DNN to accurately identify an image-to-image transform from massive input and output data pairs of training datasets without considering the physical laws governing the image formation or other domain expertise pertaining to the measurement.Consequently, the performance of deep learning approaches in solving complex physical problems relies heavily on the underlying statistical characteristics within the dataset.To improve the performance of the network under real experimental conditions, it is necessary to pay a high price for collecting a large amount of high-quality data.In addition, due to the highly customized nature of optical metrology systems, networks trained on one system may not be directly transferable to another system of the same type.Once the new input is different even slightly from the training data, data-driven DNNs may exhibit a poor generalization under diverse measurement conditions, and cannot ensure the interpretability and traceability of their output results.On the contrary, based on accurate physical models of the image formation and its inverse solutions, traditional SPD methods can achieve reliable phase measurements for different types of samples 26 , but their measurement precision is limited.If the forward physical models of the image formation or traditional solvers of the inverse problem are incorporated into the DNN, it is expected to enhance the performance of deep learning methods while utilizing fewer network parameters.Goy et al. 27 proposed a physics-informed deep learning method for phase retrieval at low photon counts that leverages physical priors to convert the raw intensity measurement with noise into an initial estimate of the object, thereby significantly improving the phase reconstruction accuracy by using deep learning.Wang et al. 28 demonstrated an unsupervised single-beam phase imaging network to reconstruct the phase of the measured diffraction pattern by integrating a numerically propagated diffraction model.Saba et al. 29 proposed a physics-informed neural network for tomographic reconstructions of biological samples, which minimizes the physical loss based on the Helmholtz equation, accurately and quickly retrieving the refractive index distribution from the scattered fields of the sample collected by different illumination directions.
For the limited ability of fringe analysis networks without physics priors, we present a physics-informed deep learning method for fringe pattern analysis (PI-FPA).A learning-enhanced Fourier transform profilometry (LeFTP) module with the prior knowledge of SPD methods is embedded in the DNN to directly provide accurate and reliable phase recovery results for new types of samples, while circumventing the requirement of collecting a large amount of high-quality data in supervised learning methods.The phase results are then refined using a lightweight DNN to further improve the accuracy and computational efficiency of single-shot phase retrieval.Experimental results show that the proposed PI-FPA exhibits superior single-shot fringe analysis performance in speed, accuracy, repeatability, and generalization to various unseen objects during training.

Principle
Phase retrieval from fringe images

I(x, y)
Phase retrieval from fringe images is a fundamental task and a representative case among many applications of deep learning in optical metrology.The fringe image is expressed as 30,31 I(x, y) = A(x, y) + B(x, y)cos[ϕ(x, y)] , (1) where and are the background intensity and the fringe amplitude, and is the phase of the tested object.Retrieving the desired from only one fringe image is an ill-posed inverse problem due to two unknown parts and .In FPP, PS methods 11 transform the original ill-posed problem into a well-posed and solvable one by projecting a set of PS patterns to obtain additional observations of the target object: I n (x, y) N ϕ(x, y) where represents -step PS images, can be obtained by the least-squares algorithm.However, when the measured object is under harsh measurement conditions, the relative motion between the object and PS pat- terns will introduce non-negligible errors into phase retrieval results 32,33 .Unlike PS methods, SPD methods can realize single-shot phase retrieval using different spatial transform techniques (such as the Fourier transform (FT) 9 and the windowed Fourier transform 10 ) under the local smoothness assumption.In Fourier transform profilometry (FTP), the Fourier transform of in Eq. (1) gives where and are the Fourier transform of and . is taken as the sum of two independent parts: the object component and the carrier frequency .Based on the Fourier shift theorem, the zero order is separated with ±1 orders and , so can be extracted by a band-pass filter and converted inversely to the retrieved phase, However, when the measured surface contains sharp edges or discontinuities, the support of the zero order and ±1 orders will be extended to cause the spectrum overlapping, precluding high-accuracy phase measurement of complex objects.

I(x, y)
Unlike traditional methods that focus on understanding the image formation and solving inverse problems, Feng et al. 16 utilized DNNs to directly estimate the sine and cosine components of for single-shot fringe analysis: where is a constant that depends on phase retrieval methods, e.g., for FT methods and for -step PS methods.However, the performance of phase retrieval networks relies heavily on a large amount of high-quality data.Once the new input is different from the training data, the reliability of phase reconstruction results output by data-driven DNNs cannot be guaranteed.

Physics-informed deep learning method for fringe pattern analysis (PI-FPA)
As shown in Fig. 1, different from traditional physicsdriven methods (FT methods) and data-driven deep learning approaches (e.g., U-Net and its derivatives) for fringe pattern analysis, the proposed PI-FPA mainly contains a LeFTP module with physics priors and a lightweight network.The LeFTP module, which parameterizes the phase retrieval process of FT methods, utilizes the learnable filters operating in the Fourier transform domain to directly output initial phases in the manner of FTP in Fig. 1(a, b).Physics-driven LeFTP is highly generalizable to provide reliable phase results for various unseen objects during training.The lightweight network refines the initial phase to further improve the phase accuracy at a low computational cost, compared with universal end-to-end image transform networks (U-Net and its derivatives).
The schematic diagram of the proposed PI-FPA is shown in Fig. 2. First, a net head with a simple convolutional structure is adopted to extract rich low-level features of the input single-frame fringe, which can reduce the effect of the zero order of the fringe after training.For the LeFTP module in Fig. 2(c), similar to traditional FT methods, the input tensor is transformed into the Fourier domain through Fourier transform and spectrum centering.Instead of the simple filtering operation of FTP, two learnable filters with multiple channels are utilized to adaptively extract the order closely related to the desired phase.Specifically, a learnable filter with size is applied to weaken the zero order located at the center C1 by weighting each feature of the input spectrum pixel by pixel: • where is the Hadamard product.Note that the un- filtered high-frequency component is kept to avoid missing details, and the redundant negative Fourier spectrum is removed.Then, a series of filtering operations are implemented to extract delicately the +1 order in various ways using another learnable filter with size: where the center of is set as C2 estimated by -step PS.Due to the asymmetry of the spectrum, a large number of reliable and initial phases can be recovered inversely from the filtered spectrum according to Eq. ( 6).Further, to optimize the phase retrieval performance of LeFTP, a priors-based initialization strategy for the filter weights is adopted to facilitate its efficient learning and avoid anchoring in local minima during the training phase by following background-normalized Fourier transform profilometry (BNFTP) 34 .The filter is initialized as an inverse Hanning window for filtering the zero-order component of the input spectrum centered on C1: In addition, the +1 order of the spectrum centered on C2 is strengthened using another Hanning filter : At present, mainstream fringe analysis approaches using deep learning exploit end-to-end fully convolutional networks in a naïve manner to build an image-to-image Physics -driven method  inverse mapping between single-frame fringe and the label phase using massive network parameters.Thanks to robust phase estimation of LeFTP, it not only helps PI-FPA to circumvent the requirement of collecting a large amount of high-quality data in supervised learning methods, but also relieves the burden of phase refinement for lightweight DNNs.The lightweight network, consisting of the context path and the spatial path inspired by BiSeNet 35,36 in Fig. 2(a) (see Supplementary information for detailed analysis), is utilized to further improve the phase accuracy at a low computational cost compared with universal end-to-end image transform networks (U-Net and its derivatives).Instead of configuring more channels for higher-level layers as U-Net, the context path aims at collecting the fringe and initial phase features with a large receptive field through fast downsampling and encoding global context information to guide the refined high-level features for learning, while the spatial path captures spatial information encoding rich detail information and outputs low-level features.In the encoder part of the context path, a fast downsampling strategy with several ConvX blocks and the Short-Term Dense Concatenate (STDC) module is used to extract the feature information with scalable receptive field and multi-scale information.In the decoder phase, the attention-based feature refinement (AFR) module and the fast upsampling based on bilinear interpolation are utilized to improve the feature resolution progressively.In the spatial path, its encoder part shares the same parameters with the context path, and captures the spatial information encoding rich detailed information and outputs low-level features.The features from the context path and the spatial path are concatenated by Feature Fusion module (FFM), and upsampled to output final phases using the predicted and in Eq. (7).The objective of PI-FPA is to minimize the joint loss of the phase and its Fourier domain: where is the ground truth obtained using -step PS, is the network's output, is the LeFTP module's output, and , , and are the 2D Discrete Fourier Transform of , , and .

Experiments
In order to verify the proposed PI-FPA under the First, a David plaster was measured to reveal singleshot phase retrieval process of PI-FPA, and FTP, LeFTP, Net head + LeFTP, and U-Net were implemented for comparison.In Fig. 3(a, b), LeFTP makes use of two learnable filters operating in the Fourier domain and re-duces the MAE of phase errors by about 18% compared with FTP.By visualizing the filter weights, it demonstrates that LeFTP facilitates adaptive spectrum extraction through learning-enhanced filtering, which provides an interpretable guide for parameter optimization of FTP to improve the phase accuracy.In addition, due to the removal of redundant negative Fourier spectra in LeFTP, the left half of the filter weights is the same as its initial state, which is not updated during network training.To further speed up the LeFTP module, it is optional to cut down the size of two learnable filters in half to reduce the total parameters of the network and improve the inference speed of the network.Further, the Net head in Fig. 3(c), taken as the filtering operation in image pre-processing, is embedded in the front of LeFTP to extract rich low-level fringe features for removing the zero order, further reducing the phase errors by about 40%.It proves that LeFTP is plug-and-play to significantly boost the performance of single-frame fringe pattern analysis.
Different from these methods above, U-Net automatically exploits massive low-level and high-level features to optimize the phase accuracy as shown in Fig. 3(d), but at the cost of computational overhead.Specifically, U-Net needs 3.5 GB of GPU memory to process single- To verify the generalization of PI-FPA for complex surfaces, we tested an industrial part, and fringe analysis results using different methods show that the phase er-rors are smaller in smooth cylindrical regions but larger in sharp edges, while PI-FPA brings better phase quality among these methods as shown in Fig. 4(a, b).Further, we adopted stereo phase unwrapping 37 to achieve singleshot 3D imaging in Fig. 4(c) (see Supplementary information for detailed analysis).As the magnified regions in Fig. 4(d), the screw thread of the workpiece, which is relatively rare in the training dataset, causes significant degradation in the performance of U-Net, precluding highprecision reconstruction of complex surfaces.The line profiles in Fig. 4(e) prove that the proposed PI-FPA with physics-driven LeFTP can successfully recover the fine profiles of the threads and provide accurate and physically consistent 3D imaging results to approach the ground truth (GT), even though the network has not seen such experimental data during the training phase.
To quantitatively analyze the 3D imaging accuracy of PI-FPA, our system was applied to measuring a dynamic scene at the camera speed of 100 Hz: a ceramic plane and In Fig. 5(d, e), we further perform temporal precision analysis by collecting long-term data over a 1.62 s period using 3-step PS, FTP, U-Net, and PI-FPA.In Table 1, quantitative analysis results of the moving plane and sphere for different methods show that the measured results obtained by PI-FPA exhibited higher 3D reconstruction accuracy with a lower temporal standard deviation (STD) of 43 4.1 μm and 47 5.1 μm.In Fig. 5(f-i), we additionally provide the measurement results of the moving plane and sphere at T = 0.81 s using different methods.Different from FTP for singleshot phase retrieval, PS methods can realize pixel-bypixel phase measurements with higher accuracy for complex shapes, but it needs to project at least three fringe patterns to obtain a phase map theoretically.As the most common and efficient case in N-step PS methods, 3-step PS is implemented for comparison.When dynamic scenes are measured, the relative motion between the object and the phase-shifting fringe patterns sequentially projected will cause motion artifacts and thus introduce non-negligible phase errors into the phase map.As a consequence, there are severe measurement errors with the RMS of 196.101 μm and 179.681 μm in the measurement results of 3-step PS in Fig. 5(f).In addition, for real-time 3D measurement based on 3-step PS, the whole procedure of 3D reconstruction is composed of phase retrieval, stereo phase unwrapping, and phase-to-height mapping, which is implemented with a graphics processing unit (GPU) 38 and several look-up tables 39 to speed up the 3D reconstruction.The 3D imaging speed is determined by the maximum between the image acquisition time and the runtime of 3D reconstruction.The 640 × 480 runtime of stereo phase unwrapping 37 and phase-toheight mapping for processing the images with the resolution of pixels is less than 5 ms on RTX2080Ti.Since 3-step PS needs to capture three fringe images and its runtime of the phase retrieval is negligible (5.22  ms) in Table 1, its 3D imaging speed is limited to 33.33 FPS.On the contrary, the single-frame fringe analysis capability of FTP can significantly improve the accuracy and repeatability of fast 3D measurement to reduce the RMS to 75.417 μm and 71.715 μm, while its runtime (2.06 ms) promotes the speed of 3D measurement to 100 FPS in Fig. 5(g).This result proves that single-frame fringe analysis methods are more suitable for dynamic scene measurement when the target's movement speed is in the same order of magnitude as the 3D imaging speed.Then, in Fig. 5(h), the RMS of the measurement error can be further decreased to 53.361 μm and 60.129 μm thanks to the powerful feature extraction capability of U-Net, but at the cost of lower inference speed (65.02 ms), precluding real-time 3D measurement.Finally, benefiting from the proposed LeFTP module and the lightweight DNN, PI-FPA takes a runtime of 18.78 ms to achieve fast single-shot phase reconstruction with higher accuracy in Fig. 5(i).However, PI-FPA only retrieves the phase of the first in the threestep PS images and reduces the 3D imaging speed to 33.33 FPS.3D measurement results in Fig. 5 confirm that PI-FPA, whether measuring the moving plane or sphere, achieves successfully single-shot 3D shape measurement with higher accuracy and good repeatability for multiple moving objects simultaneously.The whole 3D measurement results can refer to Supplementary Video S1.
Last, to further demonstrate the advantages of PI-FPA, we applied our single-shot 3D imaging system to 360-degree reconstruction of a workpiece model and non-rigid dynamic face measurement as shown in Fig. 6 and Supplementary Video S2-S3.Fig. 6(a, b) show the captured fringe images of the rotated workpiece and non-rigid dynamic face at different time points and the corresponding color-coded 3D reconstruction results using different   methods.For the rotated workpiece, the highlighted regions in Fig. 6(a) show that 3-step PS cannot recover the fine shapes of smooth surfaces due to the phase errors introduced by motion artifacts.For single-frame fringe analysis, FTP is suitable for dynamic 3D measurement, but yields coarse 3D results with low quality in terms of accuracy and resolution due to the spectrum overlapping.U-Net can further improve the quality of 3D reconstruction, but it cannot reliably retrieve the phase of the object with metal materials which is relatively rare in the training dataset, precluding the recovery of fine surfaces.This experiment demonstrates that the proposed PI-FPA can be applied for high-quality and efficient 3D modeling of complex structure parts as shown in Fig. 6(c).Similarly, for non-rigid dynamic face, there are inevitably a large amount of ripple-like measurement errors in 3D results of 3-step PS due to motion artifacts.And then, FTP is performed to significantly reduce measurement errors, but is unable to recover high-quality local details of the face.Due to the smooth and diffuse properties of faces, both PI-FPA and U-Net provide acceptable 3D face measurement results.Because of the lack of 3D label data for the tested face, it cannot identify precisely which of these two results is better, but there are slight differences in some local details, such as the left cheek and the tip of the nose in Fig. 6(b).In the whole measuring procedure, the reconstructed dynamic face at different time points verified the reliability of PI-FPA to perform fast 3D shape measurement with high completeness as well as see Supplementary Video S3.These results suggest that PI-FPA is a promising tool for fast 3D measurement and reverse modeling with high quality for objects with complex shapes.

Conclusions and discussion
In summary, we have demonstrated a physics-informed deep learning method for fringe pattern analysis (PI-FPA) that is able to achieve accurate and computationally efficient single-shot phase reconstruction and exhibits strong generalization capability to new types of samples.By introducing the LeFTP module with the prior knowledge of traditional phase demodulation methods, PI-FPA circumvents the requirement of collecting a large amount of high-quality data, while overcoming the degradation of reconstruction quality for rare samples or structures in supervised learning methods.Utilizing reliable phase results from LeFTP as the network input, PI-FPA strengthens the ability of the lightweight DNN to further improve the phase recovery accuracy at a low computational cost compared with existing end-to-end networks.The effectiveness of PI-FPA has been verified by several experiments for measuring various types of static and dynamic scenes.The single-shot phase retrieval results of the David plaster confirmed that PI-FPA can reconstruct high-quality phase information for objects with complex surfaces, while also achieving an improvement of 3.46× in its network inference speed compared with U-Net.By adopting stereo phase unwrapping, PI-FPA has the capability of single-frame 3D imaging to successfully recover the fine profiles of the industrial part with the threads, exhibiting its good generalization to rare samples never seen by the network.Temporal precision analysis results verified the high accuracy and excellent repeatability of PI-FPA for measuring multiple moving objects simultaneously.Finally, 360-degree reconstruction of a workpiece model and non-rigid dynamic face measurement revealed the applicability of PI-FPA for fast 3D measurement with high quality for objects with complex shapes and different materials.In the future, the performance of PI-FPA for phase retrieval from various types of fringe images will be investigated.We wish that PI-FPA can be applicable to other fringe analysis applications in optical metrology, further pushing the limits of fringe pattern analysis in speed, accuracy, repeatability, and generalization.

Fig. 1 |
Fig. 1 | Diagrams of the physics-driven method, physics-informed deep learning approach, and data-driven deep learning approach for fringe pattern analysis.

Fig. 2 |
Fig. 2 | Overview of the proposed PI-FPA.(a) PI-FPA including a LeFTP module and a lightweight network.(b) Net head and Net tail.(c) The phase retrieval process of the LeFTP module.

Fig. 3 |
Fig. 3 | Comparative results for single-shot fringe pattern analysis of the David model.(a-e) The phase retrieval process, wrapped phases, phase errors, and magnified views of the phase errors using FTP, LeFTP, Net head + LeFTP, U-Net, and PI-FPA.

Fig. 4 |
Fig. 4 | Comparative fringe analysis results of the industrial part.(a) The industrial part and the phase errors using FTP, U-Net, and PI-FPA.(b) The magnified views of the phase errors.(c) Single-shot 3D imaging results using different methods.(d) The magnified views of (c).(e) The line profiles in (d).

Fig. 5 | 9 360Fig. 6 |
Fig. 5 | Precision analysis for a ceramic plane and a standard sphere moving along the Z axis.(a) 3D reconstruction results using PI-FPA at different time points.(b-c) the error distributions of the sphere and plane.(d-e) temporal precision analysis results of the plane and sphere over a 1.62 s period using 3-step PS, FTP, U-Net, and PI-FPA.(f-i) the color-coded 3D reconstruction and the corresponding error distributions of the plane and the standard sphere using different methods at T = 0.81 s.