Dynamic full-color digital holographic 3D display on single DMD

Introduction

Since iPhoneX announced 3D face recognition in a high profile, the acquisition and display of 3D data arouses everyone’s enthusiasm again. Holography is the only technique that can reproduce all the depth cues in the human visual system, and holographic 3D images are free from the visual fatigue caused by the vergence accommodation conflict that occurs in most conventional 3D displays, so holography is the most promising technology for realizing 3D visualization¹. Static hologram print services have already been commercialized, but the dynamic color holographic displays are still at the research stage, and there are not yet any practical systems on the market². The core component of a dynamic color holographic display is the spatial light modulator (SLM). An SLM that provides full-complexity modulation and infinite pixels would be very convenient, but this type of SLM currently does not exist in practice. Common SLMs include the acousto-optic modulator^{3, 4} and the SLMs based on micro electro-mechanical systems (MEMS)^5-8, micro lens arrays^{9, 10} or liquid crystals (LC-SLMs)^11-13. Most practical SLMs can only perform a restricted type of modulation of incident light, such as amplitude-only or phase-only modulation, and even that restricted modulation is imperfect because the behavior of most multilevel SLMs varies strongly with the illumination wavelength. Some solutions addressing this problem include multiple combination SLMs^14-16, regional regulation of monolithic SLM^17-19 and off-axis multiplexed on single SLM²⁰. However, dynamic color matching remains challenging for all SLMs.

In this study, we present a digital holographic technology based on a single DMD for full spatial complex field and wavelength control of a light beam that is used for a dynamic color holographic display. We adopt a spatial detour phase based on a DMD that combines the square regions of 16 adjacent DMD micro-mirrors to act as a modulation unit called a super-pixel to modulate the amplitude and the phase of the target field independently. Gray-scale images are realized by loading an amplitude mask, while color separation is realized by loading different direction-blazed gratings that are phase attached to a phase mask, then the amplitude mask and phase mask are combined to realize “color super-pixel” holography. A double-lens conjugate optical system is combined with a spatial filter in the spatial spectrum plane to realize reconstructed color images, effectively eliminating horizontal and vertical color difference while retaining the high resolution and very high switch speed.

Full-color super-pixels methods

In the super-pixel method, K(N×N) neighboring pixels are combined into one super-pixel, and these pixels are denoted by E₀, E₁, ···, E_k−1, for a super-pixel with E, such that $E=\sum\limits_{i=0}^{k-1}{\alpha }_{i}{E}_{i}$, as illustrated in Fig. 1(a), where E_i can only be set to either 1 or 0. α_i denotes the possible i^th pixel complex valued weighting coefficients, ${\alpha }_{i}=$${\rm{exp}}(\rm{j}\times 2{\rm{\pi}} \times \dfrac{i}{N\times N})$. By selecting these coefficients wisely, and using sufficient numbers of pixels to form the super-pixel, we can cause E to travel around the full-complexity area. We balance the efficiency and accuracy of the modulation by selecting N=4, ${\alpha }_{i}={\rm{exp}}(\rm{j}\times {\rm{\pi}} \times \dfrac{i}{16})$ which are uniformly distributed between 0 and 2π; this allows us to create any field within a disk up to a very small discretization error^{21, 22}. As illustrated in Fig. 1(c), among the modulation results, there are 2948 types of amplitude modulation (without consideration of the phase difference), with a maximum range of approximately 5.02; there are 10655 types of phase modulation (without consideration of the amplitude difference), the adjacent phase difference reaches 0.0085 rad, and the accuracy is 13 bits (2¹³ = 8192). The modulation results for each type of amplitude or phase may correspond to the different switching states in the “color super-pixel” cell; the amplitude and phase can thus be modulated independently. Therefore, in the production of the color hologram, we divide the hologram into amplitude mask and phase mask-loaded elements.

Figure  1.  Full-Size Img PowerPoint

Combination of 4×4 neighboring pixels into one superpixel. (a) Weighting phase, where the pixels within each superpixel are distributed uniformly between 0 and 2π, each pixel can be independently modulated, and a spatial filter in the form of a circular aperture is placed in the Fourier plane of the first lens. We turn on the four pixels indicated by the yellow squares as an example. (b) The response E is the sum of the modulated pixels, as represented by the line arrow. (c) Complex target fields that can be constructed using a single super-pixel with size of 4×4, where the number of fields that can be constructed is 49291, the horizontal axis is the real part and the vertical axis is the imaginary part. (d) By loading blazed gratings in different directions for different colors of light, splitting of the differently colored light on the Fourier focal plane can be realized.

In attempts to resolve the interference of the zero-order light, higher diffraction orders of light and the higher diffraction-based reconstructed image that corresponds to the hologram caused by the SLM pixel structure on holographic reconstruction, most current research in holographic projection applications is based on filtering methods. Therefore, a spatial filter in the form of a circular aperture is placed in the Fourier plane of the DMD modulation plane. The spatial filter blocks the higher spatial frequencies so that the individual DMD pixels cannot be resolved and only the response of the super-pixel can be resolved. The size of the circular aperture acting as the spatial filter is set with an upper limit such that the highest allowed spatial frequency is no higher than $\Delta k' = \dfrac{{2{\rm{\pi}} dr}}{{\lambda f}}$ rad·pixel⁻¹, which represents the angular spectral resolution of a single DMD pixel, where λ is the wavelength of the light, f is the focal length of the first lens, d is the distance between pixels and r is the aperture size. We typically select $r(\le \dfrac{\lambda f}{16d})$ such that our system bandwidth matches the bandwidth of the target field, with an upper limit selected such that the highest allowed spatial frequency is no higher than ${\rm{\pi}}/8$ rad·pixel⁻¹. The position of the circular aperture acting as the spatial filter with respect to the super-pixel’s 1st diffraction order is chosen as $\left(x,y\right)=(-a,4a)$, where $a=\dfrac{-\lambda f}{16d}$. While the different colors of light can be separated by the position of the aperture, the diameter of the aperture demands a long focusing length.

Because the red-green-blue (RGB) color mode is used as a type of color standard in industry, by varying and overlapping the three color channels of red (R), green (G), and blue (B), we can obtain all the colors that can be perceived by human vision. The amplitude mask is directly loaded with the gray intensity image of the three RGB colors, while the phase image is loaded with gratings in different directions with their different RGB components decomposed, as illustrated in Fig. 1(d). After the displacement grating is loaded, the different RGB components move into the corresponding directions of the displacement grating, which then filters out the spectrum of the RGB components and combines them in the double-lens conjugate imaging system to form the reconstructed color image. Because the double-lens conjugate imaging system guarantees conjugate imaging of the coding plane and the display plane, matching of the different colors can be achieved by selecting the appropriate achromatic lens. In this way, the gray information of the reconstructed color image appears fuller and clearer.

We verified the fidelity of the color super-pixel holography method using the experimental setup shown in Fig. 3. The super-pixel-based phase and amplitude modulation method can be set up for any wavelength, so it is highly suitable for color projection applications. While any light wavelength can be used to illuminate the DMD when using the super-pixel method, a wider wavelength range causes more modulation errors to occur because a phase gradient will be induced in DMD chips containing approximately 1280 pixels. And only by controlling the $\left|\dfrac{{\rm{\Delta}} \lambda }{\lambda }\right|\approx$ 10%, it would allow the modulation fidelity to be maintained. In our experiments, we selected a red laser diode LD ($620 \mp 3\;{\rm{nm}}$), a green LD ($528 \mp 3\;{\rm{nm}}$) and a blue LD ($451 \mp 3\;{\rm{nm}}$) as the sources.

Our DMD (DLP Discovery 4100, 1920 pixels ×1280 pixels, 9 kHz modulation frequency) with its pixel pitch of 13.69 µm is divided into super-pixels composed of 4×4 micro-mirrors. The hologram is calculated using MATLAB software and is then loaded into the DMD projection display. We decompose one three-color image into the RGB color mode with three single gray images. As shown in Fig. 2, the amplitude mask is the superposition of each color’s gray level, while the phase mask is the superposition of each color’s displacement grating phase.

Figure  2.  Full-Size Img PowerPoint

Amplitude and phase mask generation process. (a1) Amplitude modulation mask for red. (b1) Amplitude modulation mask for green. (c1) Amplitude modulation mask for blue, where the amplitude modulation range is 0–1. (d1) Amplitude modulation mask for tri-primary color amplitude linear superposition, which can be loaded by the DMD. (a2) Phase modulation mask for red, loading a vertical blazed grating. (b2) Phase modulation mask for green, loading a horizontal blazed grating. (c2) Phase modulation mask for blue, loading a 45° tilted blazed grating, where the phase modulation range is (0,2π). (d2) Phase modulation mask for tri-primary color phase linear superposition, which can be loaded by the DMD.

The amplitude and phase of each super-pixel must be obtained using a lookup table that contains a sufficient number of points from the complex plane with super-pixel modulation. The lookup table size is selected to be 855×855 points, which is approximately 15 times the number of fields that we can construct at 49291 for the super-pixel size n = 4. Use of a lookup table means that the calculations required to determine which DMD pixels are to be turned on are minimized and thus the performance is optimized. In our implementation, it takes less than 4 MB of memory to store the required table. Loading of the table and using it to look up a DMD pattern is completed within a fraction of a second.

A spatial filter in the form of a circular aperture is placed in the Fourier plane between the lenses. This spatial filter blocks the higher spatial frequencies so that the individual DMD pixels cannot be resolved and only the response of the super-pixel is resolved. The super-pixel’s 1^st diffraction order is chosen as ($-a,4a$), because of the addition of the different RGB component displacement gratings as a phase mask, the real positions of the circular aperture in the form of the different RGB components of the color spatial filter with respect to the super-pixel’s 1^st diffraction order are chosen as Red $(-a-b,4a)$, ${\rm{Green}}(-a,4a+b)$, and Blue $(-a-b,$ $4a+b$) respectively, where $b=\dfrac{f\lambda }{md}$, λ is the wavelength of the light, f is the focal length of the first lens, $d$ is the distance between pixels, and m is the periodic number of the displacement grating.

The RGB components of the different colors are then combined in a double-lens conjugate imaging system to form a reproduction color image with different gray color information, as illustrated in Fig. 3. The lenses are placed slightly off-axis with respect to each other, which results in an extra phase factor occurring in the target plane. The constructed field is measured in the target plane on a CCD (Thorlabs DCU224C, 1280×1024) camera by off-axis digital holography. The control part of the whole system is realized using computer control software, and the DMD as well as the CCD are connected to the computer through USB interfaces.

Figure  3.  Full-Size Img PowerPoint

Schematic of the color holographic imaging display system. Three types of color semiconductor laser are collimated, expanded and combined, reflected by R2 and R3, and are incident on the DMD panel at 24°. A multi-hole aperture filter is placed at the focal plane of lens f4, and lens f5 is used to reconstruct the image on the color charge-coupled device (CCD) panel. The DMD and CCD panels are strictly conjugated by the f4 and f5 lens groups.

Results

During the course of this experiment, we initially used red and green colors to complete the experiment, and used the logo of our unit as an experimental sample. As shown in Fig. 4(a1), we used red and green to fill in the logo, and the two colors were strictly separated. Figure 4(a2) shows a hologram loaded on the DMD. Using of local amplification shows that the hologram is an approximately linear grating that is limited by single color modulation. Figure 4(a3) shows the spectrum plane image, where use of different color filter holes allows different colors to be displayed. Figures 4(a4) and 4(a5) show the red and green parts, respectively, as independent parts of the final two-color holographic display image, which is shown in Fig. 4(a6). Figure 4(b1) shows red, green, yellow and orange four-color holographic display results, where yellow and orange are achieved by superimposing red and green. The local magnification of the spectrum plane is shown in Fig. 4(b2), which presents a more complex two-dimensional grating distribution. Figures 4(b4) and 4(b5) also show the red and green parts, respectively. Because the proportion of red in orange is higher than that of yellow, it is obvious that the intensity of the orange part is weaker than that of the corresponding yellow part shown in Fig. 4(b5). Figure 4(b6) shows the final image result, where the four-color holographic display is well realized.

Figure  4.  Full-Size Img PowerPoint

Color image display results realized using a red LD and a green LD, where the logo of our research center is used as the target structure. (a1) Structural design. (a2) DMD loading image, where the local magnification image shows the loaded holographic mask. (a3) Spectrum plane. (a4) Measured red image. (a5) Measured green image; and (a6) measured double-color result image. (b1) Multicolor result map based on the red and green primary colors. (b2) DMD loaded image, where the local magnified image shows the loaded grating magnified image. (b3) Spectrum plane. (b4) Measured red image. (b5) Measured green image. (b6) Measured multicolor result image.

We have also achieved a tri-color holographic display image, as shown in Fig. 5. Figure 5(a2) shows the spectrum plane image; use of the different color89 filter holes allows different colors to be displayed. Figures 5(b2), 5(b3), and 5(b4) also show the red, green and blue images separately; pink and yellow are produced by superposition of the three primary colors, as shown in the corresponding part. Figures 5(a3) and 5(b5) show the final result images, indicating that the three-color and six-color holographic displays were well realized. We have also realized imaging of natural pictures with more detailed images and more image layers, as shown in Figs. 5(c1) and 5(c2). Because of the influence of the pixel resolution, there will be some crosstalk within the rich color details of the image.

Figure  5.  Full-Size Img PowerPoint

Color image display results realized using a red LD, a green LD and a blue LD, where the logo of our research center is again used as the target structure. (a1) Structural design. (a2) Three-color spectrum plane, where the three-color spectrum is separated in space. (a3) Measured three-color result image. (b1) Designed multi-color image. (b2) Measured red pattern. (b3) Measured green pattern. (b4) Measured blue pattern. (b5) Tri-color combination pattern. (c1) Actual loaded natural picture and (c2) image display picture.

To verify the reliability of this method, the original image (size of 1920 pixels ×1280 pixels) and the experimental image (size of 1280 pixels ×1024 pixels) are compared to verify the correctness of the method. In general, the verification criteria are the similarity of the image structure and the color. The algorithm used to verify the structural similarity is the Hash method²³. The Hash algorithm is explained as follows. The main working process involves generation of a fingerprint corresponding to each image, which is equivalent to extraction of the feature points of the image, and the fingerprints from the different images are then compared. Closer fingerprint results indicate greater similarity between the images, and our experimental verification result ranged up to 0.96, as shown in Table 1.

To verify the color similarity, we adopted two algorithms, where one was an objective algorithm and the other was a subjective algorithm. The main reason for this approach is that human beings have a strong ability to recognize structures, so the subjective algorithm is closer to the human visual cognition process. We adopted a histogram color similarity algorithm that was considered to be an objective algorithm²⁴. The objective algorithm does not consider direct correlation of the image pixels, and is limited by the CCD because its response to the different colors is not the same; in our experimental verification, the result ranged up to 0.67. The subjective algorithm uses structural similarity in the YUV (where “Y” means brightness, “U” and “V” mean chroma and concentration, respectively, and YUV is a color coding method) domain, which can also be called the structure similarity index (SSIM)^{25, 26}. To evaluate the color similarity of the images, the theoretical basis of the SSIM is that the human visual system can extract structural information from a scene in a highly adaptive manner. The SSIM is used to measure the integrity of the image structure. It is an intuitive image quality evaluation standard that reaches more than 0.98 in our experiments.

For traditional digital holographic 3D display, sizes of reconstructed objects and viewing angles are much smaller. Therefore, most of the reported digital holographic 3D displays are shown with static and simple structure samples. The combinatorial product of sizes of reconstructed and viewing angles is known as space bandwidth product (SBP) and is determined by SLM pixel counts and switching speed²⁷. To build a dynamic digital holographic 3D full-color display that meet customers’ expectations, the SBP value must be enlarged greatly. Based on the delay effect of human eyes, SBP can be extended by high-speed switching SLM. With the help of full-color super-pixel coding technology and high switching frequency of DMD, we have proposed full-color digital holographic 3D display with temporally enlarged SBP, which is expected to realize dynamic full-color 3D display. As shown in Figs. 6(a−c), the Fresnel phase was added to different parts of the loaded hologram to realize the imaging in different positions, therefore we can realize true holographic 3D display by layers. Combined with the fast switching speed (~9 kHz) of DMD, a dynamic full-color imaging (logo of SZU OPTICAL) is achieved, which is shown in Fig. 6(d). We have also supplied full-color 2D dynamic videos in Supplementary information. Limited by the current research, we use CCD for direct imaging acquisition for now. We expect to realize the dynamic 3D full-color display of naked eye, but the scattering medium is still being studied.

Figure  6.  Full-Size Img PowerPoint

Optical reconstructions of 3D display with different focal plane locations. (a) Focus on z=0. (b) Focus on z=40mm. (c) Focus on z=60 mm. (d) Dynamic full-color digital holographic imaging.

Conclusions

We adopted the color super-pixel digital holographic technology to realize the dynamic full-color 3D display, which achieves measured structural similarity of more than 88% and color similarity of more than 98%, while retaining the high switch speed of 9 kHz. When compared with traditional spatial optical modulation based on liquid crystals, the color super-pixel digital holographic technology can not only realize complex amplitude modulation, but also improve the display speed by two orders of magnitude which greatly enlarge the SBP of digital holographic display. As the switching speed of DMD device is further increased and the pixel size is further reduced, the modulation efficiency and resolution of this technology can be further improved, and full-color digital holographic 3D display on single DMD would be more widely used.