## 1. 3D capture and 3D content generation by digital holography

Three‐dimensional (3D) displays are the next generation displays. The claim for 3D imaging is indisputable in mass television, game industry, medical imaging, computer‐aided design, automated robotic systems, air traffic control, education and cultural heritage dissemination. The ultimate goal of 3D visual communication is 3D capture of a real‐life scene that is followed by creating its scaled exact optical duplicate at a remote site instantaneously or at a later time moment [1]. Tracking the development of 3D imaging devices from Wheatstone stereoscope designed in 1830 to modern full HD 3D displays with glasses (**Figure 1**) reveals some memorable periods of booming public interest to this area, e.g. the 3D theatres boom observed in 1950 as a counterpoint to the increasing popularity and commercialization of the television or the last years reaction to the ‘Avatar’ movie. Multiple parallax 3D display technology has evolved to full‐parallax displays for a naked eye observation as integral imaging displays [2] and super‐multi‐view displays [3]. However, since invention of holography by Dennis Gabor in 1948 [4] and the first holographic 3D demonstration by Emmett Leith and Juris Upatnieks in 1964 [5], the consumer public has high expectations for a truly holographic display in view that holographic imaging is the best candidate for 3D displays.

Holography is the only imaging technique which can provide all depth cues. High quality of 3D imaging from analogue white‐light viewable holograms is well known [6]. They provide a wide viewing angle due to the very fine grain size of the holographic emulsions. Realistic images can be viewed by an arbitrary number of viewers with unique perspectives. Motion parallax both vertically and horizontally and tilting of the head are possible. The viewer is capable of focusing at different depths. There are no convergence‐accommodation conflict and discontinuities between different views as for multiple parallax displays. Holographic imaging allows for building autostereoscopic displays. To realize a holographic display, the 3D scene description should be encoded as two‐dimensional (2D) holographic data. The 3D content generation is carried out by means of digital holography [7] based on the classical holographic principle. According to it, holography is a two-step process enabling storing and reconstruction of a wavefront diffracted by 3D objects. The hologram records as a 2D intensity pattern the interference of this wavefront with a mutually coherent reference wave as depicted schematically in **Figure 2**. The wave field from the object is characterized by a complex amplitude, *x*, *y*):

where the asterisk denotes a complex conjugation. The sum of intensities of the object and reference beams gives the zero‐order term. The last two terms are the +1 and –1 diffraction orders and contain the object wavefront information. The object field

Digital holography grew from a purely academic idea into a powerful tool after the recent progress in computers, digital photo‐sensors (CCDs or CMOS sensors) and spatial light modulators (SLMs). The capability of digital holography for digital analysis and synthesis of a light field forms two mutually related branches. The branch dedicated to analysis comprises methods for optical recording of holograms using digital photo‐sensors. The holograms are sampled and digitized by the photo‐sensor, stored in the computer and numerically reconstructed using different approaches to describe diffraction of light from the hologram and free space propagation to the plane of the reconstructed image [8]. Thus, capture of both amplitude and phase becomes possible enabling numerical focusing at a variable depth and observation of transparent micro‐objects without labelling [9]. The holographic data are in the form of a real‐valued 2D matrix of recorded intensity according to Eq. (1). In this branch, different techniques have emerged for the two decades of existence of digital holography as (i) digital holographic microscopy with a plane wave or a point‐source illumination [10, 11]; (ii) optical diffraction tomography with multi‐directional phase‐shifting holographic capture [12]; (iii) infrared holography in the long wavelength region for capture of large objects [13]; (iv) determination of sizes and locations and tracking of particles in a 3D volume [14]. Feature recognition based on digital holography has been proposed [15]. A lot of efforts were dedicated to instrumental or software solutions of the twin image problem [16].

The branch dedicated to synthesis of a light field comprises methods for computer generation of holograms [17] which are fed to some kind of SLMs for optical reconstruction of the images they encode. Computer‐generated holograms (CGHs) are used for holographic displays [18], holographic projection [19] and diffractive optical elements [20]. In principle, the CGHs provide the only means to generate light fields for virtual 3D objects. A CGH is a real‐valued 2D matrix of amplitude or phase data; it may have also binary representation.

Both branches are closely related to the task of direct transfer of optically captured digital holograms to a holographic display. To realize the chain 3D capture—data transfer—holographic display digital holography requires coherent light and 2D optical sensors and SLMs with high resolution and big apertures. The 4D imaging, when the time coordinate, is added to the data further aggravates the task of building a holographic display because the latter needs much higher resolution and much more information capacity than other types of 3D displays. Generally speaking, there are two ways for 3D content generation for holographic displays: (i) conversion of optically captured holographic data; (ii) computer generation of holograms. Furthermore, we discuss these two main tendencies—direct feeding of optically recorded digital holograms to a holographic display and computer generation of interference fringes from directional, depth and colour information about the 3D objects—on the basis of our experience in forming the holographic content.

## 2. 3D content generation from holographic data

The ultimate goal of digital holography is to build a system for 3D scene capture, transmission of captured data and 3D optical display. Although based on clear theoretical grounds given by Eq. (1), it is hard to fulfil this task because of limitations encountered at digital implementation of the holographic principle due to the discrete nature of photo‐sensors and display devices, their small size and low spatial resolution. The modern devices are characterized with a pixel periods from 1 to 20 μm and active areas from 1 up to 2–3 cm^{2}. 3D content generation from optically captured digital holograms should include three steps: (i) multi‐view capture by a set of cameras or by sequential recording from different perspectives (**Figure 3**); (ii) conversion of the captured data to a display data format; (iii) feeding the data to a display from many SLMs to enlarge the viewing angle (**Figure 3**).

A key problem of digital holographic capture and imaging is the very small value of the maximum angle between the object and reference beams which satisfies sampling requirement for the spatial frequency at current low spatial resolution of electrically addressable devices. In theory, the photo‐sensor must resolve the fringe pattern formed by interference of the waves scattered from all object points with the reference wave. The holographic display should support some space‐bandwidth product with regard to limitations of the human visual system. The maximum angle,

The limitation set by Eq. (2) means capture of small objects at a large distance from the camera and a small viewing angle at optical reconstruction. If the object lateral size *D* is much greater than the sensor size, the minimum distance between the object and the photo‐sensor is about

Effective operation of the holographic imaging system requires maintaining the consistent flow of data through capture, transmission and display blocks. So the other problem of 3D content generation is non‐trivial mapping of the data from 3D holographic capture with non‐overlapping camera apertures to arbitrary configuration of display devices (**Figure 3**). In the general case, the wavelength, the pixel period and the pixel number differ at capture and display sides, i.e.

In the first example, the capture parameters substantially differed from the parameters on the display side. The mapping was done for a circular holographic display under visible light illumination when the input data were extracted from a set of holograms recorded at 10.6 μm [29]. The interest in capturing holograms in the long wavelength infrared region is due to the shorter recording distance, larger viewing angle and less stringent requirements to stability of the system. The object was a bronze reproduction of the Benvenuto Cellini Perseus sculpture with a height of 33 cm [28]—a large object for digital holography. Nine off‐axis digital holograms were captured by rotating the object with an angular step of 3° using an ASi (amorphous silicon) thermal camera with

where

and the reconstructed image undergoes longitudinal and lateral magnifications:

where **Figure 4**. Shining **Figure 4** (actually, the phases of these fields were fed to each SLM). The processing allowed for combining the images created by all SLMs into a single reconstruction which could be viewed smoothly within an increased viewing angle of 24°. The video with the reconstruction can be found in reference [29]. The most remarkable fact of this data mapping from far infrared capture to a circular display was that we achieved more or less same longitudinal and lateral magnifications

The second example of data mapping is related to visualization of transparent objects by a holographic display with phase encoding of the input data. The object beam **Figure 5**). Omission of the amplitude,

## 3. Computer generation of 3D contents for holographic imaging

### 3.1. Methods for computer generation of holographic fringe patterns

A CGH is a fringe pattern that diffracts light into a wavefront with desired amplitude and phase distributions and seems to be the most appropriate choice for 3D content generation. This wavefront can be created both for real and virtual objects. The goal in developing the CGHs input data for a 3D holographic display is to have real‐time generation of large‐scale wide‐viewing angle full‐parallax colour holograms which provide photorealistic reconstruction that can be viewed with both eyes. These CGHs must support a motion parallax and the coupled occlusion effect expressed in the visible surface change according to the viewer position. For the purpose, a CGH must have a very large number of samples displayed on a device with high spatial resolution. So the most important requirements to CGH synthesis are computational efficiency and holographic image quality. The CGH computation involves digital representation of the 3D object which includes not only its geometrical shape but also texture and lighting conditions, simulation of light propagation from the object to the CGH plane, and encoding of the fringe pattern formed by the interference of the object light wavefront with a reference beam in the display data format.

There are two basic frameworks for CGH generation depending on the mathematical models of 3D target objects: (i) point cloud algorithms and (ii) polygon‐based algorithms. In the point‐cloud method [32], the 3D object is a collection of *P* self‐luminous point sources. The method traces the ray from a source ‘*p'* with spatial coordinates **Figure 6**). Each point source emits a spherical wave with an amplitude,

Polygon‐based representation is a wave‐oriented approach [33–35]. The object is a collection of *P* planar segments of a polygonal shape (**Figure 7**). Each polygon is a tilted surface source of a light field calculated by propagation of its angular spectrum of plane wave decomposition [27] using a fast Fourier transform (FFT). An angular‐dependent rotational transformation in the spectral domain is applied to find the spectrum in a plane parallel to the hologram in the global coordinate system from the spectrum in the tilted plane of the local coordinate system

The CGH synthesis of real objects can be carried out by 3D capture based on holographic means or structured light methods under coherent or incoherent illumination [41]. The output from, e.g. profilometric/tomographic reconstruction can be converted into a point cloud which allows for CGHs synthesis. The substantial advantage is the option to adapt the captured data to any holographic display. Incoherent capture of multiple projection images to generate holographic data has a lot of advantages such as incoherent illumination, no need of interferometric equipment and display of large objects. The concept was advanced 40 years ago in [42] by generating a holographic stereogram (HS). High quality of large format HSs as a ray‐based display, especially those printed by HS printers, are well known [43]. The input data for HS imaging are composed from colour and directional information. This causes a decrease of resolution for deep scenes and blurring. Introduction of a ray‐sampling plane close to the object and computation of the light wavefront from this plane to the hologram is proposed in Ref. [44]. Synthesis of a full‐parallax colour CGH from multiple 2D projection images captured by a camera scanned along a 2D grid is proposed in Refs. [45, 46]. The approach given in Ref. [46] relies on calculation of the 3D Fourier spectrum and was further improved [47] by developing a parabolic sampling of the spectrum for data extraction and needing only 1D camera scanning. Methods in which directional information from projection images is combined with a depth map are under development [48–50].

Over the last decade, the efforts were focused on improving image quality by different rendering techniques and on accelerating the CGH computation. The holographic data—amplitude and phase—should encode occlusions, materials and roughness of the object surface, reflections, surface glossiness and transparency. It is difficult to create an occlusion effect using 3D object representation as a collection of primitives—points or polygons—due to the independent contribution of all primitives to the light field. To decrease the computational cost of occlusion synthesis, a silhouette mask approach has been proposed in the polygon‐based computation [35]. The mask produced by the orthographic projection of the foreground objects blocks the wavefront of light coming from background objects. The method allows for synthesis of a very large CGH [35] but is prone to errors at oblique incidence. Computation is accelerated if occlusion is included in a light‐ray rendering process from multiple 2D projection images during the synthesis of a CGH as an HS [51]. As the method suffers from decrease of angular resolution in deep scenes, accuracy is improved by processing occlusion in the light‐ray domain along with sampling the angular information from the projection images. This is done in a virtual ray‐sampling plane [52, 53]. Furthermore, the sampled data are converted by Fourier transform to the object beam complex amplitude. Considering occlusion as geometric shadowing [54], effective CGH synthesis can be carried out by casting from each sample at the hologram plane, a bundle of rays at uniform angular separation within the diffraction angle given in Eq. (2). Such approaches are described in Refs. [54–56] with representing the 3D objects as composed from planar segments parallel to the hologram plane [55] or performing a lateral shear of the 3D scene to use the z‐buffer of the graphic processing unit (GPU) for the rays with the same direction to accelerate computation [54]. Occlusion, texture and illumination issues can be handled by computer graphics techniques. Their effective use is possible when the ray casting is applied by spatially dividing the CGH into a set of sub‐holograms and building different sets of points or polygons for them [57–59].

At specular reflection, the viewer is able to see only part of the object while the diffuse reflection sends light rays in all directions. Both types of reflections must be encoded in a CGH by adopting different reflection models to represent texture of the objects [60–62]. In the CGH synthesis, the luminance is encoded in the amplitude while reflectance is incorporated as a phase term. The task of representing reflection becomes rather complicated at non‐plane wave illumination or in the case of background illumination [60]. A perfect diffuse reflection is achieved by adding a uniformly distributed random phase. Unfortunately, this causes speckle noise at reconstruction [63]. A variety of methods have been proposed for fast synthesis of CGHs as look‐up table methods with pre‐computed fringes [63, 64], recurrence relation methods instead of directly calculating the optical path [65], introduction of wavefront recording plane [66], HS methods and many others. Hardware solutions as special‐purpose computers like ‘Holographic ReconstructioN (HORN)’ [67] or GPU computing [32, 68–70] are very effective for fast calculation because the pixels on a CGH can be calculated independently.

### 3.2. Phase‐added holographic stereogram as a fast computation approach

Effective acceleration of computation is achieved in coherent stereogram (CS) algorithms when the CGH is partitioned into segments and the directional data for each segment are sampled (**Figure 8**). A similar idea has been advanced in the diffraction‐specific fringe computation by Lucente [71] with partitioning the hologram into holographic elements called hogels and each hogel having a linear superposition of weighted basic fringes corresponding to the points in a point cloud. Each segment in the CS emits a bundle of light rays that form a wavefront as a set of patches of mismatched plane waves due to lack of depth information. This drawback was overcome by adding a distance‐related phase [72]. The phase‐added stereogram (PAS) is computationally effective if implemented by FFT. To clarify this point, we depict schematically the PAS computation with FFT in **Figure 9**.

In CS and PAS algorithms, the hologram is partitioned into *P* points. The segment size, *p'* comprises spatial frequencies *p'* and the central point, *p'* to the central point of the segment **Figure 9**). The FFT step moves the spatial frequencies to the nearest allowed values in the discrete frequency domain. The complex amplitudes remain the same. The two‐step procedure is repeated for each segment to compute the CGH.

The PAS approximation should yield a wavefront close to the wavefront provided by the Rayleigh‐Sommerfeld diffraction model that treats the propagating light from a point as a spherical wave. The complex amplitude in the reference model is given by:

We applied PAS computation to generate digital input contents for a wavefront printer developed by us for printing a white‐light viewable full‐parallax reflection hologram [73, 74]. The printed hologram was recorded as a 2D array of elemental holograms. The CGH for each elemental hologram was fed to amplitude SLM with 1920 × 1080 pixels. The object beam encoded in the CGH was extracted by spatial filtering and demagnified using a telecentric lens system. Unlike the HS printers [43], the wavefront printer uses full holographic data. That is why the synthesis of a large number of elemental holograms, e.g. 100 × 100, takes a very long time. This requires a fast computation method that provides quality of imaging close to the reference model. We solved this task by developing a fast PAS (FPAS) method [75] as a further elaboration of the already existing PAS methods. Usage of the FFT is crucial for fast PAS implementation, but this may affect negatively the quality of imaging due to spatial frequencies mapping to a predetermined coarse set of discrete values. The sampling step, **Table 1** which gives the fringe pattern at a single spatial frequency in the segment

The first improvement CPAS (compensated PAS) [76] performed some steering correction by adding the phase, which includes the difference between the spatial frequencies in the continuous and the discrete domains. The CPAS provided a better reconstructed image than the PAS with FFT at almost the same calculation time. Finer sampling was proposed in the accurate PAS (APAS) [77] by computing the FFT in an area which exceeds the segment and by properly truncating the larger‐size IFFT output. Phase compensation and directional error reduction by finer sampling were merged into a single step in the algorithm ACPAS [57] which yielded quality of reconstruction very close to the reference model. The best results are provided by the FPAS algorithm which is characterized with better phase compensation than the previous methods. This was confirmed by quality assessment with conventional image‐based objective metrics as intensity distribution and peak signal‐to‐noise ratio [75] for reconstruction of a single point and also by good quality of reconstruction from white‐light viewable colour holograms (**Figure 10**) printed by our wavefront printing system [73] on an extra‐fine grain silver‐halide emulsion Ultimate08 [78]. The CGH computed by the FPAS algorithm for each elemental hologram was displayed on an amplitude type SLM. The demagnified pixel interval was 0.42 μm at the plane of the hologram and gives a diffraction angle of ± 39.3°. For uniform illumination of the CGH on the SLM without decreasing too much the laser beam intensity we used only 852 × 852 pixels in the SLM to project CGHs. Thus, the size of the elemental hologram became equal to 0.38 mm by 0.38 mm. The printed holograms are shown in **Figure 10**; their size is 5 cm × 5 cm and 9 cm × 9 cm. The smaller hologram consisted of 131 × 131 elemental holograms. The segment size for calculating the CGH fed to a given elemental hologram was 32 × 32 pixels while the FFT computation area was 128 × 128 pixels, and each elemental hologram comprised more than 700 segments.

## 4. Conclusion

Holographic imaging is a 3D imaging with all depth cues and inherent vision comfort for the viewer. That is why the last decade was marked by rapid development of methods of 3D capture and 3D content generation for holographic display, holographic projection and holographic printing. In the chapter, we considered implementation of the holographic imaging by digital means when the input data are in the form of a 2D real‐valued matrix, which should encode the light wavefront coming from the 3D scene. This wavefront can be extracted from optically recorded holograms or synthesized numerically using various 3D scene descriptions. Holographic recording by digital photo‐sensors or computer generation of holograms for pixelated SLMs imposes severe limitations on the space‐bandwidth product of the capture/display system. We discussed two cases of data mapping from holographic capture to holographic display to show that holographic data transfer from optically recorded digital holograms to the data format of a given display is not a trivial task due to inevitable distortions introduced as a result of different capture and display parameters. Representing 3D contents as computer‐generated holograms seems more flexible and promising way to create input data for holographic displays. The main requirements are to improve the quality of imaging and computational efficiency. We presented an algorithm for fast computation of holograms and showed the good quality of imaging it provided in holographic printing of white‐light viewable reflection holograms.