Parameters of multi-convex lens array and the recording system for dusty plasma experiment with the CMOS camera D810.
The integral photography technique has an advantage in which instantaneous three-dimensional (3D) information of objects can be estimated from a single-exposure picture obtained from a single viewing port. Recently, the technique has come into use for scientific research in diverse fields and has been applied to observe fine particles floating in plasma. The principle of integral photography technique and a design of a light-field camera for dusty plasma experiments are reported. The important parameters of the system, dependences of the size of the imaging area, and the spatial resolution on the number of lenses, pitch, and focal length of the lens array are calculated. Designed recording and reconstruction system is tested with target particles located on known positions and found that it works well in the range of dusty plasma experiment. By applying the integral photography technique to the obtained experimental image array, the 3D positions of dust particles floating in an RF plasma are identified.
- dusty plasma
- integral photography
- three-dimensional reconstruction
- particle measurements
- plenoptic camera
Fine particles immersed in plasma are charged up negatively, show three-dimensional (3D) motion, and form 3D-ordered state, i.e., Coulomb crystal [1, 2, 3, 4, 5]. Diagnostic methods for 3D information about the positions of fine particles in a plasma have therefore been widely researched. Among the various dusty plasma experiments, 90° separated two CCD cameras with helping 3D computed tomographic reconstruction  and stereoscopic [7, 8] are widely used to determine the 3D position of each fine particle . They require two or more detectors; however, the locations and numbers of observation ports are considerably restricted in many plasma experiment devices. Planar laser scanning technique can obtain the 3D information of particles with one CCD camera [10, 11], but it requires a little while to scan across the wide field of view. In-line holographic techniques  and two-color gradient methods [13, 14] can obtain 3D position of dust particles from a single-exposed photograph taken from a direction; however, these methods require a 12 bit or higher dynamic range sensors. It is required that a technique can acquire the 3D information of a dusty plasma with a single-exposed photograph taken from one viewing port with a conventional dynamic range sensor.
The “integral photography technique”  is known as a principle used in naked eye 3D display and in commercial refocus cameras. Such refocus camera is also called as “plenoptic camera” or “light-field camera.” It provides 3D imaging technologies based on a small lens array or a pinhole array to capture light rays from slightly different directions. This technique has an advantage in which instantaneous 3D information of objects can be estimated from a single-exposure picture obtained from a single viewing port.
Recently, the integral photography technique has come into use for scientific research in diverse fields. Example applications are particle tracking for velocimetry [16, 17, 18, 19], microscopy measurement [20, 21], spray imaging , etc. In the research filed of plasma, 3D reconstructions of positions of particles levitating in a plasma have been demonstrated using commercial light-field Lytro cameras , and the time evolution of dusty plasmas has been measured using a commercial light-field Raytrix camera . An open-ended plenoptic camera, which is constructed with a lens array and a typical reflex CMOS camera, obtained the 3D positions of dust particles in a radio-frequency (RF) plasma [25, 26]. Dual-filter plenoptic imaging system has been applied to observe lithium pellets in a high-temperature plasma .
In this chapter, the principle, design, and experimental results of the integral photography technique for 3D imaging of dusty plasmas will be presented.
2. Principle of the integral photography analysis
Figure 1 shows a schematic of recording and 3D reconstruction system with integral photography for dusty plasma experiment. A small lens array is placed in front of the particles levitating in a plasma to obtain an array of projected image. The rays emerging from 3D objects, i.e., scattering light rays from dusts pass through the small lens array and are captured on a sensor device. A 3D spatial point of object that is a position of a dust particle should be projected to two-dimensional (2D) image points (X(i, j), Y(i, j)) through each (i, j)th lens of the array on a detector. The 3D reconstruction is then carried out computationally by creating inverse propagating rays within a virtual system similar to the recorded one.
2.1 Recording system for dusty plasma experiments
In this section, it is shown how to design a recording system for dusty plasma experiments. Figure 2 shows a schematic of relationship among lens array, detector, and imaging area on the recording system. The Cartesian (x, y, z) coordinate axes are indicated in the lower right corner of the figure. Rays are projected directly onto the sensor after passing through the lens array, which is placed at z = 0. Therefore, 3D spatial point of light source, i.e., the position of a dust particle, is projected through each lens of the array to 2D image points (X(i, j), Y(i, j)) on the detector. The array of projected images on the sensor is called as “elemental image array.”
The considerable parameters of the lens array are the number of lenses, lens pitch, and focal length. In the following discussion, we deal only with convex lens array. The focal length F of a convex lens array is calculated from the lens law as follows:
where L is the distance between the object and the lens array and l is the distance between the lens array and the sensor. Figure 3 shows F dependence on L with three values of l. The most suitable value of F can be determined by changing the experimental configuration. Particles located at misaligned positions from focused z-plane create blurred spots on the sensor.
In order to make an efficient recording system, the configuration of the CMOS sensor must be considered. The number of lenses in the array should be a multiple of the aspect ratio of the CMOS sensor. In addition, the lens pitch must take into account the size of the imaging area. After passing through the lens, rays are projected onto the sensor directly. If we assume that rays from a given dust particle are projected onto all elemental images, the limit of the imaging area is calculated using straight lines passing through the center of the outermost lens
and through pixels on the edges of the sensor area corresponding to the outermost lens
Here, n is the number of lenses along a given axis, the subscript “y” denotes the direction of the y-axis, H is the length of the CMOS sensor in the y-axis, and p is the lens pitch. Furthermore, the limits for the plane perpendicular to the z-axis are expressed as
In the same manner, the limits for the z-axis of the imaging area are expressed as
The number nx of lenses along the x-direction is calculated from ny using the aspect ratio of the CMOS sensor. Then we can design p and n from experimental requirement of L and the size of the imaging area using above equations.
An uncertainty in the reconstructed image will be attributed to the spatial resolution of an elemental image on CMOS sensor . The length h of a side of a pixel on the CMOS sensor produces uncertainties in the plane perpendicular to the z-axis as
and along the z-axis itself as
The above equation indicates that the large ratio of distances L/l and finite size of a pixel on the CMOS sensor cause a relatively large uncertainty along the z-axis.
2.2 Reconstruction of 3D position of light source
To extract the positions of projected particles (X(i, j), Y(i, j)) from the elemental image array, a subtraction technique is applied together with the color and profile thresholds. Background image is subtracted from obtained experimental image to reduce background noises such as the reflection of the laser light from the wall of the vacuum vessel. Then, the color space of the subtracted image is converted from RGB to “Lab” scale . To obtain bright pixels colored with irradiating laser from the image, the threshold values of “L” and “a,” which depend on the experimental configurations, were considered. For example, by analyzing the dust experiment shown in the Section 3.3 (see Figure 9), pixels with “L” values greater than 110 out of 255 and “a” values less than 110 out of 255 are used. Filtering with the geometrical features of the luminance distribution patch on pixels was also applied to detect the positions of dust particles from noise. The luminance centroid of each luminance distribution patch was estimated, and it was treated as the position of the particle.
Using the observed elemental image array, the 3D image of particle distribution is reconstructed in the computer. The light path arriving at the point (pz, qz) on a virtual observation plane located on z from a point (X(i, j), Y(i, j)) corresponding to (i, j)th lens on the elemental image array is calculated with ray tracing according to geometrical information, such as distance from lens array to detector and focal length of lenses. If the thin lens approximation can be applied, (pz, qz) on giving z is easily calculated as
where is the center of the (i, j)th lens.
where r is the distance between (X(i, j), Y(i, j)) and the position (x, y, z) and α is the angle between the optical axis and the incident ray. G(i, j) is a function indicating whether (i, j)th lenslet exists in a field of view or not defined as
A target light source should locate on where I(x, y, z) shows the extremal value with respect to z, i.e., at convergent points of the rays. To detect the positions of dust particles, therefore, the virtual observation plane, indicated as E and E′ in Figure 1, is scanned along z-axis.
3. Experimental setup and results
3.1 Estimation of measurement parameters for a dusty plasma experiment
In order to determine the parameters of multi-convex lens array for a dusty plasma experiment, we adjusted the side of the imaging area and L to be approximately 5 and 80 mm, respectively. The commercial, standard reflex camera (D810, Nikon) was applied as the CMOS sensor. The sensor has dimensions of 36 × 24 mm2 and h = 1/205 mm. Due to the geometrical limit of the camera D810, the distance l must be more than 65 mm. Rays are projected onto the CMOS sensor directly after passing through the lens array. Subsequently, F is estimated as 35 mm from Eq. (1). The number n of lenses is inversely proportional to p and to the size of a lens. Because of the increased number of tracing rays, an increase in n increases the resolution; however, the number of pixels illuminated by each lens decreases. For a lens array including (nx, ny) = 9 × 6 lenses, p is led as 2.2 mm and the D810 camera has 818 × 818 square pixels for each lens. The imaging area for the plane perpendicular to the z-axis lies between y1 = −2.5 mm and y2 = 2.423 mm, as estimated from Eqs. (4) and (5). The area of the measurable plane is a function of the position z of virtual plane, and it is maximized to be approximately 24 mm2 at z = 80 mm. Regarding the z-axis, the imaging area lies between z1 = 143 mm and z2 = 55 mm, as estimated from Eqs. (6) and (7). Table 1 shows the parameters of the designed multi-convex lens array and those of the recording system.
|F||35 mm (convex)|
|nx × ny||9 × 6|
|Size of a lens||2.2 × 2.2 mm2|
|Pixels per lens||818 × 818 pixels|
|Working area on x−y plane at z = 80 mm||∼24 mm2|
|Working range of z from lens array||55–143 mm|
A picture of the designed lens array using acrylic plastic is shown in Figure 4. The rim around the periphery facilitates the holding of the array. Figure 5 shows a sample image obtained with the designed lens array system. The object appears as a single green circle with ∼1 mm of diameter which is located on z = 80 mm.
3.2 Reconstruction of known target light sources
Developed recording and reconstruction system has been tested using target light sources located on known positions. In this test experiment, the typical exposure time of the camera is 1/400 s. The elemental image array obtained with the system was stored in a computer as 10 bits of data. In addition to the position (X(i, j), Y(i, j)) of the particles identified in each elemental images, the intensity I′(X(i, j), Y(i, j)) for each particle is recorded as well. Then I(x, y, z) on a virtual observation plane located on z is calculated according to Eq. (12). The position of the maxima of the brightness of the light-field I(x, y, z) determines the z, which represents the depth of particles. After calibrating the apparent pixel sizes on the images to the real scales of the dust particle cloud, the absolute (x, y, z) coordinates can be determined.
In Figure 6, the 3D positions of the known target and reconstructed particles are marked by open squares and closed circles, respectively. In this test experiment, the optical axis is set along the z-axis, the multi-convex lens array is located at z = 0, and l is set to be 77 mm. Mechanical setting errors cause 30 μm of the error bars for the target data points. The error bars of the reconstructed data points are determined according to the pixel dimensions of the recording system. From Eqs. (8) and (9), the uncertainties for the x- (y-) and z-directions are estimated as ∼6 and 108 μm, respectively. The relative error between the positions of the target and reconstructed data fits into the known range. Therefore it is concluded that the developed recording and reconstruction system works well in the range of dusty plasma experiment.
3.3 Apply to dusty plasma experiment
Finally, the developed system is applied to a dusty plasma comprising monodiverse polymer spheres (diameter = 6.5 μm) floating in a horizontal, parallel-plate RF plasma. Figures 7 and 8 show a photograph and schematic of the experimental setup. A piezoelectric vibrator is contained in an RF electrode as the injector of fine particles into a plasma. A grounded counter electrode is positioned at the upper side of the 13.56-MHz-powered electrode at the distance of 14 mm. Fine particles levitate in the plasma generated between the electrodes. A solid-state laser, which radiates light of 532 nm in wavelength, 4 mm in diameter, and ∼10 mW in radiation power, was used in our experiment to observe fine particles in the plasma using scattered laser light.
The lens array and the CMOS sensor were located at a side port of the chamber with a distance of l = 65 mm to obtain the elemental image array. An enlarged experimentally obtained elemental image array from above experimental device is shown in Figure 9. This figure shows roughly 3 × 2 elements out of the 9 × 6 elements captured by the CMOS camera. Each image element is recorded with approximately 818 × 818 square pixels on the sensor. Scattered light from the dust particles levitating in the RF plasma appears as green dots, and slightly different elemental images result from parallax differences. As shown in Figure 9, only five dust particles floated in the plasma in this experiment. They oscillated vertically in the field of view and did not form any ordered array. The 3D positions of particles determined from Figure 9 are shown in Table 2 and Figure 10. Note that the z-axis is along the optical axis and value of z is the distance from the lens array. Figure 10(a) shows a bird’s-eye view of the reconstructed distribution. Green dots indicate the 3D positions of the levitating dusts, and cross symbols indicate the projected position on the x−y plane with z = 90 mm. The x−y distribution of dust particles in Figure 10(c) agrees with the observed configuration of dust particles in Figure 9. From the reconstructed image, we can recognize that particles are randomly distributed between z = 88.4 and 89.8 mm, which are the distance from the lens array. The mean distance among particles is estimated approximately 780 μm.
4. Expected future of the integral photography technique for plasma measurement
The integral photography technique has great potential of versatile applications for plasma measurement. With the help of Mie-scattering ellipsometry technique [31, 32], it would bring information about the size of particles in addition to six-dimensional information about position and velocity. Combined with intrinsic fluorescence spectroscopy , specification of dust’s materials will be available not only for standard polymer but also for unusual target such as microorganisms [34, 35]. Moreover, deconvolution techniques [36, 37] will extend the integral photography to determine 3D distribution of spatially continuous light sources . 3D information of bremsstrahlung emissivity distribution should be obtained with pinhole ultraviolet or soft X-ray detector [39, 40, 41], instead of lenslet array for visible light.
It is still unclear how many dust particles can be counted by the system using the integral photography technique because many parameters trade off against each other. Well-designed optical and recording systems are required to identify the 3D positions for a large number of particles. The defocusing effect of objects is another considerable problem. The effect makes the positions of objects on sensor difficult to identify, and uncertainties in the reconstructed image may increase. In order to avoid such a problem, some commercial light-field cameras mount different F lens arrays simultaneously. Moreover, subpixel analyses, modern particle detection, and interpolation algorithms would enable the achievement of enhanced accuracy [42, 43].
The integral photography technique is useful for 3D observation of dusty plasmas. This technique has an advantage in which instantaneous 3D information of objects can be estimated from a single-exposure picture obtained from a single viewing port. The principle of integral photography technique and its analytical method has been explained in detail. A design of a light-field camera for dusty plasma experiments has been reported. The important parameters of the system, dependences of the size of the imaging area, and the spatial resolution on the number of lenses, pitch, and focal length of the lens array are calculated. Then, the recording and reconstruction system has been tested with target particles located on known positions and found that it works well in the range of dusty plasma experiment. By applying the integral photography technique to the obtained experimental image array, the 3D positions of dust particles floating in an RF plasma are identified.
The author appreciates Prof. Y. Hayashi and Prof. Y. Awatsuji of Kyoto Institute of Technology for fruitful suggestions from the perspective of researches for the dusty plasma and the integral photography technique, respectively. The author also thanks Prof. S. Masamune of Chubu University and Prof. H. Himura of Kyoto Institute of Technology for the comments on this study. Finally, the author would like to thank Mr. K. Tokunaga of Kyoto Institute of Technology for experimental assistance. This research is partly supported by JSPS KAKENHI Grant Numbers 15K05364, 18K18750, and 24244094.