Open Access is an initiative that aims to make scientific research freely available to all. To date our community has made over 100 million downloads. It’s based on principles of collaboration, unobstructed discovery, and, most importantly, scientific progression. As PhD students, we found it difficult to access the research we needed, so we decided to create a new Open Access publisher that levels the playing field for scientists across the world. How? By making research easy to access, and puts the academic needs of the researchers before the business interests of publishers.
We are a community of more than 103,000 authors and editors from 3,291 institutions spanning 160 countries, including Nobel Prize winners and some of the world’s mostcited researchers. Publishing on IntechOpen allows authors to earn citations and find new collaborators, meaning more people see your work not only from your own field of study, but from other related fields too.
To purchase hard copies of this book, please contact the representative in India:
CBS Publishers & Distributors Pvt. Ltd.
www.cbspd.com

customercare@cbspd.com
Department of Electrical Engineering, Yuan Ze University, Chungli,, Taiwan, ROC
ICheng Chang
Department of Computer Science and Information EngineeringNational Dong Hwa University, Hualien,, Taiwan, ROC
BorTow Chen
Opto Electronics & Systems Laboratories, Industrial Technology Research Institute, Hsinchu, Taiwan, ROC
ChingLong Huang
Opto Electronics & Systems Laboratories, Industrial Technology Research Institute, Hsinchu, Taiwan, ROC
*Address all correspondence to:
1. Introduction
Digital colour camera now has been a popular consumer equipment, has widely used in our daily life, and is highly suited for the next generation of cellular phones, personal digital assistants and other portable communication devices. The main applications of digital colour camera are used to take a digital picture for personal use, picture editing and desktop publishing, highquality printing, and image processing for advanced academic research. The digital picture is a twodimensional (2D) form with threecolour components (Red, Green, Blue). The most of efforts for a digital camera producer are focusing on the improvements of image compression, image quality, image resolution, and optical/digital zooming. However, consider the wide field of computer vision, the depth for a captured object may be another very useful and helpful information, such as surface measurement, virtual reality, object modelling and animation, it will be valuable if the depth information can be obtained while a picture is being captured. In other words, threedimensional (3D) imaging devices promise to open a very wide variety of applications, particularly, those involving a need to know the precise 3D shape of the human body, e.g. ecommerce (clothing), medicine (assessment, diagnosis), anthropometry (vehicle design), postproduction (virtual actors) and industrial design (workspace design) (Siebert & Marshall, 2000). To achieve this significant function, a novel 3D digital colour camera has been successfully developed by Industrial Technology Research Institute, Opto Electronics & Systems Laboratories (ITRIOES), in Taiwan. In this article, the previous works, algorithms, structure of our 3D digital colour camera, and 3D results, will be briefly presented.
To obtain 3D information of a given object, the approach may be considered in between a passive scheme and an active scheme. The widely known passive scheme is stereovision, which is useful to measure surfaces with welldefined boundary edges and vertexes. An algorithm to recognize singular points may be used to solve the problem of correspondence between points on both image planes. However the traditional stereoscopic system becomes rather inefficient to measure continuous surfaces, where there are not many reference points. It has also several problems in textural surfaces or in surfaces with lots of discontinuities. Under such an environment, the abundance of reference points can produce matching mistakes. Thus, an active system based on a structured light concept will be useful (Siebert & Marshall, 2000, Rocchini et al., 2001; Chen & Chen, 2003). In our 3D camera system, the constraint that codifies the pattern projected on the surface has been simplified by using a random speckle pattern, the correspondence problem can be solved by a local spatialdistance computation scheme (Chen & Chen, 2003) or a socalled compressed image correlation algorithm (Hart, 1998).
In our original design, the 3D camera system includes a stereoscopic dualcamera setup, a speckle generator, and a computer capable of highspeed computation. Figure 1(a) shows the first version of our 3D camera system including two CCD cameras needing a distance of 10 cm between its 2 lenses, and a video projector, where the used random speckle pattern in Fig. 1(b) is sent from the computer and projected via the video projector on the measuring object. Each of two cameras takes the snapshot from its own viewpoint, and can do the simultaneous colour image capturing. A local spatialdistance computation scheme or a compressed image correlation (CIC) algorithm then finds some specific speckles on the two camera images. Each of the selected speckles would have its position shown twice, one on each image. After establishing the statistic correlation of the corresponding vectors on the two images, the 3D coordinates of the spots on the object surface will be known from the 3D triangulation.
Not only the traditional stereoscopic systems (Siebert & Marshall, 2000; Rocchini et al., 2001) but also the above mentioned system (Chen & Chen, 2003) are all not easy to be used friendly and popularly due to large system scale, complicated operations, and expensiveness. Hence, to achieve the valuable features (portable, easy operation, inexpensiveness) as possessed by a 2D digital camera, we present a novel design which can be applied to a commercial digital still camera (DSC), and make the 2D camera be able to capture 3D information (Chang et al., 2002). The proposed 3D handheld camera (the second version of our 3D measurement system) contains three main components: a commercial DSC (Nikon D1 camera body), a patented threehole aperture lens (Huang, 2001; Chen & Huang, 2002), and a flash. The flash projects the speckle pattern onto the object and the camera captures a single snapshot at the same time. Accordingly, our 3D handheld camera design integrating together the speckle generating projector and the colour digital camera makes the system be able to move around freely when taking pictures.
The rest of this article is organized as follows. Section 2 reviews briefly our previous works. Section 3 presents algorithms for improving 3D measurements. The structure of our novel 3D camera is described in Section 4. Finally a conclusion is given in Section 5. Because the found 3D information should be visualized for the use, all the 3D results are currently manipulated and displayed by our TriD system (TriD, 2002), which is a powerful and versatile modelling tool for 3D captured data, developed by ITRIOES in Taiwan.
Our original 3D measurement system shown in Fig. 1(a) includes two CCD cameras and a video projector, where the used random speckle pattern shown in Fig. 1(b) is sent from the computer and projected via the video projector onto the object to be measured. In this system, to solve the correspondence problem of measuring a 3D surface, the random speckle pattern was adopted to simplify the constraint that codifies the pattern projected on the surface and the technique of spatial distance computation was applied to find the correspondence vector (or the correlation vector used in the later of this article).
To effectively perform the correspondence vector finding task, the binarization for the captured image is used in our 3D system developments. The following is our adaptive thresholding method for binarization.
Let a grey block image be defined as G having the size ofm×m. The correspondence problem is based on the local matching between two binary block images. Therefore it is important to determine the thresholding value TH, for obtaining the binary block image B. To overcome the unevenbrightness and outoffocus problem arising from the lighting environment and different CCD cameras, the brightness equalization and image binarization are used. Let m2 be the total number of pixels of a block image, and cdf(z), z = 0~255 (the grey value index, where each pixel is quantized to a 8bit data) be the cumulative distribution function of G, then a thresholding controlled by the percentile p = 0~100% is defined
THp={zpcdf(zp)≈pm2}E1
Thus for a percentile p each grey block image G will have a thresholding value THp to obtain its corresponding binary block image B, and we have
B(x,y)={1if g(x,y)≥THp0otherwiseE2
where 1 and 0 denote the nonzero (white) pixel and the zero (black) pixel, respectively. Note here that the higher the p is, the smaller the data amount having nonzero pixels.
In our previous work, our distance computation approach for finding correspondence vectors is simply described as follows. LetBx0,y0l, x_{0} = 0, s, 2s, …; and y_{0} = 0, s, 2s, …, be a binary block image in the leftcaptured image starting at the location (x_{0}, y_{0}), where s is the sampling interval from the captured image. The searched block image, Bu0,v0rstarting at the location (u_{0}, v_{0}) in the rightcaptured image, will be in the range of u_{0} in [x_{0}Rx, x_{0}+Rx] and v_{0} in [y_{0}Ry, y_{0}+Ry], where Rx and Ry depend on the system configuration. If the CCD configuration satisfies to the epipolar line constraint, then Rycan be very small. In the searching range, if a right binary block image Buf,vfr has the minimum spatial distance d(Bx0,y0l,Buf,vfr) between it toBx0,y0l, then the vector from (x_{0}, y_{0}) to (u_{
f
}, v_{
f
}) is defined to be the found correspondence vector.
Because the corresponding information used in the stereoscopic system are usually represented with the subpixel level, in this version of 3D system, a simple averaging with an area A of size w×w containing the found correspondence results (u_{
f
}, v_{
f
})s is used to obtain the desired subpixel coordinate (uf*,vf*) and is expressed by
uf*=1N∑∀(uf,vf)∈Auf and vf*=1N∑∀(uf,vf)∈AvfE3
The more details of measuring a 3D surface using this distance computation scheme can be found in the literature (Chen & Chen, 2003). A result is given in Fig. 2 for illustration, where (a) and (b) show the captured left and right images; (c) displays the reconstructed 3D surface along some manipulations performed on the TriD system (TriD, 2000).
In order to investigate the accuracy of 3D information, we have developed another approach different to our previous spatial distance computation for improving our system. This idea comes from the analysis of partical image velocimetry using compressed image correlation (Hart, 1998). In the following, under a hierarchical search scheme, pixel level computation and subpixel level computation combined with brightness compensation will be presented for approaching to the goal of improving 3D measurement.
3.1. Pixel level computation
A hierarchical search schem is adopted in pixel level computation. First let the left image be divided into a set of larger fixedsize blocks, and called level 1 the top layer. Consider a block B1l in left image, if one block B1r in right image has the best correlation then the vector V1 from the coordinate of B1l to that of B1r is found. Based on the facility of coarsetofine, the next search is confined to the range indicated by V1 in right image and the execution time can be further reduced. Hence next, let the block image B1l in level 1 be further divided into four subblocks, this is the level 2. Consider the subblock B2l in B1l having the same coordinate, by the vectorV1, the correlation process is further performed only on the neighboring subblocks centered at the coordinate ofB1r. The best correlation conducting the vector V2 from the coordinate of B2l to one subblock B2r is found. Continue this process, if the best match is found and ended at level n, then the final vector of best correlation may be expressed as
V=∑i=1nViE4
In order to reduce the computation time of correlation, a socalled correlation error function for an M×N image is used and defined as follows (Hart, 1998).
This function only uses addition and subtraction, thus the time reduction is expectable. Note here that the processed images Is are binarized after adaptive thresholding as described in Section 2, thus the information of Is are only either 1 or 0.
3.2. Subpixel level computation
In order to increase the accuracy of the correspondence finding, two schemes are combined for achieving this purpose. One is grey scale interpolation, the other is brightness compensation. For grey scale interpolation, a linear scheme is performed on the third layer of right image. In our study, the block size of the third layer is8×8. The processing includes two steps as follows.
Step 1. Use the pixel grey levels in vertical direction to interpolate the subpixel grey level, e.g., 3point interpolation, between two neighboring pixels.Step 2. Based on the pixel and subpixel grey levels found in Step 1 to interpolate the subpixel grey leves in horizontal direction. In this case, the 3point interpolation is aslo considered as example.
A comparision among pixel level, subpixel level, and after interpolation is illustrated in Fig. 3(a)(c), respectively. Here we observe the image in Fig. 3(c) that the smoothness is improved greatly within the middle image but the randomness becomes more serious at two sides. It results from the ununiform brightness between the two CCD cameras. Hence a brightness compensation schem is presented to solve this problem.
As mentioned before for correlation error function in (5), the used correlation function (CF) may be redefined as
CF=I1+I2−I1−I2I1+I2E6
where
CF={2I2I1+I2if I1≥I2,2I1I1+I2otherwise.E7
Consider (7), if two block images I_{1} and I_{2} have different brightness, the correlation from I_{1} to I_{2} will be different to that from I_{2} to I_{1}. Furthermore, it will be dominated by the block image having lower grey level distribution. As a result, the more uniform the two block image distribution, the higher accuracy the correlation; and vice versa. To compensate such ununiform brightness between two block images and reduce the error, a local compensation factor (LCF) is introduced as
thus now (6) is modified as below and named CF with brightness compensation (BC).
CFBC=I1+LCF×I2−I1−LCF×I2I1+LCF×I2E9
According to (9), results in Fig. 3(d) and3(e) show that a good quality can be obtained. Here BC 32 means that 32 feature points are used in the subcorrelation. In our experiments, the accuracy can be increased 0.20.3 mm by the scheme of interpolation with brightness compensation; however a tradeoff is that 45 times of computational time will be spent.
3.3. Results
Consider the two captured images shown in Fig. 2(a) and 2(b) respectively, three reconstructed results using pixel level computation, subpixel level computation, and the further improvement by interpolation with BC are shown in Fig. 4(a),4(b), and 4(c) , respectively. Obviously, the later result shows a better performance.
In order to further increase the accuracy of reconstructing 3D object, a suitable method is to use a high resolution CCD system for capturing more data for an object. For example, in our system, Fig. 5(a) shows a normal resolution result with652×512, whereas Fig. 5(b) shows a high resolution result with1304×1024. Their specifications are listed in Table 1.
For a high resolution CCD system, due to more data to be processed we present a simplified procedure to solve the timeconsuming problem. Consider the case of1304×1024, the processing procedure is as follows.
Step 1. Down sampling. The image is reduced to a 652×512 resolution.Step 2. Pixel level correlation with 3 levels is performed on the 652×512 image. In this step, the coarse 80×64 correlation vectors are obtained at the lowest level.Step 3. Lift the lowest level in pixel level correlation from 8×8 to 16×16 each block, and further perform the pixel level computation on the original 1304×1024 image. Thus there are 160×128 correlation vectors to be output in this step.Step 4. Based on the correlation vectors obtained in steps 3 and 4, the subpixel level correlation is performed to produce the final results.
System specification
Normal resolution system
High resolution system
Baseline (mm)
70
100
Lens (mm)
600
700
ObjectCCD Distance (mm)
35
28
(mm mm)
83 62 ^{a}
180 150
Image Density Resolution
7.7 7.7 ^{b}
7.2 6.8
Table 1.
View Range” is defined as the (object width) (object height).
^{b} “Image Density Resolution” is defined as (image width/object width)(image
height/object height), thus the unit is (pixel/mm) (pixel/mm).Comparison between a normal and a high resolution CCD system in our study.
For further demonstrating the quality of our algorithms, four objects in Fig. 6(a) and their reconstructed results in Fig. 6(b) and 6(c) are given. Note here that these results are only obtained from one view, thus they can be regarded as a 2.5D range image data. If multiple views are adopted and manipulated by our TriD system, the totally 3D result can be generated and illustrated in Fig. 7. As a result, a set of effective algorithms have been successfully developed for our 3D measurement system.
4.3. D Camera
The proposed 3D handheld camera contains three main components: a commercial DSC (Nikon D1 camera body), a patented threehole aperture lens (Huang, 2001, Chen & Huang, 2002), and a flash as shown in Fig. 8(a). The flash projects the speckle pattern onto the object and the camera captures a single snapshot at the same time. To embed the 3D information in one captured image, we devise a novel lens containing three offaxis apertures, where each aperture was attached one colour filter as depicted in Fig. 8(b), so that a captured image carries the information from three different viewing directions. Since the three different images can be extracted from filtering the captured image with red, green, and blue component, respectively, the depth information may be obtained from these images by using the algorithms introduced in Section 3.
For the sake of illustrating the principle of our triaperture structure, an example of lens with two apertures is depicted in Fig. 8(c). Three points, P1, P2, and P3 are set on the central axis, where P2 is located at focal plane; P1 and P3 located at a far and near points with respect to the lens. The rays reflected from P2 pass through aperture A and B will intersect at the same location on the image plane, whereas P1 or P3 will image two different points. Accordingly the depth information of P1 and P3 may be computed from the disparity of their corresponding points on the image plane.
To extract the depth information from the single image, the image should be separated based on the colour filter. The colour composition and decomposition in Fig. 9 and an example shown in Fig. 10 are given for illustration. In Fig. 10(a), the image shows a mixture of R, G, B colour pixels since it merges the images from different dircection and colour filters. After colour separation process, three distinguished images based on R, G, B components are obtained as shown in Fig. 10(b)10(d). Based on our depth computation algorithm embedded in TriD system, the range data is obtained as Fig. 10(e) shows. If a grey image is applied, we can obtain a grey textrued 2.5D image as Fig. 10(f) using a rendering process in TriD. Similarly, once a colour image is fed into our system, a colour textured 2.5D image may also be obtained as shown in Fig. 10(g) and10(h) with different view angles. Note here that in this processing, a crosstalk problem may be rised, i.e., G and B components may corrupt the Rfiltering image for example. In our study, this problem may be solved by increasing image intensity while an image is being captured.
The processing stages of our acquisition system using the proposed 3D camera are as follows. The camera captures two images of the target. The first snap gets speckled image (for 3D information computation), which will be spilt into three images based on the colour decomposition described before. Then the correlation process is used to compute depth information. The second snap gets original image (as a texture image) for further model rendering. For example, the human face of one author (Chang, I. C.) of this article is used for modelling. The speckled and texture images are captured and shown in Fig. 11(a) and11(b), respectively. After the TriD software system, a face 3D model and its mesh model are obtained as shown in Fig. 11(c) and 11(d) , respectively. This result demonstrates the feasibility of our 3D handheld camera system.
As described above, using the typical digital camera with our patented threehole aperture lens along with a high accuracy calculation, the entire 3D image capturing process can now be done directly with a single lens in our 3D camera system. The threehole aperture provides more 3D information than the dualcamera system because of their multiview property. The depth resolution can therefore be increased considerably. Currently this 3D camera system has reached precision of submillimetre. The main system specifications are listed in Table 2. As a result, our 3D handheld camera design integrating together the speckle generating projector and the colour digital camera makes the system be able to move around freely when taking pictures.
Light source
(flash) Whitelight random speckle pattern
CCD resolution
2000 1312
Measuring distance
500 ~
Measuring range
(X) (Y) (Z)
Resolution
(X) (Y)
Image capturing speed
1/30 second capturing time and 20 second processing time
Colour
R, G, B each 8 bits
Software
Plugin module in TriD system
Table 2.
System specifications in our 3D handheld camera design.
Threedimensional information wanted has been an important topic and interested to many real applications. However, it is not easy to obtain the 3D information due to several inherent constraints on real objects and imaging devices. In this article, based on our study in recent years, we present effective algorithms using random speckle pattern projected on an object to obtain the useful correspondence or correlation vectors and thus reconstruct the 3D information for an object. Original two CCD cameras system has also been moved to a novel 3D handheld camera containing a DSC, a patented threehole aperture lens and a flash projecting random speckle pattern. Based on the manipulations of our TriD software system, our experiments have confirmed the feasibility of the proposed algorithms and 3D camera. This result guides us to a new era of portable 3D digital colour camera.
References
1.ChangI. C.HuangC. L.HsuehW. J.2002 Novel threedimensional handheld camera based on threeaperture lens,
2.ChenB. T.HuangC. L.2002 Device and methodology of capturing threedimensional data using single len, Patent 154398 Taiwan, ROC, 20022021 .
3.ChenY. S.ChenB. T.2003Measuring of a threedimensional surface by use of a spatial distance computation,
4.HartD. P.1998Highspeed PIV analysis using compressed image correlation,
5.HuangC. L.2001 Threedimensional capturing device using single lens, Patent 139230 Taiwan, ROC, 20012020 .
6.RocchiniC.CignoniP.MontaniC.PingiP.ScopignoR.2001A low cost 3D scanner based on structured light,
7.SiebertJ. P.MarshallS. J.2000Human body 3D imaging by speckle texture projection photogrammetry,
8.TriD: Technical Report 3 3D Human Modeling & Animation Application, (2002). ver. 2.0, OptoElectronics & Systems Laboratories, Industrial Technology Research Institute, Taiwan, ROC.
Written By
YungSheng Chen, ICheng Chang, BorTow Chen and ChingLong Huang