Hyperspectral Imaging Technology Used in Tongue Diagnosis

Traditional Chinese Medicine (TCM) is a range of medical practices used in China for more than four millenniums, a treasure of Chinese people (Lukman, He, & Hui, 2007). The important role of TCM and its profound influence on the health care system in China is well recognized. The West also has drawn the attention towards various aspects of TCM in the past few years (Chan, 1995). TCM consists of a systematized methodology of medical treatment and diagnosis (Watsuji, Arita, Shinohara, & Kitade, 1999). According to the basic concept of TCM, the different body-parts, zang-viscera and fu-viscera, the meridians of the body are linked as an inseparable whole. The inner abnormality can present on outer parts, while the outer disease can turn into the inner parts (Bakshi & Pal, 2010). Therefore, some diseases can be diagnosed from the appearance of the outer body. As the significant component part of TCM theory, TCM diagnostics includes two parts: TCM Sizhen (the four diagnosis methods) and differentiation of syndromes. The TCM physician experience the gravity of health condition of a sick person by means of the four diagnosis methods depending on the doctor's body "sensors" such as fingers, eyes, noses etc. Generally, TCM Sizhen consists of the following four diagnostic processes: inspection, auscultation and olfaction, inquiry, and pulse feeling and palpation (Nenggan & Zhaohui, 2004). In the inspection diagnostic process, TCM practitioners observe abnormal changes in the patient's vitality, colour, appearance, secretions and excretions. The vital signs encompass eyes, tongue, facial expressions, general and body surface appearance. The inter-relationship between the external part of the body such as face and tongue and the internal organ(s) is used to assist TCM doctors to predict the pathological changes of internal organs. In the auscultation and olfaction process, the doctor listen the patient's voice, breathing, and coughing used to judge the pathological changes in the interior of the patient's body. Inquiry diagnosis method is refer to query patient's family history, feelings in various aspects, such as chills and fever, perspiration, appetite and thirst, as well as pain in terms of its nature and locality. Palpation approach involves pulse diagnosis (Siu Cheung, Yulan, & Doan Thi Cam, 2007). The palpation diagnosis has been accepted as one of the most powerful method to give information for making diagnosis from ancient time till now. The pulse waves are measured at six points near the wrists of both hands. The waves are different each other and give us information about different organs (Hsing-Lin, Suzuki, Adachi, & Umeno, 1993). Tongue diagnosis is another inspection diagnostic method which


Introduction
Traditional Chinese Medicine (TCM) is a range of medical practices used in China for more than four millenniums, a treasure of Chinese people (Lukman, He, & Hui, 2007).The important role of TCM and its profound influence on the health care system in China is well recognized.The West also has drawn the attention towards various aspects of TCM in the past few years (Chan, 1995).TCM consists of a systematized methodology of medical treatment and diagnosis (Watsuji, Arita, Shinohara, & Kitade, 1999).According to the basic concept of TCM, the different body-parts, zang-viscera and fu-viscera, the meridians of the body are linked as an inseparable whole.The inner abnormality can present on outer parts, while the outer disease can turn into the inner parts (Bakshi & Pal, 2010).Therefore, some diseases can be diagnosed from the appearance of the outer body.As the significant component part of TCM theory, TCM diagnostics includes two parts: TCM Sizhen (the four diagnosis methods) and differentiation of syndromes.The TCM physician experience the gravity of health condition of a sick person by means of the four diagnosis methods depending on the doctor's body "sensors" such as fingers, eyes, noses etc.Generally, TCM Sizhen consists of the following four diagnostic processes: inspection, auscultation and olfaction, inquiry, and pulse feeling and palpation (Nenggan & Zhaohui, 2004).In the inspection diagnostic process, TCM practitioners observe abnormal changes in the patient's vitality, colour, appearance, secretions and excretions.The vital signs encompass eyes, tongue, facial expressions, general and body surface appearance.The inter-relationship between the external part of the body such as face and tongue and the internal organ(s) is used to assist TCM doctors to predict the pathological changes of internal organs.In the auscultation and olfaction process, the doctor listen the patient's voice, breathing, and coughing used to judge the pathological changes in the interior of the patient's body.Inquiry diagnosis method is refer to query patient's family history, feelings in various aspects, such as chills and fever, perspiration, appetite and thirst, as well as pain in terms of its nature and locality.Palpation approach involves pulse diagnosis (Siu Cheung, Yulan, & Doan Thi Cam, 2007).The palpation diagnosis has been accepted as one of the most powerful method to give information for making diagnosis from ancient time till now.The pulse waves are measured at six points near the wrists of both hands.The waves are different each other and give us information about different organs (Hsing-Lin, Suzuki, Adachi, & Umeno, 1993).Tongue diagnosis is another inspection diagnostic method which have presented a series of tongue body segmentation algorithms such as the Bi-elliptical deformable contour (REDC) algorithm (Bo Pang et al., 2002;B Pang et al., 2005), the combination of polar edge detection and active contour model (Zuo, Wang, Zhang, & Zhang, 2004), etc. Yang, et al. presented the colour active contour models to segment the tongue body (S.Yu, Yang, Wang, & Zhang, 2007).Some other algorithms also have been presented to analyze the tongue color (C.H. Li & Yuen, 2002;Y. G. Wang, Yang, & Zhou, 2007;Zhang, Wang, & Jin, 2005), tongue shape (Z.Liu, Jing-Qi, Tao, & Qun-Lin, 2006;Xu, Tu, Ren, & Zhang, 2008), sublingual veins (Chiu, Lan, & Chang, 2002;Yan, Wang, & Li, 2009;Yan, Yu, Wang, & Li, 2008), tongue fissures and cracks (L.L. Liu & Zhang, 2007;L. L. Liu, Zhang, Kumar, & Wu, 2008;Yue Zhou, Li Shen, & Jie Yang, 2002), tongue coating (Kim et al., 2008;Wu, Zhang, Li, Wang, & Yang, 2008), etc.These studies prove that the accurate segmentation of tongue body and extraction o f t o n g u e f e a t u r e s i s i m p o r t a n t f o r computerized tongue diagnosis.Although many issues of standardization and quantification problems have been resolved by these methods, there are still some difficulties because of the limitations of images captured by traditional CCD cameras.For example, the principal difficulties are that when these images are used it is difficult to distinguish in RGB colour space between the tongue and neighbouring tissues that have a similar colour.It is difficult to distinguish between tongue coating and tongue substance and to discrimination the colour automatically and quantificationally.The thickness and transparence of the sublingual mucosa covering the sublingual veins may change due to the different degrees of varicosity.For these reasons, current methods of tongue feature extraction or diagnosis perform well only on tongue images acquired under some special conditions and often fail when the quality of image is less than ideal.Therefore, automatic tongue diagnosis becomes difficult due to the limited information of common digital images, the variation of the illumination, etc.This will undoubtedly limit the applications of these approaches in clinical medicine.Hyperspectral imaging technology, which comes from the remote sensing field (Chiou, 1984;Harsanyi & Chang, 1994), may offer a solution for these constraints.Hyperspectral remote sensing exploits the fact that all materials reflect, absorb, and emit electromagnetic energy, at specific wavelengths, in distinctive patterns related to their molecular composition (Manolakis & Shaw, 2002).If we introduce this technology to the biomedical diagnosis field, some pathological changes of organism or tissues can be detected and quantificationally analysed in a new way.Actually, hyperspectral imaging of human tissue have been used for many years for characterization and monitoring of applications in biomedicine (Chaudhari et al., 2005;Demos & Ramsamooj, 2003;Timlin J A, 2004).In our previous work, we have developed a microscopic pushbroom hyperspectral imaging system (MPHI) and apply it to analyse the rat retinal sections (Qingli Li, Wang, Zhang, Xu, & Xue, 2010;Qing Li Li, Xue, Xiao, & Zhang, 2007), human blood cells (Qingli Li, Dai, Liu, & Liu, 2009), etc.All these studies show that the hyperspectral imaging technology has significantly advanced in the area of biomedical imaging and early diagnosis.According to the electromagnetic theory, pathological changes of the object surface have a close relationship with its spectrum (Irigoyen & Herraez, 2003;Somosy, Bognar, Thuroczy, & Koteles, 2002).If hyperspectral imaging technology is introduced to computerized tongue diagnosis, both spatial and spectral data, which contain the pathological information of the tongue surface, can be obtained.Therefore, new algorithms should be developed to take advantage of the rich hyperspectral data not found in traditional CCD-based images.In this chapter, an acousto-optic tunable filter (AOTF) based hyperspectral tongue imaging system as well as its applications in computerized tongue diagnosis are presented.The remainder of the chapter is organized as follows.In Section 2 we describe the setup of the hyperspectral tongue imaging system (HTIS).Section 3 proposes the hyperspectral tongue images and the data preprocessing method.Section 4 briefly introduces the tongue body segmentation method based on hyperspectral tongue images.In Section 5, we describe the tongue colour analysis algorithm based on HTIS.In Section 6, tongue cracks extraction algorithm is presented.In Section 7, sublingual veins extraction algorithm based on the new system is proposed.Our conclusions are presented in Section 8.

Hyperspectral tongue imaging system
Most contemporary research on computerized tongue analysis focuses on operations in the gray-scale or colour [red-green-blue (RGB)] images (K.Q. Wang, Zhang, & Li, 2001).These images are captured by traditional charge coupled device (CCD) cameras which can only provide the spatial and colour information of tongue surface.The information contained in the CCD based images may not sufficient for computerized tongue diagnosis.Different from the traditional CCD cameras, hyperspectral imaging sensors in the reflective region of the spectrum (sometimes referred to as imaging spectrometers) can acquire digital images in many contiguous and very narrow spectral bands that typically span the visible, nearinfrared, and even mid-infrared portions of the spectrum.This enables the construction of an essentially continuous radiance spectrum for every pixel in the scene.Thus, hyperspectral data exploitation makes possible the automatization and quantification of tongue diagnosis based on their spatial and spectral signatures.In the past few years, we have developed the pushbroom hyperspectral tongue imaging (PHTI) system and found some useful applications in tongue diagnosis (Qingli Li, Liu, Xiao, & Xue, 2008;Q L Li, Xue, & Liu, 2008;Q. L. Li, Xue, Wang, & Yue, 2006).This PHTI system can capture tongue images from 400 nm to 800 nm with spectral resolution at 5 nm.The number of the efficient pixel is 652*620 for each single band image.However, this system needs pushbroom in one of the spatial dimensions to get the hyperspectral image of the whole scene, which makes the process time-consuming.To overcome this problem, we developed an acousto-optic tunable filter (AOTF) based hyperspectral tongue imaging system recently.AOTF is a rapid wavelength-scanning solid-state device that operates as a tunable optical band pass filter.The acoustic wave is generated by radio-frequency signals, which are applied to the crystal via an attached piezoelectric transducer (Inoue & Penuelas, 2001).When such a filter is placed in the optical train of a camera, different wavelengths can pass through it as a function of time.Unlike in a grating based instrument, no motion of the imager or object is required to obtain a complete image cube (Gupta, 2003).This new system offers the advantage of having no moving parts, having simple and compact structure, and can be scanned at very high rate which is more suitable for tongue imaging than the pushbroom one.The instrument hardware of the AOTF based HTIS is shown in figure 1.The system consists of a camera lens (Nikon AF-S DX 18-200 mm VR II lens), an AOTF (Brimrose, CVA-200), an AOTF controller (radio-frequency drive unit), a CCD camera (Beijing JionHope Technology Ltd., AM1530), a data collection board, and a computer.From the figure we can see that the instrument has similar configuration compared with the standard CCD cameras commonly used by most contemporary research except that an AOTF adapter was coupled between camera lens and CCD in the new system.The dimension of the AOTF based HTIS is 408 mm 115 (L) × 77 mm (H) × 77 mm (W) as the size of Nikon AF-S DX 18-200 mm VR II lens is 96 mm (L) × 77 mm (Ф), AOTF adapter is 254 mm (L) × 60 mm (H) × 67 mm (W), and CCD is 58 mm (L) × 58 mm (H) × 58 mm (W).The AOTF adapter with Brimrose synthesizer electronics can provide narrow bandwidth, rapid wavelength selection, and intensity control.The high spatial resolution of the AOTF camera is ≥ 100 line pair/mm @ 532 nm.The designed wavelength ranges is 400 -1000 nm and the spectral resolution is 2 -6 nm (2 nm @ 543 nm; 5 nm @ 792 nm).The minimum wavelength selection sweep interval of the AOTF is 20 ns, which can meet the imaging speed demand of tongue.The block diagram of the computer controlled experimental setup for the AOTF-based HTIS is shown in figure 2. The tongue surface is illuminated by the light sources under the control of computer.The reflected light by tongue surface are firstly collected via the camera lens, then diffracted by the AOTF adapter, and imaged on the CCD detector at last.The new system can capture image scenes in contiguous but narrow spectral bands under the control of the AOTF controller.The hyperspectral tongue images provided by the instrument can be visualized as a 3D cube (Figure 3) because of its intrinsic structure, where the cube face is a function of the spatial coordinates and the depth is a function of wavelength.In this case, each spatial point on the face is characterized by its own spectrum (often called spectral signature).This spectrum is directly corresponds to the amount of energy that the tongue represented, as hyperspectral sensors commonly utilize the simple fact that a tongue can emits light in certain frequency bands.Consequently, the hyperspectral tongue image data provides a wealth of information about an image scene which is potentially very helpful to tongue diagnosis.

Hyperspectral tongue images
As one of the most important diagnostic methods in traditional Chinese medicine (TCM), tongue diagnosis inspects tongue to examine the physiological function and pathological changes of human body (Yan et al., 2009).Therefore, it is very important to capture the whole information of tongue surface for computerized tongue diagnosis.Most existing computerized tongue diagnosis methods are using tongue images captured by traditional CCD cameras (colour-based imaging methods).These colour-based images are commonly captured and displayed as a set of three black and white images collected with red (R), green (G), and blue (B) light, i.e., at wavelengths of approximately 630 nm, 545 nm, and 435 nm, respectively (Ballard & Brown, 1982).These wavelengths are chosen to match the spectral response of the human eye (Irigoyen & Herraez, 2003;Ornberg, Woerner, & Edwards, 1999).The relationship between the hyperspectral tongue images and the colourbased RGB images can be presented by figure 4. From the figure it can be seen that the hyperspectral tongue images not only contain the whole information that RGB images contained, but also can provide some information that the RGB images not contained.Although the colour-based tongue image processing is not a very difficult task, it may be neglect some details that dose not exist in the original RGB image in some cases.When the hyperspectral tongue images are used in tongue diagnosis, all bands which cover the wavelength range from 400 nm to 1000 nm will participate in the process.So the hyperspectral-based method can get more information than the colour-based imaging methods.In addition, this is not the only reason that we introduce hyperspectral imaging technology to computerized tongue diagnosis.The technology may also have some other good applications in this field, such as tongue body segmentation, tongue colour discrimination, tongue coating analysis, tongue sublingual veins recognition, etc.Therefore, hyperspectral imaging technology is more useful than traditional CCD cameras in modernization of TCM. Figure 5 illustrate a representative subtotal of hyperspectral images captured at various wavelengths of tongue surface and sublingual veins using the AOTFbased HTIS, respectively.From the figures it can be seen that there are different tongue features in different single band images.

Spectral response calibration
One of the big advantages of the HTIS over conventional imaging techniques is the fact that it is capable of acquiring both spatial and spectral information of tongue surface.The spectral response range of the hyperspectral tongue images covering the whole visible wavelength, it can provide more information than the RGB images captured by the traditional CCD.Therefore, it is possible to identify tongue features automatically based on spectral signatures extracted from the hyperspectral tongue images.As the spectral response of the HTIS is inhomogeneous, some data pre-processing procedure should be performed before the spectra extraction.To calibrate the spectral response of the HTIS, a white plane made of Teflon is selected to be the reference plane.Before tongue image collection, a scene of hyperspectral images of the reference plane is captured.Figure 6 shows a typical intensity curve of a pixel extracted from the hyperspectral data of the reference white plane.This curve can be used to represent the spectral response of the HTIS to some extent according to its optical model and sensor model.We define the gray correction coefficient to calibrate the spectral response of the system.The gray correction coefficient k (i, j; ) of each pixel can be calculated by the following formula: ,; where k (i, j; ) is the gray correction coefficient of pixel (i, j) in band ; DN(i, j; ) is the gray value of pixel (i, j) in band of the reference hyperspectral data; DN(i, j) is the average gray value of pixel (i, j) in all bands, it can be calculated according to the following formula: where N is the total band number of the hyperspectral data.Then, the hyperspectral data of tongue can be calibrated with the gray correction coefficient and the true intensity curve can be extracted.The calibration formula is as follows: where DN' (i, j; ) is the gray value of pixel (i, j) at wavelength after calibration.In order to assess the efficiency of the calibration method, another white plane is selected as the sample and the hyperspectral data is collected under the same working conditions with the reference plane.Figure 7 shows an intensity curve of the same pixel extracted from the hyperspectral data of the sample white plane before and after calibration respectively.From the figure it can be seen that the intensity curve of the sample white plane is close to the theoretical value after calibration.Then we can get the real tongue intensity curve from the hyperspectral tongue images with the spectral response calibration method.

Tongue body segmentation
Currently there are two main issues in computerized tongue analysis.One is the objective representation of tongue's colour, texture and coating with the help of image analysis technology (Zheng, Yang, Zhou, & Wang, 2006).The other is automatic tongue body segmentation.Tongue body segmentation is the primitive work of tongue image analysis.
Recently, the main segmentation algorithms are threshold segmentation algorithm, region grow method, watershed algorithm, BEDC algorithm (B Pang et al., 2005), active contour model (for example the snake method (W.Li et al., 2004)), etc.However, these methods can not give satisfied results in the tongue segmentation application.The reason is that the automatic segmentation of anatomical structures in traditional monochromatic images is often performed using model-based nonrigid registration methods.That is to say, an automatic segmentation of a certain structure can be obtained by registering a labelled model, typically generated in a manual segmentation process, to another data set containing the structure of interest.This registration is difficult and laborious (Crum, Hartkens, & Hill, 2004).This is a problem that might be solved if the variability in the spectra of different tissue types could be used to distinguish between the human tongue and the nontongue biological substances in hyperspectral image space (Z.Liu, Yan, Zhang, & Li, 2007).After capture the hyperspectral tongue images, an algorithm which can simultaneously utilizes both the spatial and spectral information of the hyperspectral tongue imagery data can be used to segment the tongue body.To evaluate the effectiveness of the hyperspectral based method, we use a spectral angle mapper (SAM) based automatic tongue segmentation algorithm to segment tongue body.The SAM is an automated method for directly comparing image spectra to known spectra (usually determined in a lab with a spectrometer) (Kruse, Lefkoff, & Boardman, 1992).This method treats both spectra (the questioned and known) as vectors and calculates the spectral angle between them.This method is insensitive to illumination since the SAM algorithm uses only the vector direction and not the vector length.Figure 8 show the core of the SAM based automatic tongue segmentation algorithm.First, a transformed image cube (the SAM cube) is constructed by applying the SAM algorithm with each pixel in the original N*N cube with each and every other pixel in the original hyperspectral tongue imagery data cube.In other words, band 1 of the transformed SAM cube contains the spectral angle of the spectrum in pixel location (sample 1, line 1) with every other spectrum in the cube.Band 2 of the SAM cube contains the spectral angle of the spectrum in pixel location (1, 2) with every other spectrum in the original cube, etc.Thus, a 'spectrum' from the SAM cube contains information about tongue edge.The spectra of the SAM cube are then analyzed, each in turn, for edge detection.In other words, a one-dimensional edge detection technique is applied to the one-dimensional data of each SAM cube spectrum.The SAM cube spectrum band number corresponds to pixel address in the original hyperspectral tongue imagery data cube.Thus, when an edge is detected in the analysis of an SAM cube spectrum, the 'band number' of the edge is converted to the (sample, line) address of the original hypspectral tongue imagery data cube and a point is plotted on a separate output plane indicating the presence of the edge (Resmini, 2004).So the SAM based automatic tongue segmentation algorithm converts a two-dimensional kernel-based edge detection problem into a series of one-dimensional edge detection problems.To evaluate the new automatic tongue body segmentation method, several hyperspectral tongue images were captured by the AOTF based system.After the pre-processing and calibration of the hyperspectral data, we can extract the transmittance spectra of each pixel in the whole scene.Then the SAM based automatic tongue segmentation algorithm was used to segment tongue body from the hyperspectral images.Figure 9 shows the segment results by the new algorithm.The results demonstrate the efficiency of this approach.

Tongue colour analysis and discrimination
In the area of computer-assisted medical diagnosis, colour image analysis can be a useful aid in standardization and automation.Colour image analysis has been applied to assess the power of new drugs on the spreading of skin erythema (Nischik & Forster, 1997), to the identification of skin tumour border (Hance, Umbaugh, & Moss, 1996), assessment and tracking of wound severity (Hansen, Sparrow, & Kokate, 1997), and matching tongue colour images for tongue diseases diagnosis (C.H. Li & Yuen, 2002), etc.These researches show that different colours of organism have some relationship with its physiological functions and changes.So the colour retrieval from images automatically and quantitatively is very important for the purpose of disease diagnosis and prognosis assessment.
Tongue colour is one of the most important pathological features for computer aided tongue diagnosis system.By the use of digital imaging and processing technology, some physiological information of human body condition can be retrieved by analyzing colour features and colour differences extracted from tongue images (X.Wang & Zhang, 2010).For the CCD based RGB colour tongue images, colour image matching can be performed using colour coordinates or based on colour histograms of images in different colour spaces.Therefore, in order to get consistent and standard colour perception of tongue images for tongue analysis, further colour correction to determine the mapping algorithm between its unknown device-dependent RGB colour spaces and a particular known device-independent colour space is necessarily needed.In the past few years, some algorithms in colour correction of tongue images have been developed, such as polynomial-regression-based algorithm and support-vector regression (SVR)-based algorithm (Zhang, Wang, & Jin, 2005), the optimized tongue colour correction scheme (X.Wang & Zhang, 2010), etc.Although these methods can extract some tongue colour features, there are still some difficulties because of the limitations of RGB images captured by the traditional CCD cameras.The principal difficulties are first that when these images are used it is difficult to distinguish in RGB colour space between the tongue and neighbouring tissues that have a similar colour, second it is difficult to distinguish between tongue coating and tongue substance, and third www.intechopen.com

Hyperspectral Imaging Technology Used in Tongue Diagnosis
123 it is also difficult to discrimination the colour automatically and quantificationally.Finally, colour distortion of tongue image, which is often caused by the inconstancy of lighting conditions, can seriously affect the validity of diagnosis results and furthermore, impair the interchange ability among tongue images captured by different devices.For these reasons, current methods of tongue colour calibration or recognition perform well only on tongue images acquired under some special conditions and often fail when the quality of image is less than ideal (Y.G. Wang et al., 2007).So tongue colour analysis becomes difficult due to the limited information of common digital images, the variation of the illumination, etc.
According to the principle of physics, colours of an object surface have close relationship with its spectrum (Irigoyen & Herraez, 2003).As spectra of an organism in the range of wavelengths of the visible light (among 400 nm-750 nm approximately) completely includes the RGB colour space, spectra can be used to retrieve tongue colours more accurately.When the HTIS is used in tongue diagnosis, new automatic tongue colour calibration and discrimination method based on hyperspectral images can be used to recognize and classify tongue colours according to the spectral signatures rather than the colour values in RGB colour space.As different tongue colours have different reflectance spectral signatures, we use the SAM algorithm to classify tongue colours.This method is insensitive to illumination since the SAM algorithm uses only the vector direction and not the vector length.The result of the SAM tongue colour classification is an image showing the best match at each pixel.
Then different tongue colours can be classified accurately by the SAM algorithm.
In the experiment, 230 scenes hyperspectral tongue images are selected for colour recognition test.The colours of tongue substances are quantized into six classes: light white, pale red, red, maroon, purple and reddish purple, while the colours of tongue coatings include four categories: white, light yellow, yellow and gloom.To evaluate the performance of the tongue colour classification method, four experienced doctors have been asked to label the reference samples, classify all of the 230 tongues using naked eyes, and evaluate the experimental results.Due to the discrepancy between inspection results of different doctors, we treated the consensus as the ground truth.The numbers of tongues corresponding to each category of substances and coatings were given in and Tables 2. The rate of correctness for each category of substances or coatings was defined as the percentage of the number of tongues classified correctly by the proposed method to the number of tongues in this category (Qingli Li & Liu, 2009).From Tables 1 and Tables 2 it can be seen that the proposed method has good performance in terms of the rates of correctness for colour recognition of coatings and substances.Therefore, hyperspectral tongue images can be used to tongue colour analysis and discrimination.This method is effective to reduce the colour difference between images captured using different cameras or under different lighting conditions, as the method recognize and classify tongue colours using their spectral signatures rather than their colour values in RGB colour space.

Tongue cracks extraction and classification
According to the TCM theory, human tongue carries abundant information about the health status of a person.Among the information that collected from tongue surface, the appearance of cracks has been known to have the greatest clinical importance.A crack tongue, also known as a scrotal tongue, is a benign condition characterized by deep grooves (fissures) in the dorsum of the tongue.Tongue cracks refer to the surface of the tongue covered with all kinds of cracks or lines in deep or shallow shape, which are induced by the fusion or separation of the ligular papillae.Normally, the tongue surface should be smooth and soft and show no cracks.When obvious cracks appear on the tongue surface, it suggests the deficiency of Qi-blood and the consumption of Yin by excessive heat and, sometimes, the blood stasis.For instance, a deep crack in the center reaching to the tip reflects hyperactivity of Heart fire (L.L. Liu & Zhang, 2007).In the past few years, many studies have been conducted on tongue cracks extraction and classification methods based on gray or colour images.Pham and Cai (Pham & Cai, 2004) constructed an algorithms to calculate six features on a tongue in order to analyze different types of tongue disease.These features include roughness, amount of fissures, a* and b* (chromatic dimensions of L* a* b* colour space), energy, and entropy functions computed from the gray level cooccurrence matrix.Liu and Zhang et al. (L. L. Liu & Zhang, 2007;L. L. Liu et al., 2008) presented a tongue crack detection scheme based on the wide line detector (WLD) algorithm.There are also other tongue cracks extraction algorithms, such as the multiscale edge detection algorithm (Shen et al., 2003), the 2D Gabor wavelet transform coefficient energy (GWTE) algorithm (Y Zhou, L Shen, & J Yang, 2002), and the tongue texture analysis method based on the gray covariance matrix model (B.Pang et al., 2004).Although these methods can extract tongue texture or cracks correctly from tongue images, some difficulties still exist in analyzing these features in detail because of the limitations of images captured by traditional CCD cameras.In TCM, the appearance of obvious cracks on the tongue surface suggests the development of some pathological changes.According to electromagnetic theory, pathological changes of the object surface have a close relationship with its spectrum (Somosy et al., 2002).If hyperspectral imaging technology is introduced to computerized tongue diagnosis, both spatial and spectral data, which contain the pathological information of the tongue surface, can be obtained.Therefore, tongue cracks can be extracted by some new algorithms which can take advantage of the rich hyperspectral data not found in traditional CCD images (Q.L. Li, Wang, Liu, Sun, & Liu, 2010).Generally, there are three steps in a typical tongue cracks extraction procedure, which is finding, tracking, and linking (L.L. Liu & Zhang, 2007).For hyperspectral tongue images, the spectral signatures should be added to the three stages to improve the extraction accurate.In the cracks finding stage, a target-constrained interference-minimized filter (Ren & Chang, 2000) is used to perform the extraction.The filter casts the detection of a foreground signature mixed with background signatures, as a linearly constrained adaptive beam-forming problem.This technique can be used to determine a vector operator that suppresses the undesired background signatures while enhancing that of the known target signature.After perform the cracks finding procedure, we can get a gradient image with tongue cracks seed images and some noisy pixels.
Although the gradient image displays most connected cracks, some isolated spots and misidentified areas still exist in the image.So the tracking and linking process is performed to track the cracks candidates obtained in the first step.As tongue cracks commonly refer to the surface of the tongue covered with all kinds of connected cracks or lines in deep or shallow shape, a roving window of the Hueckel's operator (Ballard & Brown, 1982) is used to prolong or link the bright pixels on the histogram-sliced image.Then postprocessing is performed to reduce the misclassified pixels.Two morphological characteristics of tongue cracks are selected in this algorithm: length and area.The threshold values of them are set to 10 pixels and 100 pixels according to the evaluation standard commonly used by TCM doctors.Then the extracted cracks will be discarded if its length is very short or its area is too small.The final result shows a binary image to represent the extracted tongue cracks which can be classified easily.In the tongue cracks extraction experiment, four scenes of hyperspectral tongue images with different kind of cracks were selected and processed.
Figure 10 shows the extraction results by the proposed method.From the figure it can be seen that the hyperspectral-based tongue cracks extraction method is effective.(He & Kundu, 1991).Shape can be represented either by its contour or by its region.Contour-based descriptors such as curvature, chain codes, Fourier descriptors, etc. have been widely used as they can preserve the local information that is important in classification of complex shapes (Thakoor, Gao, & Jung, 2007).Tongue cracks can be assumed to be formed by various segments, each of which has a constant curvature.Therefore, curvature can be chosen as the feature of contour-based shape descriptors and the hidden Markov model (HMM) (Thakoor et al., 2007) can be used as the framework of tongue cracks modelling and classification.Then the generalized probabilistic descent (GPD) method (Katagiri, Juang, & Lee, 1998) was used as a training algorithm for the tongue cracks classifier.At last, tongue cracks can be classified into different typical categories.
To evaluate the hyperspectral based tongue cracks classify method, 480 scenes of hyperspectral tongue images were captured by the HTIS from in-patients in hospital.These tongue cracks can be divided into 16 typical classes named Class 1 ~ Class 16 which corresponding to figure 1 (a) ~ (p), respectively.The 16 typical tongue cracks are chosen according to the diagnostic criteria commonly used by TCM doctors (S.Wang, Wang, & Wang, 2001).The numbers of tongue cracks corresponding to each category classified by the proposed algorithm was given in Table 3 (the confusion matrix of classification).The rate of correctness for each category of cracks is defined as the percentage of the number of tongues classified correctly by the proposed method to the number of tongues in this category.From the table it can be seen that the proposed method has good performance in terms of the rates of correctness for tongue cracks classification.As a preliminary research, we just use the most simple classify method here.To classify tongue cracks more accurately, some other  classification method such as support vector machines (SVM) should be used in the future as SVM is a more accurate classifier than the maximum likelihood method and can efficiently analyze hyperspectral data directly in the hyperdimensional feature space without the need for any feature-reduction procedure (Melgani & Bruzzone, 2002).

Sublingual veins analysis
According to TCM diagnostics theory, sublingual veins collateral stem from the base of tongue and connect directly with visceras, especially the heart, spleen, liver and kidney.Under normal conditions, they are bluish purple in colour and moist.These veins travel naturally without meandering, distending or exhibiting any varicosity.The sublingual veins are sensitive indicators of the blood stasis and show the activity of the blood circulation (Chiu et al., 2002).Inspection of sublingual veins can provide valuable insights into the healthy status of human body (Yan et al., 2009).However, the subjective characteristics of the traditional method impede this objective because sublingual vein diagnoses are usually based on detailed visual discrimination, which mainly depends on the subjective analysis of the examiners.Therefore, a quantitative analysis should produce the possibility of evaluating and classifying the severity of blood stasis in a less subjective way and deriving an automatic diagnostic procedure.In recent years, some studies have shown that the accurate extraction of tongue features is important for computerized tongue diagnosis (Bakshi & Pal, 2010;Qingli Li & Liu, 2009;B. Pang et al., 2005;B. Pang et al., 2004).The automatic extraction of sublingual veins from complex scenes should also be foremost solved due to the qualities of segmentation directly influencing the subsequent feature extraction and recognition.Some experiments have been conducted on sublingual images acquired by an ordinary camera under a visible light source or an infrared light source for the extraction of sublingual veins.Takeichi and Sato (Takeichi & Sato, 1997) performed computer-assisted image analyses on the colour of the tongues of 95 medical students to enhance the accuracy and objectivity of sublingual veins inspections for determining blood stasis.This is a prior research on colour of sublingual veins.Then, Chiu et al. (Chiu et al., 2002) developed a computerized inspection system and presented a method to extract the chromatic and geometrical properties of sublingual veins quantitatively.Their system is also a colour based method that can extract the length, width, area, and colour information from the sublingual veins.Afterwards, Yan et al. (Yan et al., 2008) used monochrome industrial CCD with enhanced near infrared sensitivity to capture sublingual vein images.More recent studies by them (Yan et al., 2009) have focused on the pixel-based sublingual vein segmentation algorithm and adaptive sublingual vein segmentation algorithm for colour sublingual images with visible and low contrasts.Although many issues on the standardization and quantification of sublingual veins have been resolved, there are still some difficulties because of the limitations of these kinds of images (Q.Li, Wang, Liu, Guan, & Xu, 2011).For example, the thickness and transparence of the sublingual mucosa covering the sublingual veins may change due to the different degrees of varicosity.This change may lead to the sublingual veins being clear in some sublingual images but blurry in others, the contours of the sublingual veins are difficult to extract.
If the HTIS were used to capture the sublingual vein images, it can provide both the spectral and spatial information of the sublingual veins.The spectra extracted from the hyperspectral sublingual images can be represented as a non-stationary sequence of feature vectors, and the spectral correlation and band-to-band variability can be characterized using some models.For example, the SAM or the hidden Markov model (HMM) can be used to extract the sublingual veins from hyperspectral images.Unlike existing approaches, the new method can recognize sublingual veins using their spectral signatures rather than their gray values.To evaluate the effectiveness of the HTIS on sublingual veins analysis, some scenes of hyperspectral sublingual images were captured.Figure 5 (b) shows some single band images of sublingual veins.Here we use the improved spectral angle mapper algorithm to segment the sublingual veins.Spectral angle mapper (SAM) algorithm is a tool that permits rapid mapping of spectral similarity of one image spectrum to another spectrum (Kruse et al., 1992;Park, Windhama, Lawrencea, & Smitha, 2007).The algorithm determines the spectral similarity between two spectra by calculating the 'angle' between them.The angle between the endmember spectra vector and each pixel vector in N-dimensional space is compared.Smaller angles represent closer matches to the reference spectra.This algorithm can extract the target from hyperspectral images effectively by mapping the spectral similarity.However, the wavelengths often shift several bands with the influence of noise in the real HTIS which lead to some extraction errors.To overcome this disadvantage, an improved spectral angle mapper (ISAM) algorithm is used to extract the sublingual veins.
The ISAM algorithm calculate the spectral angle (SA) not only between the reference spectral vector and the testing spectral vector, but also between the reference spectral vector and the testing spectral vector with shift 2 bands both forward and backward, respectively.Then the maximum was selected as the real SA value between the two vectors as the following formula where j = -2, -1, 0, 1, 2. This method is insensitive to illumination since the ISAM algorithm uses only the vector direction and not the vector length.It also can reduce the wavelengths shift errors effectively.Figure 12 shows the extraction results by the ISAM algorithm with SA = 0.1.From the figure it can be seen that the ISAM algorithm can segment the sublingual veins accurately.As sublingual veins may be correlated with certain diseases (Pham & Cai, 2004), several quantitative features for classification in a computer-aided tongue disease diagnosis is necessary.According to the diagnostic standards of experienced doctors, two quantitative features can be selected after sublingual vein extraction, that is, the breadth feature and the chromatic feature.The breadth feature denotes the breadth measurement of the segmented sublingual veins, the value of which is the indicator of a special disease (Yan et al., 2009).The chromatic feature is another important pathological feature of sublingual veins.The colours used as diagnostic measurements in sublingual vein disease diagnoses are usually compounds of several colours due to the complex disorder caused by some pathological processes.The existing colour analysis mainly utilizes the RGB triple values or hue values of single pixels or the mean of the tongue texture block (C.H. Li & Yuen, 2002;B. Pang et al., 2004), which is difficult due to the limited information of these kinds of images.As presented in the previous section, we can use spectra to retrieve organism colours, as the wavelength range of the hyperspectral tongue imaging system covering the whole visible light includes the RGB colour spaces.Therefore, different colours of sublingual veins can be represented by the corresponding reflectance spectral curves.With these quantitative sublingual vein features, some classifiers, such as Bayesian networks, neural networks, and support vector machines, can be used to model the relationship between these quantitative features and diseases.

Conclusion
Among the four diagnostic processes of TCM: inspection, auscultation and olfaction, inquiry, and pulse feeling and palpation, the examination of tongue is one of the most important approaches for getting significant evidences in diagnosing the patient's health conditions.However, owing to its drawbacks in quantification and standardization, the development of tongue diagnosis is stagnated (Zhang, Wang, Zhang et al., 2005).
Computerized methods for TCM allow researchers to identify required information more efficiently, discover new relationships which are obscured by merely focusing on Western medicine, and bridge the gaps between Western Medicine and TCM (Lukman et al., 2007).Therefore, getting the overall information about tongue surface is very important for computerized tongue diagnosis system.In this chapter, an AOTF based HTIS which can capture hyperspectral images of human tongue at a series of wavelengths is developed and used in tongue diagnosis.The basic principles and instrumental systems of the new system, the data pre-processing method as well as some applications are presented.Compared with the pushbroom hyperspectral tongue imager used in our previous works (Q.Li et al., 2008;Q. L. Li et al., 2006), this new type of hyperspectral tongue imaging system has the advantage of having no moving parts and can be scanned at very high rates.As the hyperspectral tongue images can provide more information than the CCD based images, we can find some successful applications in computerized tongue diagnosis such as tongue body segmentation, tongue colour analysis and discrimination, tongue cracks extraction and classification, sublingual veins analysis, etc.Preliminary experiments show that the AOTFbased hyperspectral tongue imaging system is superior to the traditional CCD based methods because the hyperspectral images can provide more information about the tongue surface.In future studies, we will extract the quantitative features of the tongue surface and find some methods to model the relationship between these features and certain diseases.

Fig. 6 .
Fig. 6.An intensity curve extracted from the hyperspectral data of the white plane.

Fig. 8 .
Fig. 8.The core of the SAM based automatic tongue segmentation algorithm

Fig. 11 .
Fig. 11.Typical tongue crack categoriesAfter tongue cracks extraction, they should be classified into several categories to help doctors to diagnose disease.According to the TCM theory, there are 16 kinds of typical tongue cracks as shown in figure11(named as Ba zi, Duo dian, Gui, Ju chi, Lai she, Qu chong, Qu she, Shi xin, Shui zi, Si zhi, Tai Yang, Xuan zhen, Yi zi, Yu gu, and Zuoyou pie cracks, respectively).These typical tongue cracks can be regarded as objects with different shapes which commonly observed in the clinical tongue diagnosis in TCM.We can classify these cracks by a shape classifier which comprises a feature extraction stage and a shape matching stage.Shape classification is a basic problem in computer vision.Its application has been found in various areas such as industrial part identification, target identification, character recognition, and medical diagnosis(He & Kundu, 1991).Shape can be represented either by its contour or by its region.Contour-based descriptors such as curvature, chain codes, Fourier descriptors, etc. have been widely used as they can preserve the local information that is important in classification of complex shapes(Thakoor, Gao, & Jung, 2007).Tongue cracks can be assumed to be formed by various segments, each of which has a constant curvature.Therefore, curvature can be chosen as the feature of contour-based shape descriptors and the hidden Markov model (HMM)(Thakoor et al., 2007) can be used as the

Fig. 12 .
Fig. 12. Sublingual veins extraction results.(a)~(d) false colour images composed by single band images at 630 nm, 540 nm, and 430 nm as R, G, and B channels.(e)~(h) the extraction results by the ISAM algorithm.

Table 1 .
Colour recognition results and rates of correctness for different categories of substances

Table 2 .
Color recognition results and rates of correctness for different categories of coatings

Table 3 .
Tongue cracks classification results and rate of correctness for each category