Classifying by Bayesian Method and Some Applications

Abstract

This chapter sums up and proposes some results related to classification problem by Bayesian method. We present the classification principle, Bayes error, and establish its relationship with other measures. The determination for Bayes error in reality for one and multi-dimensions is also considered. Based on training set and the object that we need to classify, an algorithm to determine the prior probability that can make to reduce Bayes error is proposed. This algorithm has been performed by the MATLAB procedure that can be applied well with real data. The proposed algorithm is applied in three domains: biology, medicine, and economics through specific problems. With different characteristics of applied data sets, the proposed algorithm always gives the best results in comparison to the existing ones. Furthermore, the examples show the feasibility and potential application in reality of the researched problem.

1. Introduction

Classification problem is one of the main subdomains of discriminant analysis and closely related to many fields in statistics. Classification is to assign an element to the appropriate population in a set of known populations based on certain observed variables. It is an important development direction of multivariate statistics and has applications in many different fields [25, 27]. Recently, this problem is interested by many statisticians in both theories and applied areas [14–18, 22–25]. According to Tai [22], we have four main methods to solve the classification problem: Fisher method [6, 12], logistic regression method [8], support vector machine (SVM) method [3], and Bayesian method [17]. Because Bayesian method does not require normal condition for data and can classify for two and more populations it has many advantages [22–25]. Therefore, it has been used by many scientists in their researches.

Given k populations {w_i}, with probability density functions (pdfs) and the prior probabilities respectively {f_i} and {q_i}, i = 1, 2,…, k, where $q_i\in (0;1),{\displaystyle {\displaystyle \sum _{i=1}^k}{q}_i=1.}$ Pham–Gia et al. [17] used the maximum function of pdfs as a tool to study about Bayesian method and obtained important results. The classification principle and Bayes error were established based on the g_max(x) = max{q₁f₁(x), q₂f₂(x), …, q_kf_k(x)}. The relationship between the upper and lower bounds of the Bayes error and the L¹—distance of the pdfs and the overlap coefficient of the pdfs—were established. The function g_max(x) played a very important role in the classification problem by Bayesian method and Pham–Gia et al. [17] continued to do research on it. Using the MATLAB software, Pham–Gia et al. [18] succeeded in identifying g_max(x) for some cases of the bivariate normal distribution. With similar development, Tai [22] has proposed the L¹—distance of the {q_if_i(x)}—and established its relationship with Bayes error. This distance is also used to calculate Bayes error as well as to classify new element. This research has been applied in classifying ability to repay debt of bank customers. However, we think that the survey of two Bayesian approach relevant research was not yet completed. There are some relations between Bayes error and other statistical measures.

Bayesian method has many advantages. However, to our knowledge, the field of applications of this method in practice is narrower than other methods. We can find many applications in banking and medicine using Fisher method, SVM method, logistic method [1, 3, 8, 12]. Recently, all statistics software can effectively and quickly process the classification of large data sets and multivariate statistics using either three of the methods mentioned above, whereas the Bayesian method does not have this advantage. The cause of this problem is the ambiguity in determining prior probability, in estimating pdfs, and the complexity in calculating Bayes error. Although all these issues have been discussed by many authors, the optimal methods have yet to be found [22, 25]. In this chapter, we consider to estimate the pdf and to calculate Bayes error to apply in reality. We will present the problem on how to determine the prior probability in this chapter. In case of noninformation, we normally choose prior probabilities by uniform distribution. If we have some types of past data or training set, the prior probabilities are estimated either by Laplace method: q_i = (n_i + n/k)/(N + n) or by the frequencies of the sample: q_i = n_i/N, where n_i and N are the number of elements in the ith population and training set, respectively, n is the number of dimensions, and k is the number of groups. The above-mentioned approaches have been studied and applied by many authors [14, 15, 22, 25]. We will also propose an algorithm to determine prior probability based on the training set, classified objective, and fuzzy cluster analysis. The proposed algorithm is applied in some specific problems of biology, medicine, and economics and has advantages over existing approaches. All calculations are performed by MATLAB procedures.

The next section of this chapter is structured as follows. Section 2 presents the classification principle and Bayes error. Some results of the Bayes error are also established in this section. Section 3 resolves the related problems in real application of the Bayes method. There are estimation of pdfs and determination of Bayes error in case of one dimension and multidimension. This section also proposes an algorithm to determine prior probability. Section 4 applies the proposed algorithm in real problems and compares outcome results to those obtained using existing approaches. Section 5 concludes this chapter.

2. Classifying by Bayesian method

The classification problem by Bayesian method has been presented in many documents [15, 16, 27], where the classification principle and the Bayes error are established based on Bayes theorem. In this section, we present them via the maximum function of q_if_i(x), i = 1, 2, …, k that they have advantages over existing approaches in real application [17, 18, 21–25]. This section also establishes the upper and lower bounds of the Bayes error and the relationships of Bayes error with other measures in statistical pattern recognition.

2.1. Classification principle and Bayes error

Given k populations w₁, w₂, …, w_k with q_i ∈ (0;1) and f_i(x) are the prior probability and pdf of ith population, respectively, i = 1, 2, …, k. According to Pham–Gia et al. [17], element x₀ will be assigned to w_i if

$g_i(x_0)=g_{\max }(x_0),i=1,2,\dots ,k$ E1

where $g_i(x)=q_i{f}_i(x),g_{\max }(x)=\max \{q_1{f}_1(x),q_2{f}_2(x),\dots ,q_k{f}_k(x)\}\text{.}$

Bayes error is given by the formula:

$P{e}_{1,2,\dots ,k}^{(q)}={\displaystyle {\displaystyle \sum _{i=1}^k}{\displaystyle {\displaystyle \underset{R^n\backslash R_i^n}{\int }}{q}_i{f}_i}dx}=1-{\displaystyle {\displaystyle \sum _{i=1}^k}{\displaystyle {\displaystyle \underset{R_i^n}{\int }}{q}_i{f}_i(x)dx}},$ E2

where $R_i^n=\{x|q_i{f}_i(x)>q_j{f}_j(x),\forall i\ne j,i,j=1,2,\dots ,k\},(q)=(q_1,q_2,\dots ,q_k)\text{.}$

From Eq. (2), we can prove the following result:

$\begin{array}{lll}P{e}_{1,2,\dots ,k}^{(q)}\hfill & =\hfill & {\displaystyle {\displaystyle \sum _{j=1}^k}{\displaystyle {\displaystyle \underset{R^n\backslash R_j^n}{\int }}{q}_j{f}_j(x)dx}}\hfill \\ \hfill & =\hfill & {\displaystyle {\displaystyle \sum _{j=1}^k}\left[{\displaystyle {\displaystyle \underset{R^n}{\int }}{q}_j{f}_j(x)dx-{\displaystyle {\displaystyle \underset{R_j^n}{\int }}{\displaystyle \underset{1\le l\le k}{\max }}\left\{q_l{f}_l(x)\right\}dx}}\right]}\hfill \\ \hfill & =\hfill & \underset{R^n}{\int }\sum _{j=1}^k{q}_j{f}_j(x)dx-\sum _{j=1}^k\underset{R_j^n}{\int }\underset{1\le l\le k}{\max }\left\{q_l{f}_l(x)\right\}dx\hfill \\ \hfill & =\hfill & 1-{\displaystyle {\displaystyle \underset{R^n}{\int }}{\displaystyle \underset{1\le l\le k}{\max }}\left\{q_l{f}_l(x)\right\}dx}\hfill \end{array}$ E20

or

$P{e}_{1,2,\dots ,k}^{(q)}=1-{\displaystyle {\displaystyle \underset{R^n}{\int }}{g}_{\max }(x)dx\text{.}}$ E3

The correct probability is determined by $C{e}_{1,2,\dots ,k}^{(q)}=1-P{e}_{1,2,\dots ,k}^{(q)}\text{.}$

For k = 2, we have

$P{e}_{1,2}^{(q,1-q)}={\displaystyle {\displaystyle \underset{R^n}{\int }}\min \{q{f}_1(x),(1-q)f_2(x)\}dx={\lambda }_{1,2}^{(q,1-q)}=\frac{1}{2}[1-{\Vert q{f}_1,(1-q)f_2\Vert }_1]},$ E4

where

${\lambda }_{1,2}^{(q,1-q)}$ is the overlap area measure of qf₁(x) and (1−q)f₂(x) and ${\Vert q{f}_1,(1-q)f_2\Vert }_1={\displaystyle {\displaystyle \underset{R^n}{\int }}|}q{f}_1(x)-(1-q)f_2(x)|dx\text{.}$

2.2. Some results about Bayes error

Theorem 1. Let f_i(x), i =1, 2, …, k, k ≥ 3 be k pdfs defined on $R^n,n\ge 1,q_i\in (0;1)\text{.}$ We have the relationships of Bayes error with other measures as follow:

1. $P{e}_{1,2,\dots ,k}^{(q)}\le 1-\frac{1}{k-1}\left(1-{\displaystyle {\displaystyle \prod _{j=1}^k}{q}_j^{{\alpha }_j}{D}_T{(f_1,f_2,\dots ,f_k)}^{\alpha }}\right),$ E5

2. $P{e}_{1,2,\dots ,k}^{(q)}\le {\displaystyle {\displaystyle \sum _{i<j}}{q}_i^{\beta }{q}_j^{1-\beta }{D}_T{(f_i,f_j)}^{(\beta ,1-\beta )}},$ E6

3. $\{(k-1)-{\displaystyle {\displaystyle \sum _i}{\displaystyle {\displaystyle \sum _j}{\Vert g_i,g_j\Vert }_1\}/k\le P{e}_{1,2,\dots ,k}^{(q)}\le 1-(1/2){\displaystyle \underset{i<j}{\max }}}}\{{\Vert g_i,g_j\Vert }_1\}-{\displaystyle \underset{i}{\min }}\{q_i\},$ E7

4. $0\le P{e}_{1,2,\dots ,k}^{(q)}\le {\max }_i\left\{q_i\right\},$ E8

where

$\alpha =({\alpha }_1,{\alpha }_2,\dots ,{\alpha }_k);{\alpha }_j,\beta \in (0,1),{\displaystyle {\displaystyle \sum _{j=1}^k}{\alpha }_j=1}$ , i, j = 1, 2, …, k, and

$D_T{(f_1,f_2,\dots ,f_k)}^{\alpha }={\displaystyle {\displaystyle \underset{R^n}{\int }}{\displaystyle {\displaystyle \prod _{j=1}^k}{[f_j(x)]}^{{\alpha }_j}dx}}$ is affinity of Toussaint [26].

Proof:

1. For each j = 1,2,…,k, we have

   ${\left({\displaystyle {\displaystyle \sum _{j=1}^k}{q}_j{f}_j}\right)}^{{\alpha }_i}\ge {\left(q_i{f}_i\right)}^{{\alpha }_i},i=1,2,\dots ,k\text{.}$ E21

   Therefore,

   ${\left({\displaystyle {\displaystyle \sum _{j=1}^k}{q}_j{f}_j}\right)}^{{\alpha }_1+{\alpha }_2+\dots +{\alpha }_k}\ge {\displaystyle {\displaystyle \prod _{j=}^k}{\left(q_j{f}_j\right)}^{{\alpha }_j}}\iff {\displaystyle {\displaystyle \sum _{j=1}^k}{q}_j{f}_j}\ge {\displaystyle {\displaystyle \prod _{j=}^k}{\left(q_j{f}_j\right)}^{{\alpha }_j}}\text{.}$ E9

   On the other hand,

   ${\left({\displaystyle \underset{1\le j\le k}{\min }}\left\{q_j{f}_j\right\}\right)}^{{\alpha }_1}\le {\left(q_1{f}_1\right)}^{{\alpha }_1},\dots \dots ,{\left({\displaystyle \underset{1\le j\le k}{\min }}\left\{q_j{f}_j\right\}\right)}^{{\alpha }_k}\le {\left(q_k{f}_k\right)}^{{\alpha }_k},$ E22

   So

   ${\left({\displaystyle \underset{1\le j\le k}{\min }}\left\{q_j{f}_j\right\}\right)}^{{\alpha }_1+\cdots +{\alpha }_k}\le {\displaystyle {\displaystyle \prod _{j=1}^k}{\left(q_j{f}_j\right)}^{{\alpha }_j}}\text{.}$ E23

   or

   ${\displaystyle \underset{1\le j\le k}{\min }}\left\{q_j{f}_j\right\}\le {\displaystyle {\displaystyle \prod _{j=1}^k}{\left(q_j{f}_j\right)}^{{\alpha }_j}}\text{.}$ E10

   Combining Eqs. (9) and (10), we obtain

   $0\le {\displaystyle {\displaystyle \sum _{j=1}^k}{q}_j{f}_j-{\displaystyle {\displaystyle \prod _{j=1}^k}{\left(q_j{f}_j\right)}^{{\alpha }_j}\le }}{\displaystyle {\displaystyle \sum _{j=1}^k}{q}_j{f}_j-{\displaystyle \underset{1\le j\le k}{\min }}\left\{q_j{f}_j\right\}}\text{.}$ E24

   Because ${\displaystyle {\displaystyle \sum _{j=1}^k}{q}_j{f}_j-{\displaystyle \underset{1\le j\le k}{\min }}\left\{q_j{f}_j\right\}}$ includes (k−1) terms, we have

   ${\displaystyle {\displaystyle \sum _{j=1}^k}{q}_j{f}_j-{\displaystyle \underset{1\le j\le k}{\min }}\left\{q_j{f}_j\right\}}\le (k-1){\displaystyle \underset{1\le j\le k}{\max }}\left\{q_j{f}_j\right\}\text{.}$ E25

   Thus,

   $0\le {\displaystyle {\displaystyle \sum _{j=1}^k}{q}_j{f}_j-{\displaystyle {\displaystyle \prod _{j=1}^k}{\left(q_j{f}_j\right)}^{{\alpha }_j}\le }}(k-1){\displaystyle \underset{1\le j\le k}{\text{max}}}\left\{q_j{f}_j\right\}\text{.}$ E26

   Integrating the above relation, we obtain:

   $1-{\displaystyle {\displaystyle \prod _{j=1}^k}{q}_j^{{\alpha }_j}{D}_T{(f_1,f_2,\dots ,f_k)}^{\alpha }\le (k-1){\displaystyle {\displaystyle \underset{R^n}{\int }}{g}_{\max }(x)dx}}\text{.}$ E11

   Using ${\displaystyle {\displaystyle \underset{R^n}{\int }}{g}_{\max }(x)=1-P{e}_{1,2,\dots ,k}^{(q)}}$ for Eq. (11), we have Eq. (5).

2. From Eq. (2), we have

   $\begin{array}{ll}P{e}_{1,2,\dots ,k}^{(q)}& ={\displaystyle {\displaystyle \sum _{j=1}^k}{\displaystyle {\displaystyle \underset{R^n\backslash R_j^n}{\int }}{q}_j{f}_j(x)dx}}\\ & =\sum _{j=1}^k\sum _{j\ne i}\underset{R_j^n}{\int }\min \{q_i{f}_i(x),q_j{f}_j(x)\}dx\\ & ={\displaystyle {\displaystyle \sum _{i<j}}{\displaystyle {\displaystyle \underset{R_i^n}{\int }}\min \{q_i{f}_i(x),}}{q}_j{f}_j(x)\}dx\text{.}\end{array}$ E27

   Since

   ${\left[\min \right\{q_i{f}_i(x),q_j{f}_j(x)\left\}\right]}^{\beta }\le {\left(q_i{f}_i\right)}^{\beta }\text{and}{\left[\min \right\{q_i{f}_i(x),q_j{f}_j(x)\left\}\right]}^{1-\beta }\le {\left(q_i{f}_i\right)}^{1-\beta },$ E28

   then

   $\min \{q_i{f}_i(x),q_j{f}_j(x)\}\le {\left(q_i{f}_i\right)}^{\beta }{\left(q_j{f}_j\right)}^{1-\beta }\text{.}$ E29

   Integrating the above inequality, we obtain:

   $P{e}_{1,2,\dots ,k}^{(q)}\le {\displaystyle {\displaystyle \sum _{i<j}}{\displaystyle {\displaystyle \underset{R_i^n}{\int }}[{(q_i{f}_i(x))}^{\beta }}}{(q_j{f}_j(x))}^{1-\beta }]dx\le {\displaystyle {\displaystyle \sum _{i<j}}{q}_i^{\beta }{q}_j^{1-\beta }{D}_T{(f_i,f_j)}^{(\beta ,1-\beta )}}dx\text{.}$ E30

3. We have

   ${\displaystyle {\displaystyle \underset{R^n}{\int }}\max \{g_1(x),g_2(x),\dots ,g_k(x)\}dx\ge {\displaystyle \underset{i<j}{\max }}}{\displaystyle {\displaystyle \underset{R^n}{\int }}\max \{g_i(x),g_j(x)\}dx}$ E31

   On the other hand,

   $\begin{array}{lll}{\displaystyle \underset{i<j}{\max }}\left\{{\displaystyle {\displaystyle \underset{R^n}{\int }}\max \{g_i(x),g_j(x)\}dx}\right\}\hfill & =\hfill & {\displaystyle \underset{i<j}{\max }}\left\{\frac{1}{2}{\Vert g_i,g_j\Vert }_1+\frac{1}{2}\left(q_i+q_j\right)\right\}\hfill \\ & \hspace{0.17em}\hspace{1em}\ge {\displaystyle \underset{i<j}{\max }}\left\{\frac{1}{2}{\Vert g_i,g_j\Vert }_1\right\}+{\displaystyle \underset{i<j}{\min }}\left\{\frac{1}{2}\left(q_i+q_j\right)\right\}\hfill \\ \hfill & \hspace{0.17em}\hspace{1em}\ge \hfill & {\displaystyle \underset{i<j}{\max }}\left\{\frac{1}{2}{\Vert g_i,g_j\Vert }_1\right\}+{\displaystyle \underset{i<j}{\min }}\left\{\left(q_1,q_2,\dots ,q_k\right)\right\}.\hfill \end{array}$ E32

   Hence,

   ${\displaystyle {\displaystyle \underset{R^n}{\int }}{g}_{\max }(x)dx\ge \frac{1}{2}{\displaystyle \underset{i<j}{\max }}\left\{{\Vert g_i,g_j\Vert }_1\right\}+{\displaystyle \underset{i<j}{\min }}\left\{\left(q_1,q_2,\dots ,q_k\right)\right\}}\text{.}$ E12

   We also have $\begin{array}{r}{\displaystyle {\displaystyle \sum _{i<}}{\displaystyle {\displaystyle \sum _j}\left|g_i-g_j\right|\ge }}{\displaystyle {\displaystyle \sum _{j=1}^k}\left[\max \left\{g_1,g_2,\cdots g_k\right\}-g_j\right]}\\ =k\left[\max \left\{g_1,g_2,\cdots g_k\right\}\right]-{\displaystyle {\displaystyle \sum _{j=1}^k}{g}_j}\end{array}$

   Therefore,

   $\max \left\{g_1,g_2,\cdots g_k\right\}\le \frac{1}{k}{\displaystyle {\displaystyle \sum _{i<}}{\displaystyle {\displaystyle \sum _j}\left|g_i-g_j\right|}+\frac{1}{k}{\displaystyle {\displaystyle \sum _{j=1}^k}{g}_j}}\text{.}$ E13

   Since

   ${\displaystyle {\displaystyle \underset{R^n}{\int }}{g}_i(x)dx=q_i}\text{and}{\displaystyle {\displaystyle \sum _{i=1}^k}{q}_i=1}$ , the inequality Eq. (13) becomes:

   ${\displaystyle {\displaystyle \underset{R^n}{\int }}{g}_{\max }(x)}dx\le \frac{1}{k}{\displaystyle {\displaystyle \sum _{i<j}}{\Vert g_i,g_j\Vert }_1+\frac{1}{k}}\text{.}$ E14

   Replacing ${\displaystyle {\displaystyle \underset{R^n}{\int }}{g}_{\max }(x)=1-P{e}_{1,2,\dots ,k}^{(q)}}$ to Eqs. (12) and (14), we have Eq. (7).

4. We have

   $q_i{f}_i(x)\le \max \left\{q_1{f}_1(x),q_2{f}_2(x),\dots ,q_k{f}_k(x)\right\}\le {\displaystyle {\displaystyle \sum _{i=1}^k}{q}_i{f}_i(x)}$ for all i = 1,…,k.

   Integrating the above relation, we obtain:

   $q_i\le {\displaystyle {\displaystyle \underset{R^n}{\int }}{g}_{\max }(x)dx\le 1}\text{.}$ E33

   Above inequality is true for all i = 1,…,k, so

   $\max \left\{q_i\right\}\le {\displaystyle {\displaystyle \underset{R^n}{\int }}{g}_{\max }(x)dx\le 1}\text{.}$ E34

Replacing ${\displaystyle {\displaystyle \underset{R^n}{\int }}{g}_{\max }(x)=1-P{e}_{1,2,\dots ,k}^{(q)}}$ in above relation, we have Eq. (8).

From the result of Eqs. (5) and (6), with ${\alpha }_1={\alpha }_2=\dots ={\alpha }_k=1/k,$ , we have the relationship between Bayes error and affinity of Matusita [11]. Especially, when k = 2, we have the relationship between $P{e}_{1,2}^{(q,1-q)}$ and Hellinger’s distance.

In addition, we also have the relation between Bayes error and overlap coefficients as well as L¹–distance of {g₁(x), g₂(x), …, g_k(x)} (see Ref. [22]). For special case: q₁ = q₂ = … = q_k = 1/k, we had established expressions about relations between Bayes error and L¹–distance of {f₁(x), f₂(x), …, f_k(x)}, $P{e}_{1,2,\dots ,k}^{(1/k)}$ and $P{e}_{1,2,\dots ,k+1}^{(1/(k+1))}$ (see Ref. [17]).

3. Related problems in applying of Bayesian method

To apply Bayesian method in reality, we have to resolve three main problems: (i) Determine prior probability, (ii) compute Bayes error, and (iii) estimate pdfs. In this section, we propose an algorithm to solve for (i) based on fuzzy cluster analysis and classified objective that can reduces Bayes error in comparing with traditional approaches. For (ii), Bayes error is established by closed expression for general case and determine it by an algorithm to find maximum function of g_i(x), i = 1, 2, …, k for one dimension case. The quasi-Monte Carlo method is proposed to compute Bayes error in this section. For (iii), we review the problem to estimate pdfs by kernel function method where the bandwidth parameter and kernel function are specified.

3.1. Prior probability

In the n-dimensions space, given N populations $N^{(0)}=\{W_1^{(0)},W_2^{(0)},\dots ,W_N^{(0)}\}$ with data set Z = [z_ij]_nxN. Let matrix $U={\left[{\mu }_{ik}\right]}_{c\times n}$ , where μ_ik is probability of the kth element belonging to w_i. We have ${\mu }_{ik}\in [0,1]$ and satisfies the following conditions:

${\displaystyle {\displaystyle \sum _{i=1}^c}{\mu }_{ik}}=1,0<{\displaystyle {\displaystyle \sum _{k=1}^N}{\mu }_{ik}}<N,1\le i\le c,1\le k\le N\text{.}$ E35

We call

$M_{zc}=\left\{U={\left[{\mu }_{ik}\right]}_{cxN}|{\mu }_{ik}\in \left[0,1\right],\forall i,k;{\displaystyle {\displaystyle \sum _{i=1}^c}{\mu }_{ik}}=1,\forall k;0<{\displaystyle {\displaystyle \sum _{k=1}^N}{\mu }_{ik}},\forall i\right\}$ E15

be fuzzy partitioning space of k populations,

$D_{ikA}^2={\Vert z_k-{\text{v}}_i\Vert }_{\text{A}}^2={\left(z_k-{\text{v}}_i\right)}^{\text{T}}\text{A}\left(z_k-{\text{v}}_i\right)$ is the matrix whose element $d_{ik}^2$ is the square of distance from the object z_k to the ith representative population. This representative is computed by the following formula:

${\text{v}}_i=\frac{{\displaystyle {\displaystyle \sum _{k=1}^N}{({\mu }_{ik})}^m{\text{z}}_k}}{{\displaystyle {\displaystyle \sum _{k=1}^N}{({\mu }_{ik})}^m}},1\le i\le c,$ E16

where m ∈ [1,∞) is the fuzziness parameter.

Given the data set Z including c known populations w_1, w₂, …, w_c. Assume x₀ is an object that we need to classify. To identify the prior probabilities when classifying x₀, we propose the following prior probability by fuzzy clustering (PPC) algorithm:

Algorithm 1. Determining prior probability by fuzzy clustering (PPC); Input: The data set Z = ${[z_{ij}]}_{n\times N}$ of c populations {w1, w2, …, wc}, x0, ε, m and the initial partition matrix $U=U^{\left(0\right)}={[{\mu }_{ij}]}_{c\times N+1}$ , where μij = 1 if the jth object belongs to the wi and μij = 0 for the opposite, $i={\displaystyle \overline{1,c}};j={\displaystyle \overline{1,N}},{\mu }_{ij}=1/c$ for j = N + 1.; Output: The prior probability ${\mu }_{i(N+1)},i=1,2,\dots c\text{.}$

Repeat: Find the representative object of wi: ${\text{v}}_i=\frac{{\displaystyle {\displaystyle \sum _{k=1}^N}{({\mu }_{ik})}^m{\text{z}}_k}}{{\displaystyle {\displaystyle \sum _{k=1}^N}{({\mu }_{ik})}^m}},1\le i\le c$; Compute the matrix ${\left[D_{ik}\right]}_{c\times N+1}$ (the pairwise distance between objects and representative objects).; Update the new partition matrix U^(new) by the following principle: If Dik > 0 for all $i=1,2,\dots ,c;k=1,2,\dots ,N+1$ then; ${\mu }_{ik}{}^{\left(\text{new}\right)}=\frac{1}{{\displaystyle {\displaystyle \sum _{j=1}^c}{\left(D_{ik}/D_{jk}\right)}^{2/\left(m-1\right)}}},i\ne j=1,2,\dots ,c$; Else, ${\mu }_{ik}^{\left(\text{new}\right)}=0$; End; Compute $S=\Vert U^{\left(\text{new}\right)}-U\Vert ={\text{max}}_{ik}\left(\left|{\mu }_{ik}^{\left(\text{new}\right)}-{\mu }_{ik}\right|\right)$; $U=U^{\left(\text{new}\right)}$; Until S < ε; The prior probability ${\mu }_{i(N+1)},i=1,2,\dots c$ (the final column of the matrix U);

In the above algorithm, we have:

1. ε is a really small number and is chosen arbitrarily. The smaller ε is, the more iterations time is taken. In the examples of this chapter, we choose ε = 0.001.

2. The distance matrix D_ik depends on the norm-inducing matrix A. When A = I, D_ik is the matrix of Euclidean distances. Besides, there are several choices of A, such as diagonal matrix or the inverse of the covariance matrix. In this chapter, we chose the Euclidean distances in the numerical examples and applications.

3. m is the fuzziness parameter, when m = 1, the fuzzy clustering becomes the nonfuzzy clustering. When m → ∞, the partition becomes completely fuzzy μ_ik = 1/c. The determining of this parameter, which affects the analysis result, is difficult. Even though Yu et al. [28] proposed two rules to determine the supermom of m for clustering problems, the searching of the specific m was done by meshing method (see [2, 4, 5, 9] for more details). By this process, the best m among several of given values will be chosen. In this chapter, m is also identified by meshing method for the classification problem. The best integer m between 2 and 10 will be used.

At the end of the PPC algorithm, we obtain the prior probabilities of x₀ based on the last column of the partition matrix U $({\mu }_{i(N+1)},i=1,2,\dots c)$ . The PPC algorithm helps us determine the prior probabilities via the closeness degree between the classified object and the populations. Each object will receive its suitable prior probabilities.

In this chapter, Bayesian method with prior probabilities calculated by the uniform distribution approach, the ratio of samples approach, the Laplace approach, and the proposed PPC algorithm approach are respectively called BayesU, BayesR, BayesL, and BayesC.

Example 1. Given the studied marks (scale 10 grading system) of 20 students. Among them, nine students have marks that are lower than 5 (w₁: fail the exam) and 11 students have marks that are higher than 5 (w₂: pass the exam). The data are given in Table 1.

Objects	Marks	Groups	Objects	Marks	Groups
1	0.6	w1	11	5.6	w2
2	1.0	w1	12	6.1	w2
3	1.2	w1	13	6.4	w2
4	1.6	w1	14	6.4	w2
5	2.2	w1	15	7.3	w2
6	2.4	w1	16	8.4	w2
7	2.4	w1	17	9.2	w2
8	3.9	w1	18	9.4	w2
9	4.3	w1	19	9.6	w2
10	5.5	w2	20	9.8	w2

Table 1.

The studied marks of 20 students and the actual classifications.

Assume that we need to classify the ninth object, x₀ = 4.3, to one in two populations. Using the PPC algorithm, we have the following final partition matrix:

$\left(\begin{array}{cccccccccccccccccccc}0.957& 0.973& 0.981& 0.993& 1& 0.997& 0.997& 0.830& 0.321& 0.290& 0.158& 0.1& 0.1& 0.01& 0.009& 0.037& 0.045& 0.054& 0.062& 0.724\\ 0.043& 0.027& 0.019& 0.007& 0& 0.003& 0.003& 0.170& 0.679& 0.710& 0.842& 0.9& 0.9& 0.99& 0.991& 0.963& 0.955& 0.946& 0.938& 0.276\end{array}\right)$ E36

This matrix shows the prior probabilities when assigning the ninth object to w₁ and w₂ are 0.724 and 0.276, respectively. Meanwhile, the prior probabilities determined by BayesU, BayesR, and BayesL are (0.5; 0.5), (0.421; 0.579), and (0.429; 0.571), respectively.

From the data in Table 1, we might estimate the pdfs f₁(x) and f₂(x) and compute the values q₁f₁(x) and q₂f₂(x), where q₁ and q₂ are the calculated prior probabilities. The results of classifying x₀ by four approaches: BayesU, BayesR, BayesL, and BayesC are given in Table 2.

Methods	Priors	gmax(x0)	Populations	Bayes errors
BayesU	(0.5; 0.5)	0.0353	2	0.0538
BayesR	(0.421; 0.579)	0.0409	2	0.0558
BayesL	(0.429; 0.571)	0.0403	2	0.0557
BayesC	(0.724; 0.276)	0.0485	1	0.0241

Table 2.

The results when classifying the ninth object.

Because the actual population of x₀ is w₁, only BayesC gives the true result. The Bayes error of BayesC is also the smallest. Thus, in this example, the proposed method improves the drawback of the traditional method in determining the prior probabilities.

3.2. Determining Bayes error

Theorem 2. Let f_i(x), i =1, 2, …, k, k ≥ 3 be k pdfs defined on Rⁿ, n ≥ 1 and let q_i ∈ (0;1),

$\{\begin{array}{l}{R}_1^n=\left\{x\in R^n:q_1{f}_1(x)>q_j{f}_j(x),2\le j\le k\right\},\\ R_k^n=\left\{x\in R^n:q_k{f}_k(x)>q_j{f}_j(x),1\le j\le k\right\},\\ R_l^n=\left\{x\in R^n:q_i{f}_i(x)>q_l{f}_l(x),1\le i\le k,2\le l\le k-1,i\ne l\right\}\text{.}\end{array}$ E17

The Bayes error is determined by

$P{e}_{1,2,\dots ,k}^{(q)}=1-{\displaystyle {\displaystyle \underset{R_1^n}{\int }}{q}_1{f}_1(x)dx}-{\displaystyle {\displaystyle \sum _{l=2}^{k-1}}{\displaystyle {\displaystyle \underset{R_l^n}{\int }}{q}_l{f}_l(x)dx}}-{\displaystyle {\displaystyle \underset{R_k^n}{\int }}{q}_{k}f{}_k(x)dx}\text{.}$ E18

Proof:

To obtain Eq. (18), we need to prove two following results:

$R_i^n\cap R_j^n=\phi ,(1\le i\ne j\le k)$ E37

and ${\displaystyle {\displaystyle \bigcup _{i=1}^k}{R}_i^n=R_1^n\cup }{\displaystyle {\displaystyle \bigcup _{i=2}^{k-1}}{R}_i^n}\cup R_k^n=R^n,f_{\max }(x)=f_i(x),\forall x\in R_i^n\text{.}$

Let ${\displaystyle \overline{A}}=R^n\backslash A,$ we have

${{\displaystyle \overline{R}}}_{ij}=\left\{x\in R^n:q_i{f}_i(x)\le q_j{f}_j(x)\right\},R_{ij}=\left\{x\in R^n:q_i{f}_i(x)>q_j{f}_j(x)\right\},\left(1\le i,j\le k\right)\text{.}$ E38

From Eq. (17), we obtain

$R_1^n={\displaystyle {\displaystyle \bigcap _{j=2}^k}{R}_{1j}},R_l^n={\displaystyle {\displaystyle \bigcap _{i\ne k}}{{\displaystyle \overline{R}}}_{il}},\left(2\le l<k\right)\text{.}$ E39

Therefore,

$R_1^n\cap R_l^n=({\displaystyle {\displaystyle \bigcap _{j=2}^k}{R}_{ij})}\cap ({\displaystyle {\displaystyle \bigcap _{i\ne k}}{{\displaystyle \overline{R}}}_{il}})\subset R_{il}\cap {{\displaystyle \overline{R}}}_{1l}=\phi \Rightarrow R_1^n\cap R_l^n=\phi ,\left(2\le l<k\right)\text{.}$ E40

On the other hand, from antithesis style of D’Morgan, we have

${\displaystyle \overline{R_1^n\cup R_l^n}}=({\displaystyle {\displaystyle \bigcup _{j=2}^n}{{\displaystyle \overline{R}}}_{ij})}\cup ({\displaystyle {\displaystyle \bigcup _{i\ne k}}{R}_{il}})\subset {{\displaystyle \overline{R}}}_{il}\cap R_{1l}=\phi \Rightarrow R_1^n\cup R_l^n=R^n,\left(2\le l<k\right)\text{.}$ E41

Similarly,

$R_k^n\cap R_l^n=\phi ,\left(2\le l<k\right),R_1^n\cap R_k^n=\phi ,$ E42

so

$\begin{array}{cc}{\displaystyle {\displaystyle \bigcup _{i=1}^k}{R}_i^n}& =R^n,\cup ({\displaystyle {\displaystyle \bigcup _{l=2}^{k-1}}{R}_l^n})\cup R_k^n=R_1^n\cup ({\displaystyle {\displaystyle \bigcup _{l=2}^{k-1}}{R}_l^n})\cup R_k^n\\ & =({\displaystyle {\displaystyle \bigcup _{l=2}^{k-1}}{R}_1^n\cup R_l^n})\cup ({\displaystyle {\displaystyle \bigcup _{l=2}^{k-1}}{R}_k^n\cup R_l^n})=R^n\cup R^n=R^n\Rightarrow {\displaystyle {\displaystyle \bigcup _{i=1}^k}{R}_i^n}=R^n\text{.}\end{array}$ E43

In addition, from Eq. (17), we can directly find out

$g_{m\text{ax}}(x)=g_i(x),\forall x\in R_i^n,\left(1\le i\le k\right)\text{.}$ E44

For k = 2, q₁ = q₂ = 1/2, we consider the two following special cases:

1. If f₁(x) and f₂(x) are two one-dimension normal pdfs ( $N({\mu }_i,{\sigma }_i),$ i = 1, 2), without loss of generality, we suppose that ${\mu }_1<{\mu }_2$ (for ${\mu }_1\ne {\mu }_2$ ), ${\sigma }_1<{\sigma }_2$ (for ${\sigma }_1\ne {\sigma }_2$ ), then

   $P{e}_{1,2}^{(1/2,1/2)}=\{\begin{array}{l}\frac{1}{2}[\underset{-\infty }{\overset{x_1}{\int }}{f}_2(x)dx+\underset{x_1}{\overset{+\infty }{\int }}{f}_1(x)dx],if{\sigma }_1={\sigma }_2,\\ \frac{1}{2}[\underset{-\infty }{\overset{x_2}{\int }}{f}_1(x)dx+\underset{x_2}{\overset{x_3}{\int }}{f}_2(x)dx+\underset{x_3}{\overset{+\infty }{\int }}{f}_1(x)dx],if{\sigma }_1<{\sigma }_2,\end{array}$ E45

   where

   $\begin{array}{l}{x}_1=\frac{{\mu }_1+{\mu }_2}{2},x_2=\frac{({\mu }_1{\sigma }_2^2-{\mu }_2{\sigma }_1^2)-{\sigma }_1{\sigma }_2\sqrt{{({\mu }_1-{\mu }_2)}^2+K}}{{\sigma }_2^2-{\sigma }_1^2},\\ x_3=\frac{({\mu }_1{\sigma }_2^2-{\mu }_2{\sigma }_1^2)+{\sigma }_1{\sigma }_2\sqrt{{({\mu }_1-{\mu }_2)}^2+K}}{{\sigma }_2^2-{\sigma }_1^2},K=2({\sigma }_2^2-{\sigma }_1^2)\ln \left(\frac{{\sigma }_2}{{\sigma }_1}\right)\ge 0.\end{array}$ E46

   For μ₁ = μ₂ =μ, the above result becomes:

   $P{e}_{1,2}^{(1/2,1/2)}=\{\begin{array}{l}1,\text{ if}{\sigma }_{\text{1}}={\sigma }_2,\\ \frac{1}{2}[{\displaystyle {\displaystyle {\displaystyle \underset{-\infty }{\overset{x_4}{\int }}}}{f}_1(x)dx+{\displaystyle {\displaystyle {\displaystyle \underset{x_4}{\overset{x_5}{\int }}}}{f}_2(x)dx+{\displaystyle {\displaystyle {\displaystyle \underset{x_5}{\overset{+\infty }{\int }}}}{f}_1(x)dx]}}}\text{ if }{\sigma }_{\text{1}}<{\sigma }_2,\end{array}$ E47

   where $x_4=\mu -{\sigma }_1{\sigma }_2\sqrt{E}$ and $x_5=\mu +{\sigma }_1{\sigma }_2\sqrt{E}$ with $E=\frac{2}{{\sigma }_2^2-{\sigma }_1^2}\ln \left(\frac{{\sigma }_2}{{\sigma }_1}\right)\ge 0.$

2. If f₁(x) and f₂(x) are two n-dimension normal pdfs $(N({\mu }_i,{\text{Σ}}_i),n\ge 2,i=1,2)$ then

   $P{e}_{1,2}^{(1/2,1/2)}=\frac{1}{2}\left[{\displaystyle {\displaystyle \underset{R_1}{\int }}{f}_2(x)dx+{\displaystyle {\displaystyle \underset{R_2}{\int }}{f}_1(x)dx}}\right],$ E48

   where

   $\begin{array}{l}{R}_1^n=\left\{x:d(x)\le 0\right\},R_2^n=\left\{x:d(x)>0\right\},\hfill \\ d(x)=[{\mu }_1^T{\left({\text{Σ}}_1\right)}^{-1}-{\mu }_2^T{\left({\text{Σ}}_2\right)}^{-1}]x-\frac{1}{2}{x}^T[{\left({\text{Σ}}_1\right)}^{-1}-{\left({\text{Σ}}_2\right)}^{-1}]x-m,\hfill \\ m=\frac{1}{2}[\ln \frac{|{\text{Σ}}_1|}{|{\text{Σ}}_2|}+{\mu }_1^T{({\text{Σ}}_1)}^{-1}{\mu }_1-{\mu }_2^T{({\text{Σ}}_2)}^{-1}{\mu }_2]\text{.}\hfill \end{array}$ E49

In case of n = 2, d(x) can be straight lines or parabola or ellipses or hyperbola.

3.3. Maximum function in the classification problem

To classify a new element by the principle (1) and to determine Bayes error by the formula (3), we must find g_max(x). Some authors, such as Pham–Gia et al. [15, 17] and Tai [21, 22], have surveyed relationships between g_max(x) with some related quantities of classification problem. The specific expression for g_max(x) in some special case has been found [18]. However, the general expression for all of cases is a complex problem that has not been still found yet.

Given k pdfs f_i(x) and q_i, i = 1, 2, …, k with q₁ + q₂ + …+ q_k = 1 and let g_i(x) = q_if_i(x), g_max(x) = max{g_i(x)}. Now, we take interest in determining g_max(x).

(a) For one dimension

In this case, we can find g_max(x) by the following algorithm:

Algorithm 2. Find the gmax(x) function; Input: gi(x) = qifi(x), where fi(x) and qi are the probability density function and the prior probability of wi, i = 1, 2, …, k, respectively.; Output: The gmax(x) function.

Find all roots of the equations $g_i(x)-g_j(x)=0,i={\displaystyle \overline{1,k-1}},j={\displaystyle \overline{i+1,k}}$ .; Let B be the set of all roots.; For xlm ∈ B (the roof of equation $g_l(x)-g_m(x)=0$ ) do; For p ∈{1,2,…,k}\{l,m} do; If $g_l\left(x_{lm}\right)<g_p\left(x_{lm}\right)$ then $B=B\backslash \left\{x_{lm}\right\}$; End; End; End; Arrange the elements of B in order from smallest to largest: $B=\left\{x_1,x_2,\dots ,x_h\right\},x_1<x_2<\dots <x_h$

(Determine the function gmax(x) in interval (−∞,x1]); For i = 1 to k do; If $g_i\left(x_1-{\epsilon }_1\right)=\max \left\{g_1\left(x_1-{\epsilon }_1\right),g_2\left(x_1-{\epsilon }_1\right),\dots ,g_k\left(x_1-{\epsilon }_1\right)\right\}$ then; $g_{\max }\left(x\right)=g_i\left(x\right)$ , for all x ∈ (−∞,x1]; End; End; (Determine the function gmax (x) in interval $(x_j,x_{j+1}]$ , $j={\displaystyle \overline{1,h-1}}$ ); For i =1 to k do; For j =1 to h-1 do; If $g_i(x_j+{\epsilon }_2)=\max \{g_1(x_j+{\epsilon }_2),g_2(x_j+{\epsilon }_2),\dots ,g_k(x_k+{\epsilon }_2)\}$ then; $g_{\max }(x)=g_i(x)$ , for all $x\in (x_j,x_{j+1}]$; End; End; End; (Determine the function gmax (x) in interval (h,+∞)); For i = 1 to k do; If $g_i\left(x_h+{\epsilon }_3\right)=\max \left\{g_1\left(x_h+{\epsilon }_3\right),g_2\left(x_h+{\epsilon }_3\right),\dots ,g_k\left(x_h+{\epsilon }_3\right)\right\}$ then; $g_{\max }(x)=g_i(x)$ , for all $x\in \left(h,+\infty \right)$; End; End

In the above algorithm, ε₁, ε₂, ε₃ are the positive constants such that:

$x_1+{\epsilon }_1<x_2,x_h-{\epsilon }_3>x_{h-1},x_i-{\epsilon }_2<x_{i-1}\text{and}{x}_i+{\epsilon }_2<x_{i+1}\text{.}$ E50

From this algorithm, we have written a MATLAB code to find the g_max(x). When g_max(x) is determined, we will easily calculate Bayes error by using formula (3), as well as classify a new element by principle (1).

Example 2. Given seven populations having univariate normal pdfs {f₁, f₂,…, f₇} with specific parameters as follows (Figure 1):

Figure 1.

The graph of seven one-dimension normal pdfs, f_max(x) and g_max(x).

$\begin{array}{l}{\mu }_1=0.3,{\mu }_2=4.0,{\mu }_3=9.1,{\mu }_4=1.9,{\mu }_5=5.3,{\mu }_6=8,{\mu }_7=4.8,\\ {\sigma }_1=1.0,{\sigma }_2=1.3,{\sigma }_3=1.4,{\sigma }_4=1.6,{\sigma }_5=2,{\sigma }_6=1.9,{\sigma }_7=\mathrm{2.3.}\end{array}$ E51

Using codes written with $q_i=1/7,g_i(x)=q_i{f}_i(x),i=1,2,\mathrm{..},7,$ we have the results:

$g_{\max }(x)=\{\begin{array}{lll}{g}_1\hfill & \text{if}\hfill & -1.28<x\le 0.99\text{,}\hfill \\ g_2\hfill & \text{if}\hfill & 2.58<x\le 4.89\text{,}\hfill \\ g_3\hfill & \text{if}\hfill & 8.30<x\le 12.52\text{,}\hfill \\ g_4\hfill & \text{if}\hfill & \{-7.86<x\le -1.28\}\cup \{0.99<x\le 2.58\}\text{,}\hfill \\ g_5\hfill & \text{if}\hfill & 4.89<x\le 6.65\text{,}\hfill \\ g_6\hfill & \text{if}\hfill & \{6.65<x\le 8.30\}\cup \{12.52<x\le 23.33\}\text{,}\hfill \\ g_7\hfill & \text{if}\hfill & \{x\le -7.86\}\cup \{x>23.33\}\text{.}\hfill \end{array}$ E52

(b) For multidimension

In multidimension cases, it should be very complicated to obtain closed expression for g_max(x). The difficulty comes from the various forms of the intersection space curves between the pdfs surfaces. This problem has been interested by many authors in Refs. [17, 18, 21–25]. Pham–Gia et al. [18] have attempted to find the function g_max(x); however, it has been only established for some cases of bivariate normal distribution.

Example 3. Given the four bivariate normal pdfs N(μ_i, Σ_i) with the following specific parameters [16]:

$\begin{array}{l}{\mu }_1=\left[\begin{array}{c}40\\ 20\end{array}\right],{\mu }_2=\left[\begin{array}{c}48\\ 24\end{array}\right],{\mu }_3=\left[\begin{array}{c}43\\ 32\end{array}\right],{\mu }_4=\left[\begin{array}{c}38\\ 28\end{array}\right],\\ {\text{Σ}}_1=\left(\begin{array}{cc}35& 18\\ 18& 20\end{array}\right),{\text{Σ}}_1=\left(\begin{array}{cc}28& -20\\ -20& 25\end{array}\right),{\text{Σ}}_1=\left(\begin{array}{cc}15& 25\\ 25& 65\end{array}\right),{\text{Σ}}_1=\left(\begin{array}{cc}5& -10\\ -10& 7\end{array}\right)\end{array}$ E53

With q₁ = 0.25, q₂ = 0.2, q₃ = 0.4, and q₄ = 0.15, we have the graphs of g_i(x) = q_if_i(x) and their intersection curves as shown in Figure 2.

Figure 2.

The graph of three bivariate normal pdfs and their g_max(x).

Here, we do not find the expression of g_max(x). We compute Bayes error instead by taking integration of g_max(x) by quasi-Monte Carlo method [17]. An algorithm for doing calculations has been constructed, and a corresponding MATLAB procedure is used in Section 4.

3.4. Estimate the probability density function

There are many parameter and nonparameter methods to estimate pdfs. In the examples and applications of Section 4, we use the kernel function method, the popular one in practice nowadays. It has the following formula:

${\displaystyle \stackrel{⌢}{f}}(x)=\frac{1}{N{h}_1{h}_2\dots h_n}{\displaystyle {\displaystyle \sum _{i=1}^N}{\displaystyle {\displaystyle \prod _{j=1}^n}{f}_j\left(\frac{x_j-x_{ij}}{h_j}\right)}},$ E19

where x_j, j = 1,2,…,n are variables, x_j, i = 1,2,…,N are the ith data of the jth variable, h_j is the bandwidth parameter for the jth variable, f_j(.) is the kernel function of the jth variable which is usually normal, Epanechnikov, biweight, and triweight. According to this method, the choice of smoothing parameter and the type of kernel function play an important role and affect the result. Although Silverman [20], Martinez and Martinez [10], and some other authors [7, 13, 27] had discussions about this problem, the optimal choice still has not been found yet. In this chapter, the smoothing parameter is from the idea of Scott [19] and the kernel function is the Gaussian one. We have also written the code by MATLAB software to estimate the pdfs in n-dimensions space using this method.

We have written the complete code for the proposed algorithm by MATLAB software. It is applied effectively for the examples of Section 4.

4. Some applications

In this section, we will consider three applications in three domains: biology, medicine, and economics to illustrate for present theories and to test established algorithms. They also show that the proposed algorithm presents more advantages than the existing ones.

Application 1. We consider classification for well-known Iris flower data, which have been presented in many documents like in Ref. [17]. These data are often used to compare the new method and existing ones in classifying. The three varieties of Iris, namely, Setosa (Se), Versicolor (Ve), and Virginica (Vi), have data in four attributes: X1 = sepal length, X2 = sepal width, X3 = petal length, and X4 = petal width.

In this application, the cases of one, two, three and four variables are respectively considered to classify for three groups (Se), (Ve), and (Vi) by Bayesian method with different prior probabilities. The purpose of this performance is to compare the results of BayesC with BayesU, BayesR, and BayesL. Because the numbers of the three groups are equal, and the results of BayesU, BayesR, and BayesL are the same. The correct probability of methods is summarized in Table 3.

Variables	B BayesU = BayesL = BayesR	BayesC
X1	0.667	0.679
X2	0.668	0.579
X3	0.903	0.916
X4	0.815	0.827
X1, X2	0.715	0.807
X1, X3	0.893	0.895
X1, X4	0.807	0.850
X2, X3	0.891	0.898
X2, X4	0.809	0.815
X3, X4	0.843	0.866
X1, X2, X3	0.892	0.919
X1, X2, X4	0.764	0.810
X1, X3, X4	0.762	0.814
X2, X3, X4	0.736	0.822
X1, X2, X3, X4	0.725	0.745

Table 3.

The correct probability (%) in classifying Iris flower.

Table 3 shows that in almost all cases, the results of proposed algorithm are better than those using other algorithms, and in the case using three variables X1, X2, and X3, it gives the best results.

Application 2. This application considers thyroid gland disease (TGD). Thyroid gland is an important and the largest gland in our body. It is responsible for the metabolism and work process of all cells. Some of the common diseases of gland thyroid are hypothyroidism, hyperthyroidism, thyroid nodules, and thyroid cancer. They are dangerous diseases. Recently, the rate of thyroid gland disease has been increasing in some poor countries. Data includes 3772 person with 3541 for ill group (I) and 231 ones for nonill group (NI). Detail for this data is given in http://www.cs.sfu.ca/wangk/ucidata/dataset/thyroid–disease, in which the surveyed variables are Age (X1), Query on thyroxin (X2), Anti-thyroid medication (X3), Sick (X4), Pregnant (X5), Thyroid surgery (X6), Thyroid Stimulating Hormone (X7), Triiodothyronine (X8), Total thyroxin (X9), T4U measured (X10), and Referral source (X11). In this application, this chapter will use random 70% of the data size (2479 elements belong to group I and 162 elements belong to group NI) as the training set to determine significant variables, to estimate pdfs, and to find suitable model. About 30% of the remaining data will be used as test set (1062 elements belong to group I and 69 elements belong to group NI). The result of Bayesian method is also compared to others.

To assess the effect of independent variables in TGD, we build the logistic regression model log(p/1−p) with variables Xi, i = 1, 2, …, 11 (p is the probability of TGD). The analytical results are summarized in Table 4.

Variable	Sig.	Variable	Sig.
X1	0.000	X7	0.304
X2	0.279	X8	0.000
X3	0.998	X9	0.995
X4	0.057	X10	0.999
X5	0.997	X11	0.000
X6	0.997	Const	0.992

Table 4.

Value Sigs of logistic regression model.

In Table 4, the three variables X1, X8, and X11 in bold face have statistical significance in classifying the two groups (I) and (NI) at 5% level, so we use them to classify TGD.

Applying the PPC algorithm for cases of one variable, two variables, and three variables with all prior probabilities, we obtain the results given in Table 5.

Table 5 shows that the correct probability is high, in which BayesC always gives the best result in all three cases of variables. BayesC gives the almost exact result with three variables. We also compare BayesC with existing methods (Fisher, SWM, and logistic) for all the above three cases. All cases show that BayesC is more advantageous than others in reducing Bayes error.

Cases	Variables	BayesU	BayesR	BayesL	BayesC
One variable	X1	91.13	97.47	97.46	97.97
	X8	90.72	98.51	98.50	98.65
	X11	90.53	97.48	97.47	98.19
Two variables	X1, X8	98.73	98.77	98.77	99.78
	X1, X11	98.11	98.65	97.65	99.44
	X8, X11	98.71	98.77	98.77	99.82
Three variables	X1, X8, X11	98.35	98.89	98.89	99.96

Table 5.

The correct probability (%) in classifying TGD by Bayesian method from training set.

Using the best results for each case of methods from Table 6, classifying for test set (1131 elements), we have the results given in Table 7.

Methods	One variable	Two variables	Three variables
Logistic	93.90	93.90	93.90
Fisher	72.30	73.60	71.70
SVM	93.87	93.87	93.87
BayesC	98.65	99.82	99.96

Table 6.

The correct probability (%) for optimal models of methods in classifying TGD.

From Table 7, we see that with the test set, BayesC also gives the best result.

Methods	Correct numbers	False numbers	Correct probability
Logistic	835	296	73.8
Fisher	835	296	73.8
SVM	1062	69	90.9
BayesC	1062	69	93.9

Table 7.

Compare the correct probability (%) in classifying TGD from test set.

Application 3. This application considers the problem of repaying bank debt (RBD) by customers. In bank credit operations, determining the repayment ability of customers is really important. If the lending is too easy, the bank may have bad debt problems. In contrast, the bank will miss a good business. Therefore, in the current years, the classification of credit application on assessing the ability to repay bank debt has been specially studied and has been a difficult problem in Vietnam. In this section, we appraise this ability of companies in Can Tho city (CTC), Vietnam by using the proposed approach. We collect a data on 214 enterprises operating in key sectors as agriculture, industry, and commerce, including 143 cases of good debt (G) and 71 cases of bad debt (B). Data are provided by responsible organizations of CTC. Each company is evaluated by 13 independent variables in the expert opinion. The specific variables are given in Table 8.

Xi	Independent variables	Detail
X1	Financial leverage	Total debt/total equity
X2	Reinvestment	Total debt/total equity
X3	Roe	Net profit/equity
X4	Interest	(Net income + depreciation)/total assets
X5	Floating capital	(Current assets − current liabilities)/total assets
X6	Liquidity	(Cash + Short-term investments)/current liabilities
X7	Profits	Net profit/total assets
X8	Ability	Net sales/Total assets
X9	Size	Logarithm of total assets
X10	Experience	Years in business activity
X11	Agriculture	Agricultural and forestry sector
X12	Industry	Industry and construction
X13	Commerce	Trade and services

Table 8.

The surveyed independent variables.

Because of sensitive problem, author has to conceal real data and use training data set. The steps to perform in this application are similar as in Application 2. Training set has 100 elements belonging to group G and 50 elements belonging to group B, and the test set has 43 elements belonging to group G and 21 elements belonging to group B. With training set, the logistic regression model shows only three variables X1, X4, and X7 have statistical significance at 5% level, so we use these three variables to perform BayesU, BayesR, BayesL, and BayesC. Their results are given in Table 9.

Cases variables		BayesU	BayesR	BayesL	BayesC
One variable	X1	86.21	86.14	84.13	87.13
	X4	81.12	82.91	86.16	88.19
	X7	83.21	84.63	83.14	84.52
Two variables	X1, X4	87.25	88.72	87.19	89.06
	X1, X7	88.16	88.34	83.26	89.56
	X4, X7	89.25	89.04	89.02	91.34
Three variables	X1, X5, X7	91.15	91.53	90.17	93.18

Table 9.

The correct probability (%) in classifying RBD by Bayesian method from training set.

From Table 9, we see that BayesC gives the highest probability in all the cases. We also use logistic method, Fisher, and SVM with training set to find the best results. We have the correct probability given in Table 10.

Using the best model for each case of methods from Table 10 to classify the test set (67 elements), we obtain the results given in Table 11.

Methods	One variable	Two variables	Three variables
Logistic	84.04	88.29	88.69
Fisher	84.73	80.73	79.32
SWM	82.34	82.03	83.07
BayesC	88.19	91.34	93.18

Table 10.

The correct probability (%) for optimal models of methods in classifying RBD.

Once again from Table 11, we see that with test data, BayesC also gives the best result.

Methods	Correct numbers	False numbers	Correct probability
Logistic	53	11	82.81
Fisher	52	12	81.25
SVM	53	11	82.81
BayesC	57	7	89.06

Table 11.

Compare the correct probability (%) in classifying RBD from test set.

5. Conclusion

This chapter presents the classification algorithm by Bayesian method in both theory and application aspect. We establish the relations of Bayes error with other measures and consider the problem to compute it in real application for one and multidimensions. An algorithm to determine the prior probabilities which may decrease Bayes error is proposed. The researched problems are applied in three different domains: biology, medicine, and economics. They show that the proposed approach has more advantages than existing ones. In addition, a complete procedure on MATLAB software is completed and is effectively used in some real applications. These examples show that our works present potential applications for research on real problems.