On the Use of Modified Winsorization with Graphical Diagnostic for Obtaining a Statistically Optimal Classification Accuracy in Predictive Discriminant Analysis

Abstract

In predictive discriminant analysis (PDA), the classification accuracy is only statistically optimal if each group sample is normally distributed with different group means, and each predictor variance is similar between the groups. This can be achieved by accounting for homogeneity of variances between the groups using the modified winsorization with graphical diagnostic (MW-GD) method. The MW-GD method involves the identification and removal of legitimate contaminants in a training sample with the aim of obtaining a true optimal training sample that can be used to build a predictive discriminant function (PDF) that will yield a statistically optimal classification accuracy. However, the use of this method is yet to receive significant attention in PDA. An alternative statistical interpretation of the graphical diagnostic information associated with the method when confronted with the challenge of differentiating between a variable shape in the groups of the 2-D area plot remains a problem to be resolved. Therefore, this paper provides a more comprehensive analysis of the idea and concept of the MW-GD method, as well as proposed an alternative statistical interpretation of the informative graphical diagnostic associated with the method when confronted with the challenge of differentiating between a variable shape in the groups of the 2-D area plot.

1. Introduction

Binary classification when compared to multiple classification has wide range of real-world applications in many areas of human endeavors, such as criminal justice, education, medicine, email analysis, human resources management, pattern recognition, energy and environmental management, financial data analysis and economics, production systems management and technical diagnosis, marketing among others. Where the classification criterion comprises one or several predictor variables along with a categorical criterion, such a prediction will require the use of a predictive discriminant analysis (PDA). PDA is still the optimal method when the cost of misclassifying groups is clearly different and when there is greater interest in the accuracy of classifying separate groups. In most cases, evaluating the proportion of correct classification of a predictive discriminant function (PDF) in all sub-populations is equivalent to the estimation of the actual hit rate, $P^{\left(a\right)}$ [1, 2]. That is, $P^{\left(a\right)}$ is the expected proportion of correct classification when a PDF that is built from a given training sample is validated on training samples from the sample population. In PDA, to improve or optimize classification accuracy or actual hit rate, researchers often rely on feature selection methods. The aim of feature selection methods in PDA is to choose the best subset of important predictor variables that will effectively reduce the intricacy of the PDF, thus facilitate interpretation, enhance or optimize the classification accuracy, and reduce the training time. Nevertheless, the promise of optimizing classification accuracy using variable selection methods is almost always unfulfilled, because the derived PDF is often obtained from a training sample that does not meet near optimal condition [1, 3, 4, 5, 6]. The actual hit rate of a PDF may be considered statistically optimal only if the assumptions of normality and/or homogeneity of variances are taken into account [5, 7]. This means that having a better subset is not a guarantee for achieving a statistically optimal classification accuracy.

In general, the task of enhancing or improving classification accuracy was examined in two ways. Several researchers use feature or variable selection techniques to select the best subset of predictors to construct a classification model. In addition to conventional feature selection techniques, including the stepwise and all possible subset methods [4, 8, 9]. Some widely known and used methods include the principal component analysis (PCA) used to obtain a set of low-dimensional features from a large set of features [10, 11]. The branch and bound technique which uses a greedy procedure to obtain the best subset [12], the genetic search algorithm [13, 14], the shrinkage methods [10, 15], the particle swarm optimization (PSO) approach which is a meta-heuristic technique used to enhance classification accuracy [16], representative methods based on dictionary learning (DL) for classification [17, 18, 19], support vector machines (SVMS) [20], and the hyper parameter tuning approach [21, 22]. We have the sequential analysis approach as well [23]. The heteroscedastic discriminant analysis merged with feature selection [24] and the modified leave-one-out (LOOCV) cross-validation method used as an alternative to the all-possible subset method [25]. A PDF’s classification accuracy is only statistically optimal if each group sample is normally distributed with different group means, and each predictor variance is similar between the groups [7]. None of these basic assumptions regarding the validity/reliability of the PDF are considered by any of the above methods. To address these gaps in feature selection techniques, other investigators are seeking alternatives to robust PDA by replacing conventional estimators with robust estimators. Some variants of these alternative methods include the dimensionality reduction/feature extraction for outlier detection (DROUT) [26], the minimum covariance determinant (MCD) [27], S-estimators [28], one-step M-estimator (MOM), and winsorized one-step M-estimator (WMOM) [29]. These other methods concentrate on building a robust PDF for deviations due to the presence of outliers in the training sample. Besides the presence of outliers in most training samples, there are also hidden influential observations resulting either from an incorrect distributional assumption or an inherent variability of the dataset [30]. Oftentimes, these hidden influential observations are not considered by any of the above methods for optimizing an actual hit rate. Consequently, the PDF’s solution obtained by either of the two approaches may be optimal but not statistically optimal. To overcome the problem of hidden influential observations, Iduseri and Osemwenkhae [6] proposed a novel method for attaining an optimal training sample. Their method otherwise known as modified winsorization with graphical diagnostic (MW-GD) method yielded a PDF’s solution which was statistically optimal for both the training sample that gave rise to it and for other training samples from the same population. However, the graphical diagnostic associated with this new method may be difficult to interpret if there are no significant differences between a variable shape in the groups of the 2-D area plot, and yet there is evidence of hidden influential observations in the training sample.

This paper provides a more comprehensive analysis of the idea and concept of the MW-GD method, as well as proposed an alternative statistical interpretation of the informative graphical diagnostic associated with the method when confronted with the challenge of differentiating between a variable shape in the groups of the 2-D area plot. The remaining sections of this paper are organized as follows. Sections 2 and 3 discuss the problems posed by the presence of outliers and legitimate contaminants in the training sample that yields the PDF, the concept of statistical optimality of the PDF classification accuracy, and the robustness of PDF, respectively. Section 4 describes in details the idea and concept of the modified winsorization with a graphical diagnostic for obtaining a statistically optimal training sample, as well as presents the proposed alternative statistical or numerical interpretation of the informative graphical diagnostic. Section 5 presents the results and discussions based on the application of two real life samples, while Section 6 presents the conclusions.

2. Outliers and legitimate contaminants in PDA

In PDA, an outlier is an observation which is not a member of a group, and is often indicative of an incorrect measurement or an incorrect allocation of the unit or observation. Such an outlying observation can cause severe problems that even the robustness of PDA may not overcome. Over the last two decades, many articles have been published about detecting outliers in discriminant analysis (DA) [31, 32, 33, 34, 35, 36, 37, 38]. In PDA, a popular means of treating outliers is to construct multiple PDFs with assumed outliers added and with assumed outliers removed [1]. The primary issue with this method is whether potential outliers should be remove one at a time, two at a time, or all at a time. With the SPSS DISCRIMINANT procedure, the chi-squared distribution is used to establish the typicality probability. These typicality probabilities are used to identify potential outliers in the context of PDA. However, Huberty and Olejnik [1] pointed out that when the group covariance matrices are not equal, the unit typicality probabilities are difficult to interpret because different distance metrics are used in the calculation. A common distance index used for detecting outliers or influential observations in the context of PDA is the Mahalanobis distance [39] which is also calculated as a byproduct in SPSS DISCRIMINANT procedure.

However, there are also hidden influential observations (or legitimate contaminants) resulting either from an incorrect distributional assumption (i.e., when the data turns out with a different structure than originally assumed) or an inherent variability of the dataset, see Osborne [30], and Iglewicz and Hoaglin [40] for more details. While hidden influential observations may actually belong to a training sample, but if not distributed randomly may reduce normality which often leads to violation of sphericity and multivariate normality assumptions in PDA. Hidden influential observation can also adversely affect the quality of the PDA solution and its generality. But how to identify and remove hidden influential observations before building a classification model (particularly in the PDA) has not receive any significant attention in the literature by statisticians or by methodologist and therefore not by any substantive researchers. Besides, the SPSS typicality or Mahalanobis index may not be able to identify hidden influential observations because their unit often belongs to a different group compared to outliers.

Therefore, much emphasis should be placed on cleaning the training sample to ensure that it meets its near-optimum condition by removing all legitimate contaminants from the training sample. This method is similar to optimizing decision trees (in particular classification trees) which consists in reducing the amount of impurity—see Myatt [41] for details. In the context of PDA, it will improve the similarity of each predictor variable variance between groups, thus improving the approximation of the true shape. This will in no doubt guarantee the statistical optimality of the PDF solution or classification accuracy.

3. Statistical optimality of a PDF classification accuracy

A PDF’s classification accuracy is only statistically optimal if each group sample is normally distributed with different group means, and each predictor variance is similar between the groups [5, 7]. In addition to the above requirements, it is recommended that there be at least four to five times as many cases as predictors in order to produce more accurate estimates. Note that in PDA, the failure of the training sample to meet the assumption of normality can result in a decrease in efficiency and accuracy—see Lachenbruch [42] as cited in Klecka [43]. However, a minor violation of this assumption will not decrease the accuracy of the classification. As long as the distributions of predictors are reasonably comparable, the estimation of most multivariate parameters does not require multivariate normality [44]. Moreover, under the central limit theorem, there is no need to worry about the assumption of normality as long as each group sample contains a very large number of observations. As a general rule, a PDF will still perform strongly against non-normality as long as the smallest group has over 20 cases, and the number of predictors is less than six [45]. Due to these robustness properties of PDA, researchers are barely concerned about the assumption of normality.

But where non-normality is due to outliers and/or hidden influencing observations other than skewness, violating this assumption has serious consequences, because PDA is very sensitive to outliers [45]. Likewise, more cases could be classified into the group with greater dispersion when the assumption of equality of variance-covariance matrices is not tenable [45]. In addition, the likelihood of belonging to a group may be distorted, and the PDF may also not be able to separate the groups as much as possible [43]. The accuracy of the discriminant weights estimates may be reduced if the variances of the predictors are not all similar between groups. They may be precise but not unbiased [46]. When the homogeneity of variance test is significant, it indicates that the training sample is contaminated with outliers and/or hidden influence observations, and the significance tests are unreliable [3, 45]. It is apparent from the foregoing that, if the assumption of homogeneity of variances is not satisfied, it is probable that the assumption of multivariate normality is not equally satisfied. This suggests that multivariate normality and homogeneity of variance assumptions can be taken into account if outliers and hidden influential observations are completely removed from a training sample. The practice of researchers relying on the robustness properties of PDA without checking for outliers and hidden influential observations, which may hinder maximal separation between the groups seems to be a norm. This practice is further encouraged by the general acceptance of a hit rate of 25% above that of chance. Assuming that you get a 95% hit rate, it is certain that you will not care about the two basic assumptions of PDA.

Whereas the reason for such good performance might be that the data support simple linear or quadratic separation boundaries. The general belief that linear classifiers are robust to minor violations of its basic assumptions (in particular is the assumption of multivariate normality) is often not tenable. Studies have shown that the reliability of a PDF solution is dependent upon adherence to the underlying assumptions [5]. The primary objective of PDA is classification, and if the percentage of correct classifications is not satisfactory, it is likely that variances in predictors are not similar across groups. That is to say the training sample is not statistically optimal. Therefore, it is necessary to adopt a screening method that will effectively identify and remove legitimate contaminants from training samples before using them to build a PDF. Iduseri and Osemwenkhae [6] proposed the modified winsorization with graphical diagnostic (MW-GD) method to identify and remove legitimate contaminants from training samples. The MW-GD method produced a statistically optimal training sample when applied to a real dataset, and the resulting PDF yielded a hit rate that was statistically optimal. As a result, the uncertainty about the PDF’s actual hit rate was greatly reduced. However, the informative graphical diagnostic associated the proposed method may be difficult to interpret if there are no significant differences between a variable shape in the groups of the 2-D area plot.

This paper proposes an alternative statistical interpretation of the informative graphical diagnostic when confronted with the challenge of differentiating between a variable shape in the groups of the 2-D area plot.

4. Identification and removal of legitimate contaminants in PDA

4.1 Correction for bias of discriminant weights in PDA

In general, extreme scores or outliers bias estimates of any parameter. One notable way to correct for bias is to change the data by changing the scores so as to reduce the impact of the extreme scores or adjust the shape of the distribution. Notable variants of changing the scores method include transforming the data, trimming the data and winsorizing the data. However, in PDA, where one is interested in differences between set of variables or groups, transformation may not be a good choice to correct for bias of discriminant weights. This is because, transformation can change the units of measurements, which may in turn affects the interpretation of the data because the data now relate to a different construct compared to the original data [47]. Similarly, trimming of data seemed odd since one could just discard lots of data. To overcome these inherent drawbacks associated with both methods, the winsorization method was adopted. These approach involves replacing a percentage of the highest score with the next highest score in the data and the same percentage of the lowest score are replaced with the next lowest score in the data. One major challenge with this method is that even the next higher or lowest score might still be an extreme score. Another variant of the winsorization involves replacing extreme score with a score three standard deviations from the mean. This variant of winsorization also suffers a major drawback. As noted by Field [47], the standard deviation will be biased by extreme scores, so this means that you are replacing scores with a value that has been biased by extreme scores. To address the observed shortcomings of both variants of the winsorization method, Iduseri and Osemwenkhae [6] proposed the modified winsorization with graphical diagnostic (MW-GD) method. The method proved very effective in identifying and removing legitimate contaminants.

4.2 The modified winsorization with graphical diagnostic (MW-GD) method

In this section, the modified winsorization with graphical diagnostic (MW-GD) method originally proposed by Iduseri and Osemwenkhae [6] is presented. In addition, a proposed alternative statistical interpretation of the informative graphical diagnostic associated with MW-GD method when confronted with the challenge of differentiating between bar shapes of the 2-D area plot is also presented. The MW-GD method, which involves a three-step procedure, will effectively identify and eliminate legitimate contaminants from predictor variables so that their variances between the groups are similar. The aim is to ensure that the training sample, $D_N^t$ satisfies the basic assumptions (particularly the assumption of homogeneity of variances) of the PDA. The three steps procedure produced an optimal training sample that was used to construct a PDF whose percentage of correct classification was not only statistically optimal for the training sample that produced it, but also for other training samples from the same population.

4.2.1 Algorithm for the modified winsorization with graphical diagnostic (MW-GD)

Let $D_N^t=\left[x_1,x_2,\cdots ,x_N\right]\in {\Re }^{P\times N}$ be a training sample matrix that comprises N observations set${\left\{\left(x_i,y_i\right)\right\}}_{i=1}^n$, obtained from a real dataset using any of the conventional feature selection techniques, where $x_i\in \left\{1,2,\cdots ,P\right\}$ denote the corresponding predictor variable label, $y_i\in \left\{1,2,\cdots ,K\right\}$ denote the corresponding group label, $P$ is the number of predictor variables, and $K$ is the number of groups.

Step 1: Identification of the predictor variables with legitimate contaminants.

For the training sample, $D_N^t$ the scores or observations of the predictor variables, $X_1,\cdots ,X_N$ are first arranged in ascending order so that the extreme scores at both ends can be identified. For each predictor variable in the unordered training sample, $D_N^t$, a pair of score (i.e., the most extreme and the least extreme scores initially identified at both ends of each distribution) is deleted and replaced with the median value, before the mean of the remaining scores is calculated. The median value was adopted in order to satisfy the assumption of independence of all cases. The median is a position average independent from all other cases, whereas the mean depends on all other cases. Consequently, substituting the identified influential observations with their median value takes into account the assumption that all observations must be independent. This process is repeated for other lower pairs of extreme values and stops when five modified winsorized means are obtained for each predictor variable. When the modified winsorized means values are plotted, the predictor variable with bar shapes that are not similar between groups in the 2-D area plot becomes the predictor variable with legitimate contaminants.

Step 2: Removing legitimate contaminants from the identified predictor variables.

To determine what percentage of winsorization is required to eliminate the legitimate contaminants, the modified winsorization process is repeated only for the predictive variable(s) identified with the legitimate contaminants until the highest hit rate is attained, thus obtaining a near optimal training sample given as:

$$D_{N\left(\mathit{\text{optimal}}\right)}^t=\left[x_1,x_2,\cdots ,x_P\right]$$E1

Step 3: Obtaining a statistically optimal hit rate.

The optimal training sample of Eq. (1), was then used to build the optimized PDF, $Z_{\left(\mathit{opt}\right)}$ given as:

$$\begin{array}{l}{Z}_{\left(\mathit{\text{optimal}}\right)}=u_1{X}_1+u_2{X}_2+\cdots +u_P{X}_P\\ =\eta \left(D_{N\left(\mathit{\text{optimal}}\right)}^t\right)\end{array}$$E2

where $Z_{\left(\mathit{\text{optimal}}\right)}$ is the optimized PDF, $u_i$ are the discriminant weights, $X_i$ are the predictor variables and $\eta \left(D_{N\left(\mathit{\text{optimal}}\right)}^t\right)$ shows that the PDF is constructed with an optimal training sample.

To get a statistically optimal hit rate, let:

$$d_j=\left\{\begin{array}{c}1\mathit{\text{if}}{\hat{Z}}_j=Z_j\\ 0\mathit{\text{otherwise}}\end{array}\right.$$E3

where ${\hat{Z}}_j$ is the predicted response for the $\mathit{jth}$ observation in the optimized training sample, $Z_j$ is the value for the $\mathit{jth}$ observation in the optimized training sample. Therefore, a statistically optimal hit rate for the optimized PDF in (2) is given as:

$$P^{\left(a\right)}=\frac{1}{N^t}\left(\sum _{j=1}^{N^t}{d}_j\right)\times 100$$E4

where $N^t$ is the total number of cases over all groups in the optimized training sample. If we redefine $d_j$ as:

$$d_j=\left\{\begin{array}{c}1\mathit{\text{if}}{\hat{Z}}^{-j}=Z_j\\ 0\mathit{\text{otherwise}}\end{array}\right.$$E5

where ${\hat{Z}}^{-j}$ is the predicted response for the $\mathit{jth}$ observation computed with the $\mathit{jth}$ observation removed from the training sample. The leave-one-out cross-validation (LOOCV) estimate of the optimized hit rate (4) is given by:

$${\hat{P}}_{\mathit{\text{LOOCV}}}^{\left(a\right)}=\frac{1}{N}\left(\sum _{j=1}^{N_v}{d}_j\right)\times 100$$E6

where $n_v$ is the number of validation samples.

4.3 The proposed alternative statistical interpretation for the informative graphical diagnostic

As described in Step 1 of Section 4.2.1, the graphical representation of the modified winsorized means makes it possible to easily identify the predictor variable(s) whose variance is not significantly similar between the groups. If the variance of a predictor variable is not similar between groups, the bars representing the modified winsorized mean values for the variable in the 2-D area plot will not have similar shape. Such a variable is interpreted to be the identified variable that contains legitimate contaminants. This means that the more the shape of the bars differ between the groups, the easier it is to interpret the 2 D area plot. It is therefore necessary to provide an alternative interpretation when it is difficult to differentiate between a variable shape in the groups of the 2-D area plot. Therefore, a simple statistical or numerical interpretation is proposed for the informative graphical diagnosis when it is difficult to differentiate between a variable shape in the groups of the 2-D area plot. This alternative numerical interpretation consists of the following two simple steps:

Step 1: Fitting the modified winsorized means values to a linear regression model.

For each group, the modified winsorized means and their corresponding winsorized percentage values for each predictor variable are fitted to a linear regression model given as:

$$\begin{array}{l}{Y}_{11}=a+b_{11}X\\ ⋮\\ Y_{1P}=a+b_{1P}X\end{array}$$E7

and

$$\begin{array}{l}{Y}_{21}=a+b_{21}X\\ ⋮\\ Y_{2P}=a+b_{2P}X\end{array}$$E8

where, $Y_{11},\cdots ,Y_{1P}$ and $Y_{21},\cdots ,Y_{2P}$ are the modified winsorized mean values for the $P$ predictor variables in groups 1 and 2, respectively, and $X$ is the corresponding winsorized percent values.

Step 2: Obtaining the absolute difference between the corresponding regression coefficients for the groups.

The absolute difference between the obtained regression coefficients (i.e., the slope) in group 1 and 2 is computed as:

$${\delta }_{\mathit{abs}}=b_{1i}-b_{2i},i=1,\cdots ,p$$E9

where the subscripts 1 and 2 represents groups 1 and 2. The predictor variable that has an absolute difference of 0.75 or greater will be the variable identified with legitimate contaminants. In PDA, if two samples are equal in size, there is always a 50/50 chance. Most researchers would accept a classification accuracy of 25% greater than that caused by chance. Hence, the choice of 0.75 as the decision boundary.

5. Computational results and discussion

5.1 Using the modified winsorization with graphical diagnostic (MW-GD) method

To evaluate the effectiveness of the modified winsorization with graphical diagnostic (MW-GD) method, two real samples (see [6] for the two data sets) were considered with the second used as a validation sample. The first training sample is from a renowned financial journal, among Japanese business leaders, which can be compared to the Economist, Financial Times, and Business Week in Europe and the United States of America. This dataset contains 50 observations from each of the two groups of Japanese financial institutions, each bank being evaluated using the following seven performance indexes: (1) return on total assets (= total profits/average total assets), (2) labor profitability (= total profits/total employees), (3) equity to total assets (= total equity/average total assets), (4) total net working capital, (5) return on equity (= earnings available for common/average equity), (6) cost-profit ratio (= total operating expenditures/total profits), and (7) bad loan ratio (= total bad loans/total loans). However, taking into account the beneficial effect of feature selection and outlier detection as a preprocessing step, a best subset and critical value of Mahalanobis distance were first obtained using the SPSS stepwise method, and its compute command for critical value of Mahalanobis distance. The stepwise approach produced a best subset which comprises return on total assets (X₁), labor profitability (X₂), equity to total assets (X₃), and bad loan ratio (X₇). The SPSS output of the constructed PDF based on the training sample of four variables given by:

$$Z=0.005X_1+0.006X_2+0.004X_3+0.005X_7$$E10

Thus, the classification accuracy of the PDF, $Z$(10) and its LOOCV estimate are given by Eqs. (11) and (12).

$$P^{\left(a\right)}=\frac{1}{N^t}\left(\sum _{j=1}^{N^t}{d}_j\right)\times 100=86.0\%$$E11

$${\hat{P}}_{\mathit{\text{LOOCV}}}^{\left(a\right)}=\frac{1}{N}\left(\sum _{j=1}^{n_v}{d}_j\right)\times 100=85\%$$E12

While the two critical values from the SPSS outputs for 0.95 and 0.99 used as the probabilities in the IDF.CHISQ function with four predictor variables were given as:

$$\begin{array}{l}\mathit{\text{COMPUTEcritical}}=\mathit{IDF}.\mathit{\text{CHISQ}}\left(0.95,4\right)=9.49\\ \mathit{\text{COMPUTEcritical}}=\mathit{IDF}.\mathit{\text{CHISQ}}\left(0.99,4\right)=13.28\end{array}$$

The Mahalanobis distance values for all cases, as reported in the case wise statistics table, were all lower than the two critical values. This means that there are neither outliers nor hidden influential observations in the dataset or training sample. Next, the MW-GD algorithm was applied to the training sample, $D_N^t$ consisting of four predictor variables.

At step 1 of the MW-GD method, a total of five modified winsorized means for the four predictor variables were obtained. The summary of the winsorized means values is presented in Table 1.

Groups	No. of replaced sample points	Modified winsorized mean values				Winsorized %
		X1	X2	X3	X7
1	0	428.70	329.48	536.34	821.32	0
	2	421.08	317.32	533.90	824.50	4
	4	415.50	307.60	533.70	827.40	8
	6	412.28	298.32	533.56	830.36	12
	8	408.90	290.30	534.46	833.62	16
	10	406.20	282.70	535.42	835.56	20
2	0	341.50	177.20	368.46	778.70	0
	2	335.24	173.20	366.92	785.80	4
	4	329.66	169.16	365.20	790.74	8
	6	324.76	166.68	362.94	794.22	12
	8	319.86	166.22	360.68	797.30	16
	10	315.44	165.94	358.66	800.12	20

Table 1.

Modified winsorized means for up to five pairs of winsorized values.

To provide a meaningful interpretation of Table 1, the modified winsorized means or averages for both groups, as shown in Table 1, were plotted with a 2D area plot in Excel Package. The process involves entering the first variable X₁ modified winsorized averages for both groups into Excel spreadsheet in pairs (with X₁ values for group 1 and 2 occupying the first 2 columns from row 1 to row 6), followed by variable X₂ (with X₂ values for group 1 and 2 occupying column 3 to column 4 from row 8 to 13), followed by variables X₃ and X₇ proceeding downward in steps. The graphical representation is presented in Figure 1.

Figure 1.

Graphical representation of the modified winsorized mean values in Table 1.

A cursory look at Figure 1 shows that the winsorized average values for the four predictor variables in both groups represented by the 2-D area plot have similar shape (or have similar variances within the groups) except for predictor variable X₂. Also, in Figure 1, the bar shape of the predictor variable X₂ in group 1 looks like a rectangle whereas in group 2 it looks like a trapezium. The observed difference in the shape of the variable X₂ bar indicates that variable X₂ does not have similar variances in the groups, and therefore becomes the only variable with legitimate contaminants. It therefore means that the training sample does not satisfy the assumption of homogeneity of variances. This finding was corroborated by the result of the Box M test for equality of variance-covariance matrices for this training sample, which was significant. Apart from manually replacing the extreme values on both ends with the median value, for each percentage of winsorization using R aggregate () function, the average calculation time needed to generate each row result for groups 1 and 2 in Table 1 was 2 seconds.

At step 2 of the MW-GD method, the modified winsorization process was performed for only variable X₂. For each percent of winsorization, a PDF is constructed using all four predictor variables. The summary of hit rate results for each percent of winsorization is presented in Table 2. Table 2 shows that the highest hit rate of 97.00 was achieved when 5 data points at both ends of the data were replaced by the median value. This means that all the legitimate contaminants in variable X₂ was completely replaced at 20% winsorization. In other words, at 20% winsorization, the lack of homogeneity of variances observed in the variable X₂ was taken into account, thus obtaining a near optimal training sample, $D_{N\left(\mathit{opt}\right)}$:

% of winsorization	No. of replaced sample points	% of training sample correctly classified
4	2	85.00
8	4	88.00
12	6	91.00
16	8	94.00
20	10	97.00a
24	12	96.00

Table 2.

Summary of hit rate results for each percent of modified winsorization for predictor variable, X₂.

^a

Optimal winsorization occurs at 20% with hit rate = 97.00%.

The optimized training sample, $D_{N\left(\mathit{opt}\right)}$ was then used to construct a PDF. The SPSS output for the obtained PDF is given as:

$$Z_{\mathit{opt}}=0.003X_1+0.018X_2+0.001X_3+0.004X_7$$E13

Thus, the PDF, $Z_{\mathit{opt}}$(13) hit rate and its LOOCV estimate are given by Eqs. (14) and (15).

$$P^{\left(a\right)}=\frac{1}{N^t}\left(\sum _{j=1}^{N^t}{d}_j\right)\times 100=97.0\%$$E14

$${\hat{P}}_{\mathit{\text{LOOCV}}}^{\left(a\right)}=\frac{1}{N}\left(\sum _{j=1}^{n_v}{d}_j\right)\times 100=91\%$$E15

In addition to the dataset from Japanese banks, a second real dataset was used to validate the first sets of results (11), (12), (14), and (15). This validation sample was obtained from the academic records of junior secondary school (JSS) 2, University Demonstration Secondary School (UDSS), University of Benin, Nigeria. The dataset contains 30 observations for both classes: Science and Art. The dataset consists of average scores for the three consecutive terms obtained for eleven (11) subjects, including English Language (X₁), Mathematics (X₂), Integrated Science (X₃), Social Studies (X₄), Introductory Technology (X₅), Business Studies (X₆), Home Economics (X₇), Agricultural Science (X₈), Fine Art (X₉), Physical and Health Education (X₁₀), and Computer Studies (X₁₁). Using the SPSS stepwise method, a subset of three variables comprising introductory technology (X₅), physical and health education (X₁₀), and computer science (X₁₁) was obtained. The SPSS output for the PDF is given as:

$$Z=0.135X_5-0.102X_{10}+0.058X_{11}$$E16

Thus, the PDF, $Z$(16) hit rate and its LOOCV estimate are given by Eqs. (17) and (18).

$$P^{\left(a\right)}=\frac{1}{N^t}\left(\sum _{j=1}^{N^t}{d}_j\right)\times 100=85.0\%$$E17

$${\hat{P}}_{\mathit{\text{LOOCV}}}^{\left(a\right)}=\frac{1}{N}\left(\sum _{j=1}^{n_v}{d}_j\right)\times 100=83\%$$E18

Also, the two critical values from the SPSS outputs for 0.95 and 0.99 used as the probabilities in the IDF.CHISQ function with three predictor variables were given as:

$$\begin{array}{l}\mathit{\text{COMPUTEcritical}}=\mathit{IDF}.\mathit{\text{CHISQ}}\left(0.95,4\right)=7.81\\ \mathit{\text{COMPUTEcritical}}=\mathit{IDF}.\mathit{\text{CHISQ}}\left(0.99,4\right)=11.34\end{array}$$

The Mahalanobis distance values for all cases, as reported in the case wise statistics table, were all lower than the two critical values. This means that there are neither outliers nor hidden influential observations in the dataset or training sample. Once again, the proposed algorithm is applied to this second training sample, $D_N^t$ consisting of three predictor variables.

At step 1 of the MW-GD method, a total of five modified winsorized means for the three predictor variables were obtained. The summary of the modified winsorized means values is presented in Table 3.

Groups	No. of replaced sample points	Modified winsorized mean values			Winsorized %
		X5	X10	X11
1	0	67.27	69.77	73.30	0.00
	2	67.40	70.07	73.43	6.67
	4	67.43	70.17	73.60	13.33
	6	67.47	70.13	73.67	20.00
	8	67.53	70.13	73.70	26.67
	10	67.60	70.17	73.73	33.33
2	0	51.47	62.40	63.93	0.00
	2	51.27	62.53	63.97	6.67
	4	51.07	62.70	63.73	13.33
	6	50.97	62.83	63.60	20.00
	8	50.77	63.00	63.50	26.67
	10	50.73	63.23	63.30	33.33

Table 3.

Winsorized means for up to five pairs of winsorized values.

Again, to interpret Table 3, the modified winsorized means or averages for both groups, as shown in column three of Table 3, were plotted with a 2D area plot in Excel Package. The graphical representation is presented in Figure 2.

Figure 2.

Graphical representation of winsorized mean values in Table 3.

A cursory look at Figure 2 shows that the winsorized average values for the four predictor variables in both groups represented by the 2-D area plot have similar shape (or have similar variances within the groups). The similar shape shown by the three variables in each group indicates that there are no legitimate contaminants in the training sample, $D_N^t$. This implies that the fit between the training sample, $D_N^t$ and the basic assumptions of PDA is sufficient (in particular is the assumption of homoscedasticity) to construct a PDF whose hit rate can be said to be statistically optimal. Therefore, for this dataset, the MW-GD algorithm ends at step 1. The initial training sample of three variables obtained from the second real dataset using SPSS stepwise method is therefore an optimal training sample.

5.2 Using the proposed alternative statistical interpretation for the informative graphical diagnostic

If the modified winsorized means for each variable in Table 1 are denoted as the independent variable Y$\left(Y=Y_1,Y_2,Y_3,Y_4\right)$, and the winsorized percent as the dependent variable X, then the six pairs of values for X (0, 4, 8, 12, 16, 20) and Y₁ (428.70, 421.07, 415.50, 412.28, 408.90, 406.20) are in good accordance to a regression model.

At step 1 of the proposed alternative statistical interpretation for the informative graphical diagnostic, each of the six values of the modified winsorized means for the four variables in groups 1 and 2 with the six values of the winsorized percent are fitted to a linear regression model. The summary of the obtained values of the regression coefficient, $b$ for the four fitted regression models each for groups 1 and 2 are presented in Table 4.

Group	Regression coefficients, b
	X1	X2	X3	X7
1	−1.088	−2.316	−0.022	0.725
2	−1.295	−0.569	−0.500	1.036
${\delta }_{\mathit{abs}}=b_{1i}-b_{2i}$	0.2	1.8	0.5	0.3

Table 4.

Regression coefficients, b of fitted regression model to Table 1 data.

At step 2 of the proposed alternative approach, an absolute difference between the obtained regression coefficients (i.e., the slope) for each variable in group 1 and 2 was also obtained. The summary of the absolute values is presented as the last row of Table 4.

The two step approach of the proposed alternative method was repeated using the data of Table 3. The summary of the obtained values of the regression coefficient, $b$ for the three fitted regression models each for groups 1 and 2, and the summary of the absolute values is presented as the last row of Table 5.

Group	Regression coefficients, b
	X5	X10	X11
1	0.009	0.009	0.013
2	−0.023	0.024	−0.020
${\delta }_{\mathit{abs}}=b_{1i}-b_{2i}$	0.0	0.0	0.0

Table 5.

Regression coefficients, b of fitted regression model to Table 3 data.

A cursory look at Table 4 shows that the regression coefficient (−2.316) of X₂ in group 1 is notably lower than the corresponding regression coefficient (−0.569) of variable X₂ in group 2, and the regression coefficients obtained for the variables X₁, X₃, and X₇ in group 1 and in group 2, respectively. Also, the value of 1.8 which is the absolute difference between the regression coefficients of X₂ in group 1 and in group 2 is significantly greater than the decision boundary value of 0.75. This indicates that the variable X₂ does not have similar variances in groups formed by the dependent, and thus becomes the variable identified with legitimate contaminants. In Table 5, the values of the regression coefficients for the three variables in groups 1 and 2 are equivalently equal. Also, the values of the absolute difference between the regression coefficients of variable X₅, X₁₀, and X₁₁ in group 1 and in group 2 are all equal, and significantly lower than the decision boundary value of 0.75. This indicates that there are no legitimate contaminants in variable X₅, X₁₀, and X₁₁, respectively. This therefore implies that the fit between the validation sample, $D_N^t$ and the basic assumptions of PDA is sufficient to construct a PDF whose hit rate can be said to be statistically optimal.

6. Conclusions

This paper addresses the issue of achieving a statistically optimal classification accuracy in PDA by first achieving an optimal training sample. For the first real dataset, a training sample of four variables was obtained using the SPSS stepwise method. The training sample gave a hit rate of 86.0%. When all legitimate contaminants in one of the four variables have been identified and eliminated using the MW-GD method, an optimal training sample was achieved. The optimized training sample was used to construct the PDF, $Z_{\mathit{opt}}$(13) which gave a classification accuracy of 97.0% when tested on the initial training sample of four variables. This significant increase in classification accuracy suggests that the use of the WM-GD method seems to effectively enhance the similarity of each predictor variable variances between groups, thus taken into account the basic assumptions needed to achieve a statistically optimal classification accuracy. Using the second real dataset, a training sample of three variables was obtained and used to construct the PDF, $Z$(16), which yielded 85.0% hit rate. When the modified mean values of the three variables were plotted, the bar shape of the three variables in the two groups was similar. This means that the PDF, $Z$(16) hit rate of 85.0% cannot be increased further because the training sample that gave birth to it was statistically optimal.

The uniqueness of the MW-GD method lies in its ability to effectively identify and eliminate legitimate contaminants in one or several predictor variables, thus resolving any significant differences in the variances of the predictor variables between the groups. In other words, the MW-GD method is unique in its ability to sufficiently account for the basic assumptions required to achieve statistically optimal classification accuracy in PDA. As a result, an optimal training sample obtained from the first real dataset gave a statistically optimal hit rate of 97.0% compared to an initial maximum hit rate of 86.0%. For the second real dataset, the method was successful in confirming the optimality of the initial training sample obtained using the SPSS stepwise method. Similarly, the graphical diagnostic was able to identify the predictor variable(s) whose variance was not similar within the groups. Consequently, the graphical diagnostic associated with the proposed method could be used as an alternative graphical test of homogeneity of variances in PDA.

Another important contribution to the MW-GD method in this paper was the proposed alternative statistical interpretation for the graphical diagnostic associated with the MW-GD method demonstrated in Subsection 5.2. This proposed alternative statistical interpretation proved very effective in terms of identifying the variable with legitimate contaminants, and could serve as a useful alternative tool for identifying variables with legitimate contaminants in the event of any difficulties in differentiating between a variable shape in the groups of the 2-D area plot.

Finally, two real training samples have been used. Consequently, the validity of the experimental results is limited to the scope of the datasets used. Therefore, this paper believes that more experimental results are needed in order to reach a final conclusion on the efficiency of the MW-GD method compared to classical alternatives known to improve classification accuracy in PDA.

Nomenclature

DNt	this is the complete list of objects used in building the predictive discriminant function (PDF)
DNt	this is the complete list of objects without outliers and hidden influential observation used in building the predictive discriminant function (PDF)
K	the number of groups or categorical criterion
N	total number of cases over all groups in a dataset
Nt	total number of cases over all groups in the optimized training sample
nv	the number of validation samples
P	the number of predictor variables
Pa	this is the percentage of cases on the diagonal of the confusion matrix, or simply the percent of correct classification
p^a	this is the hit-rate obtained by applying a rule based on a particular training sample to future samples taken from the same population
p^LOOCVa	this is the leave-one-out cross-validation (LOOCV) estimate of the optimized hit rate
Xi	predictor variables
ui	discriminant weights
Z	a predictive discriminant function (PDF) created by a linear combination of observable variables
ZOptimal	an optimized predictive discriminant function (PDF) created by a linear combination of observable variables without outliers and hidden influential observations
Zj	value for the jth observation in the optimized training sample
Z^j	predicted response for the jth observation in the optimized training sample
Z^−j	predicted response for the jth observation calculated with the jth observation removed from the training sample