Data mining and machine learning studies on the subject “defect prediction.”
Software engineering is one of the most utilizable research areas for data mining. Developers have attempted to improve software quality by mining and analyzing software data. In any phase of software development life cycle (SDLC), while huge amount of data is produced, some design, security, or software problems may occur. In the early phases of software development, analyzing software data helps to handle these problems and lead to more accurate and timely delivery of software projects. Various data mining and machine learning studies have been conducted to deal with software engineering tasks such as defect prediction, effort estimation, etc. This study shows the open issues and presents related solutions and recommendations in software engineering, applying data mining and machine learning techniques.
- software engineering tasks
- data mining
- text mining
In recent years, researchers in the software engineering (SE) field have turned their interest to data mining (DM) and machine learning (ML)-based studies since collected SE data can be helpful in obtaining new and significant information. Software engineering presents many subjects for research, and data mining can give further insight to support decision-making related to these subjects.
Figure 1 shows the intersection of three main areas: data mining, software engineering, and statistics/math. A large amount of data is collected from organizations during software development and maintenance activities, such as requirement specifications, design diagrams, source codes, bug reports, program versions, and so on. Data mining enables the discovery of useful knowledge and hidden patterns from SE data. Math provides the elementary functions, and statistics determines probability, relationships, and correlation within collected data. Data science, in the center of the diagram, covers different disciplines such as DM, SE, and statistics.
This study presents a comprehensive literature review of existing research and offers an overview of how to approach SE problems using different mining techniques. Up to now, review studies either introduce SE data descriptions , explain tools and techniques mostly used by researchers for SE data analysis , discuss the role of software engineers , or focus only on a specific problem in SE such as defect prediction , design pattern , or effort estimation . Some existing review articles having the same target  are former, and some of them are not comprehensive. In contrast to the previous studies, this article provides a systematic review of several SE tasks, gives a comprehensive list of available studies in the field, clearly states the advantages of mining SE data, and answers “how” and “why” questions in the research area.
The novelties and main contributions of this review paper are fivefold.
First, it provides a general overview of several SE tasks that have been the focus of studies using DM and ML, namely, defect prediction, effort estimation, vulnerability analysis, refactoring, and design pattern mining.
Second, it comprehensively discusses existing data mining solutions in software engineering according to various aspects, including methods (clustering, classification, association rule mining, etc.), algorithms (k-nearest neighbor (KNN), neural network (NN), etc.), and performance metrics (accuracy, mean absolute error, etc.).
Third, it points to several significant research questions that are unanswered in the recent literature as a whole or the answers to which have changed with the technological developments in the field.
Fourth, some statistics related to the studies between the years of 2010 and 2019 are given from different perspectives: according to their subjects and according to their methods.
Five, it focuses on different machine learning types: supervised and unsupervised learning, especially on ensemble learning and deep learning.
This paper addresses the following research questions:
RQ1. What kinds of SE problems can ML and DM techniques help to solve?
RQ2. What are the advantages of using DM techniques in SE?
RQ3. Which DM methods and algorithms are commonly used to handle SE tasks?
RQ4. Which performance metrics are generally used to evaluate DM models constructed in SE studies?
RQ5. Which types of machine learning techniques (e.g., ensemble learning, deep learning) are generally preferred for SE problems?
RQ6. Which SE datasets are popular in DM studies?
The remainder of this paper is organized as follows. Section 2 explains the knowledge discovery process that aims to extract interesting, potentially useful, and nontrivial information from software engineering data. Section 3 provides an overview of current work on data mining for software engineering grouped under five tasks: defect prediction, effort estimation, vulnerability analysis, refactoring, and design pattern mining. In addition, some machine learning studies are divided into subgroups, including ensemble learning- and deep learning-based studies. Section 4 gives statistical information about the number of highly validated research conducted in the last decade. Related works considered as fundamental by journals with a highly positive reputation are listed, and the specific methods they used and their categories and purposes are clearly expressed. In addition, widely used datasets related to SE are given. Finally, Section 5 offers concluding remarks and suggests future scientific and practical efforts that might improve the efficiency of SE actions.
2. Knowledge discovery from software engineering data
This section basically explains the consecutive critical steps that should be followed to discover beneficial knowledge from software engineering data. It outlines the order of necessary operations in this process and explains how related data flows among them.
Software development life cycle (SDLC) describes a process to improve the quality of a product in project management. The main phases of SDCL are planning, requirement analysis, designing, coding, testing, and maintenance of a project. In every phase of software development, some software problems (e.g., software bugs, security, or design problems) may occur. Correcting these problems in the early phases leads to more accurate and timely delivery of the project. Therefore, software engineers broadly apply data mining techniques for different SE tasks to solve SE problems and to enhance programming efficiency and quality.
Figure 2 presents the data mining and knowledge discovery process of SE tasks including data collection, data preprocessing, data mining, and evaluation. In the data collection phase, data are obtained from software projects such as bug reports, historical data, version control data, and mailing lists that include various information about the project’s versions, status, or improvement. In the data preprocessing phase, the data are preprocessed after collection by using different methods such as feature selection (dimensionality reduction), feature extraction, missing data elimination, class imbalance analysis, normalization, discretization, and so on. In the next phase, DM techniques such as classification, clustering, and association rule mining are applied to discover useful patterns and relationships in software engineering data and therefore to solve a software engineering problem such as defected or vulnerable systems, reused patterns, or parts of code changes. Mining and obtaining valuable knowledge from such data prevents errors and allows software engineers to deliver the project on time. Finally, in the evaluation phase, validation techniques are used to assess the data mining results such as k-fold cross validation for classification. The commonly used evaluation measures are accuracy, precision, recall, F-score, area under the curve (AUC) for classification, and sum of squared errors (SSE) for clustering.
3. Data mining in software engineering
In this review, we examine data mining studies in various SE tasks and evaluate commonly used algorithms and datasets.
3.1 Data mining in defect prediction
A defect means an error, failure, flaw, or bug that causes incorrect or unexpected results in a system . A software system is expected to be without any defects since software quality represents a capacity of the defect-free percentage of the product . However, software projects often do not have enough time or people working on them to extract errors before a product is released. In such a situation, defect prediction methods can help to detect and remove defects in the initial stages of the SDLC and to improve the quality of the software product. In other words, the goal of defect prediction is to produce robust and effective software systems. Hence, software defect prediction (SDP) is an important topic for software engineering because early prediction of software defects could help to reduce development costs and produce more stable software systems.
Various studies have been conducted on defect prediction using different metrics such as code complexity, history-based metrics, object-oriented metrics, and process metrics to construct prediction models [10, 11]. These models can be considered on a cross-project or within-project basis. In within-project defect prediction (WPDP), a model is constructed and applied on the same project . For within-project strategy, a large amount of historical defect data is needed. Hence, in new projects that do not have enough data to train, cross-project strategy may be preferred . Cross-project defect prediction (CPDP) is a method that involves applying a prediction model from one project to another, meaning that models are prepared by utilizing historical data from other projects [14, 15]. Studies in the field of CPDP have increased in recent years [10, 16]. However, there are some deficiencies in comparisons of prior studies since they cannot be replicated because of the difference in utilizing evaluation metrics or preparation way of training data. Therefore, Herbold et al.  tried to replicate different CPDP methods previously proposed and find which approach performed best in terms of metrics such as F-score, area under the curve (AUC), and Matthews correlation coefficient (MCC). Results showed that 7- or 8-year approaches may perform better. Another study  replicated prior work to demonstrate whether the determination of classification techniques is important. Both noisy and cleaned datasets were used, and the same results were obtained from the two datasets. However, new dataset gave better results for some classification algorithms. For this reason, authors claimed that the selection of classification techniques affects the performance of the model.
Numerous defect prediction studies have been conducted using DM techniques. In the following subsections, we will explain these studies in terms of whether they apply ensemble learning or not. Some defect prediction studies in SE are compared in Table 1. The objective of the studies, the year they were conducted, algorithms, ensemble learning techniques and datasets in the studies, and the type of data mining tasks are shown in this table. The bold entries in Table 1 have better performance than other algorithms in that study.
|Ref.||Year||Task||Objective||Algorithms||Ensemble learning||Dataset||Evaluation metrics and results|
|||2011||Classification||Comparative study of various ensemble methods to find the most effective one||NB||Bagging, boosting, RT, RF, RS, AdaBoost, Stacking, and Voting||NASA datasets: CM1 JM1 KC1 KC2 KC3 KC4 MC1 MC2 MW1 PC1 PC2 PC3 PC4 PC5||10-fold CV, ACC, and AUC|
Vote 88.48% random forest 87.90%
|||2013||Classification||Comparative study of class imbalance learning methods and proposed dynamic version of AdaBoost.NC||NB, RUS, RUS-bal, THM, SMB, BNC||RF, SMB, BNC, AdaBoost.NC||NASA and PROMISE repository: MC2, KC2, JM1, KC1, PC4, PC3, CM1, KC3, MW1, PC1||10-fold CV|
Balance, G-mean and AUC, PD, PF
|||2014||Classification||Comparative study to deal with imbalanced data||Base Classifiers: C4.5, NB|
Sampling: ROS, RUS, SMOTE
|AdaBoost, Bagging, boosting, RF||NASA datasets: CM1, JM1, KC1, KC2, KC3, MC1, MC2, MW1, PC1, PC2, PC3, PC4, PC5||5 × 5 CV, MCC, ROC, results change according to characteristics of datasets|
|||2015||Clustering/classification||To show that the selection of classification technique has an impact on the performance of software defect prediction models||Statistical: NB, Simple Logistic|
Clustering: KM, EM
Rule based: Ripper, Ridor
Nearest neighbor: KNN
DTs: J48, LMT
|Bagging, AdaBoost, rotation forest, random subspace||NASA: CM1, JM1, KC1, KC3, KC4, MW1, PC1, PC2, PC3, PC4|
PROMISE: Ant 1.7, Camel 1.6, Ivy 1.4, Jedit 4, Log4j 1, Lucene 2.4, Poi 3, Tomcat 6, Xalan 2.6, Xerces 1.3
|10 × 10-fold CV|
AUC > 0.5
Scott-Knott test α = 0.05, simple logistic, LMT, and RF + base learner outperforms KNN and RBF
|||2015||Classification||Average probability ensemble (APE) learning module is proposed by combining feature selection and ensemble learning||APE system combines seven classifiers: SGD, weighted SVMs (W-SVMs), LR, MNB and Bernoulli naive Bayes (BNB)||RF, GB||NASA: CM1, JM1, KC1, KC3, KC4, MW1, PC1, PC2, PC3, PC4|
PROMISE (RQ2): Ant 1.7, Camel 1.6, Ivy 1.4, Jedit 4, Log4j 1, Lucene 2.4, Poi 3, Tomcat 6, Xalan 2.6, Xerces 1.3
|10 × 10-fold CV, AUC > 0.5|
Scott-Knott test α = 0.05, simple logistic, LMT, and RF + base learner outperforms KNN and RBF
|[22, 23]||2016||Classification||Comparative study of 18 ML techniques using OO metrics on six releases of Android operating system||LR, NB, BN, MLP, RBF|
SVM, VP, CART, J48, ADT, Nnge, DTNB
|Bagging, random forest, Logistic model trees, Logit Boost, Ada Boost||6 releases of Android app: Android 2.3.2, Android 2.3.7, Android 4.0.4, Android 4.1.2, Android 4.2.2, Android 4.3.1||10-fold, inter-release validation|
AUC for NB, LB, MLP is >0.7
|||2016||Classification||Caret has been applied whether parameter settings can have a large impact on the performance of defect prediction models||NB, KNN, LR, partial least squares, NN, LDA, rule based, DT, SVM||Bagging, boosting||Cleaned NASA JM1, PC5|
Proprietary from Prop-1 to Prop-5
Apache Camel 1.2, Xalan 2.5–2.6
Eclipse Platform 2.0–2.1–3.0, Debug 3.4, SWT 3.4, JDT, Mylyn, PDE
|Out-of-sample bootstrap validation technique, AUC|
Caret AUC performance up to 40 percentage points
|||2017||Regression||Aim is to validate the source code metrics and identify a suitable set of source code metrics||5 training algorithms: GD, GDM, GDX, NM, LM||Heterogeneous linear and nonlinear ensemble methods||56 open-source Java projects from PROMISE Repository||10-fold CV, t-test, ULR analysis|
Neural network with Levenberg Marquardt (LM) is the best
|||2017||Classification||Replicate 24 CDPD approaches, and compare on 5 different datasets||DT, LR, NB, SVM||LE, RF, BAG-DT, BAG-NB, BOOST-DT, BOOST-NB||5 available datasets: JURECZKO, NASA MDP, AEEEM, NETGENE, RELINK||Recall, PR, ACC, G-measure, F-score, MCC, AUC|
|||2017||Classification||Just-in-time defect prediction (TLEL)||NB, SVM, DT, LDA, NN||Bagging, stacking||Bugzilla, Columba, JDT, Platform, Mozilla, and PostgreSQL||10-fold CV, F-score|
|||2017||Classification||Adaptive Selection of Classifiers in bug prediction (ASCI) method is proposed.||Base classifiers: LOG (binary logistic regression), NB, RBF, MLP, DT||Voting||Ginger Bread (2.3.2 and 2.3.7), Ice Cream Sandwich (4.0.2 and 4.0.4), and JellyBean (4.1.2, 4.2.2 and 4.3.1)||10-fold, inter-release validation|
AUC for NB, LB, MLP is >0.7
|||2018||Classification||MULTI method for JIT-SDP (just in time software defect prediction)||EALR, SL, RBFNet|
Unsupervised: LT, AGE
|Bagging, AdaBoost, Rotation Forest, RS||Bugzilla, Columba, Eclipse JDT, Eclipse Platform, Mozilla, PostgreSQ||CV, timewise-CV, ACC, and POPT|
MULTI performs significantly better than all the baselines
|||2007||Classification||To found pre- and post-release defects for every package and file||LR||—||Eclipse 2.0, 2.1, 3.0||PR, recall, ACC|
|||2014||Clustering||Cluster ensemble with PSO for clustering the software modules (fault-prone or not fault-prone)||PSO clustering algorithm||KM-E, KM-M, PSO-E, PSO-M and EM||Nasa MDP, PROMISE|
|||2015||Classification||Defect identification by applying DM algorithms||NB, J48, MLP||—||PROMISE, NASA MDP dataset: CM1, JM1, KC1, KC3, MC1, MC2, MW1, PC1, PC2, PC3||10-fold CV, ACC, PR,|
FMLP is the best
|||2015||Classification||To show the attributes that predict the defective state of software modules||NB, NN, association rules, DT||Weighted voting rule of the four algorithms||NASA datasets: CM1, JM1, KC1, KC2, PC1||PR, recall, ACC, F-score|
NB > NN > DT
|||2016||Classification||Authors proposed a model that finds fault-proneness||NB, LR, LivSVM, MLP, SGD, SMO, VP, LR Logit Boost, Decision Stamp, RT, REP Tree||RF||Camel1.6, Tomcat 6.0, Ant 1.7, jEdit4.3, Ivy 2.0, arc, e-learning, berek, forrest 0.8, zuzel, Intercafe, and Nieruchomosci||10-fold CV, AUC|
AUC = 0.661
|||2016||Classification||GA to select suitable source code metrics||LR, ELM, SVML, SVMR, SVMP||—||30 open-source software projects from PROMISE repository from DS1 to DS30||5-fold CV, F-score, ACC, pairwise t-test|
|||2016||—||Weighted least-squares twin support vector machine (WLSTSVM) to find misclassification cost of DP||SVM, NB, RF, LR, KNN, BN, cost-sensitive neural network||—||PROMISE repository: CM1, KC1, PC1, PC3, PC4, MC2, KC2, KC3||10-fold CV, PR, recall, F-score, G-mean|
Wilcoxon signed rank test
|||2016||—||A multi-objective naive Bayes learning techniques MONB, MOBNN||NB, LR, DT, MODT, MOLR, MONB||—||Jureczko datasets obtained from PROMISE repository||AUC, Wilcoxon rank test|
CP MO NB (0.72) produces the highest value
|||2016||Classification||A software defect prediction model to find faulty components of a software||Hybrid filter approaches|
FISHER, MR, ANNIGMA.
|—||KC1, KC2, JM1, PC1, PC2, PC3, and PC4 datasets||ACC, ent filters, ACC 90%|
|||2017||Classification||Propose an hybrid method called TSC-RUS + S||A random undersampling based on two-step cluster (TSC)||Stacking: DT, LR, kNN, NB||NASA MDP: i.e., CM1, KC1, KC3, MC2, MW1, PC1, PC2, PC3, PC4||10-fold CV, AUC, (TSC-RUS + S) is the best|
|||2017||Classification||Analyze five popular ML algorithms for software defect prediction||ANN, PSO, DT, NB, LC||—||Nasa and PROMISE datasets: CM1, JM1, KC1, KC2, PC1, KC1-LC||10-fold CV|
ANN < DT
|||2018||Classification||Three well-known ML techniques are compared.||NB, DT, ANN||—||Three different datasets|
DS1, DS2, DS3
|ACC, PR, recall, F, ROC|
DT > ANN > NB
|||2018||Classification||ML algorithms are compared with CODEP||LR, BN, RBF, MLP, alternating decision tree (ADTree), and DT||Max, CODEP, Bagging J48, Bagging NB, Boosting J48, Boosting NB, RF||PROMISE: Ant, Camel, ivy, Jedit, Log4j, Lucene, Poi, Prop, Tomcat, Xalan||F-score, PR, AUC ROC|
Max performs better than CODEP
3.1.1 Defect prediction using ensemble learning techniques
Ensemble learning combines several base learning models to obtain better performance than individual models. These base learners can be acquired with:
Different learning algorithms
Different parameters of the same algorithm
Different training sets
The commonly used ensemble techniques bagging, boosting, and stacking are shown in Figure 3 and briefly explained in this part. Bagging (which stands for bootstrap aggregating) is a kind of parallel ensemble. In this method, each model is built independently, and multiple training datasets are generated from the original dataset through random selection of different feature subsets; thus, it aims to decrease variance. It combines the outputs of each ensemble member by a voting mechanism. Boosting can be described as sequential ensemble. First, the same weights are assigned to data instances; after training, the weight of wrong predictions is increased, and this process is repeated as the ensemble size. Finally, it uses a weighted voting scheme, and in this way, it aims to decrease bias. Stacking is a technique that uses predictions from multiple models via a meta-classifier.
Some software defect prediction studies have compared ensemble techniques to determine the best performing one [10, 18, 21, 39, 40]. In a study conducted by Wang et al. , different ensemble techniques such as bagging, boosting, random tree, random forest, random subspace, stacking, and voting were compared to each other and a single classifier (NB). According to the results, voting and random forest clearly exhibited better performance than others. In a different study , ensemble methods were compared with more than one base learner (NB, BN, SMO, PART, J48, RF, random tree, IB1, VFI, DT, NB tree). For boosted SMO, bagging J48, and boosting and bagging RT, performance of base classifiers was lower than that of ensemble learner classifiers.
In study , a new method was proposed of mixing feature selection and ensemble learning for defect classification. Results showed that random forests and the proposed algorithm are not affected by poor features, and the proposed algorithm outperforms existing single and ensemble classifiers in terms of classification performance. Another comparative study  used seven composite algorithms (Ave, Max, Bagging C4.5, bagging naive Bayes (NB), Boosting J48, Boosting naive Bayes, and RF) and one composite state-of-the art study for cross-project defect prediction. The Max algorithm yielded the best results regarding F-score in terms of classification performance.
Bowes et al.  compared RF, NB, Rpart, and SVM algorithms to determine whether these classifiers obtained the same results. The results demonstrated that a unique subset of defects can be discovered by specific classifiers. However, whereas some classifiers are steady in the predictions they make, other classifiers change in their predictions. As a result, ensembles with decision-making without majority voting can perform best.
One of the main problems of SDP is the imbalance between the defect and non-defect classes of the dataset. Generally, the number of defected instances is greater than the number of non-defected instances in the collected data. This situation causes the machine learning algorithms to perform poorly. Wang and Yao  compared five class-imbalanced learning methods (RUS, RUS-bal, THM, BNC, SMB) and NB and RF algorithms and proposed the dynamic version of AdaBoost.NC. They utilized balance, G-mean, and AUC measures for comparison. Results showed that AdaBoost.NC and naive Bayes are better than the other seven algorithms in terms of evaluation measures. Dynamic AdaBoost.NC showed better defect detection rate and overall performance than the original AdaBoost.NC. To handle the class imbalance problem, studies  have compared different methods (sampling, cost sensitive, hybrid, and ensemble) by taking into account evaluation metrics such as MCC and receiver operating characteristic (ROC).
As shown in Table 1, the most common datasets used in the defect prediction studies [17, 18, 19, 39] are the NASA MDP dataset and PROMISE repository datasets. In addition, some studies utilized open-source projects such as Bugzilla Columba and Eclipse JDT [26, 27], and other studies used Android application data [22, 23].
3.1.2 Defect prediction studies without ensemble learning
Although use of ensemble learning techniques has dramatically increased recently, studies that do not use ensemble learning are still conducted and successful. For example, in study , prediction models were created using source code metrics as in ensemble studies but by using different feature selection techniques such as genetic algorithm (GA).
To overcome the class imbalance problem, Tomar and Agarwal  proposed a prediction system that assigns lower cost to non-defective data samples and higher cost to defective samples to balance data distribution. In the absence of enough data within a project, required data can be obtained from cross projects; however, in this case, this situation may cause class imbalance. To solve this problem, Ryu and Baik  proposed multi-objective naïve Bayes learning for cross-project environments. To obtain significant software metrics on cloud computing environments, Ali et al. used a combination of filter and wrapper approaches . They compared different machine learning algorithms such as NB, DT, and MLP [29, 37, 38, 41].
3.2 Data mining in effort estimation
Software effort estimation (SEE) is critical for a company because hiring more employees than required will cause loss of revenue, while hiring fewer employees than necessary will result in delays in software project delivery. The estimation analysis helps to predict the amount of effort (in person hours) needed to develop a software product. Basic steps of software estimation can be itemized as follows:
Determine project objectives and requirements.
Design the activities.
Estimate product size and complexity.
Compare and repeat estimates.
SEE contains requirements and testing besides predicting effort estimation . Many research and review studies have been conducted in the field of SEE. Recently, a survey  analyzed effort estimation studies that concentrated on ML techniques and compared them with studies focused on non-ML techniques. According to the survey, case-based reasoning (CBR) and artificial neural network (ANN) were the most widely used techniques. In 2014, Dave and Dutta  examined existing studies that focus only on neural network.
The current effort estimation studies using DM and ML techniques are available in Table 2. This table summarizes the prominent studies in terms of aspects such as year, data mining task, aim, datasets, and metrics. Table 2 indicates that neural network is the most widely used technique for the effort estimation task.
|Ref.||Year||Task||Objective||Algorithms||Ensemble learning||Dataset||Evaluation metrics and results|
|||2008||Regression||Ensemble of neural networks with associative memory (ENNA)||NN, MLP, KNN||Bagging||NASA, NASA 93, USC, SDR, Desharnais||MMRE, MdMRE and PRED(L)|
For ENNA PRED(25) = 36.4
For neural network PRED(25) = 8
|||2009||Regression||Authors proposed the ensemble of neural networks with associative memory (ENNA)||NN, MLP, KNN||Bagging||NASA, NASA 93, USC, SDR, Desharnais||Random subsampling, t-test|
MMRE, MdMRE, and PRED(L)
ENNA is the best
|||2010||Regression||To show the effectiveness of SVR for SEE||SVR, RBF||—||Tukutuku||LOOCV, MMRE, Pred(25), MEMRE, MdEMRE|
SVR outperforms others
|||2011||Regression||To evaluate whether readily available ensemble methods enhance SEE||MLP, RBF, RT||Bagging||5 datasets from PROMISE: cocomo81, nasa93, nasa, sdr, and Desharnais|
8 datasets from ISBSG repository
|MMRE, MdMRE, PRED(25)|
RTs and Bagging with MLPs perform similarly
|||2012||Regression||To show the measures behave in SEE and to create good ensembles||MLP, RBF, REPTree,||Bagging||cocomo81, nasa93, nasa, cocomo2, desharnais, ISBSG repository||MMRE, PRED(25), LSD, MdMRE, MAE, MdAE|
Pareto ensemble for all measures, except LSD.
|||2012||Regression||To use cross-company models to create diverse ensembles able to dynamically adapt to changes||WC RTs, CC-DWM||WC-DWM||3 datasets from ISBSG repository (ISBSG2000, ISBSG2001, ISBSG) 2 datasets from PROMISE (CocNasaCoc81 and CocNasaCoc81Nasa93)||MAE, Friedman test|
Only DCL could improve upon RT
CC data potentially beneficial for improving SEE
|||2012||Regression||To generate estimates from ensembles of multiple prediction methods||CART, NN, LR, PCR, PLSR, SWR, ABE0-1NN, ABE0-5NN||Combining top M solo methods||PROMISE||MAR, MMRE, MdMRE, MMER, MBRE, MIBRE.|
Combinations perform better than 83%
|||2012||Classification/regression||DM techniques to estimate software effort.||M5, CART, LR, MARS, MLPNN, RBFNN, SVM||—||Coc81, CSC, Desharnais, Cocnasa, Maxwell, USP05||MdMRE, Pred(25), Friedman test|
Log + OLS > LMS, BC + OLS, MARS, LS-SVM
|||2013||Clustering/classification||Estimation of software development effort||NN, ABE, C-means||—||Maxwell||3-fold CV and LOOCV, RE, MRE, MMRE, PRED|
|||2014||Regression||ANNs are examined using COCOMO model||MLP, RBFNN, SVM, PSO-SVM Extreme learning Machines||—||COCOMO II Data||MMRE, PRED|
PSO-SVM is the best
|||2014||—||A hybrid model based on GA And ACO for optimization||GA, ACO||—||NASA datasets||MMRE, the proposed method is the best|
|||2015||Regression||To display the effect of data preprocessing techniques on ML methods in SEE||CBR, ANN, CART|
Preprocessing rech: MDT, LD, MI, FS, CS, FSS, BSS
|—||ISBSG, Desharnais, Kitchenham, USPFT||CV, MBRE, PRED (0.25), MdBRE|
|||2016||Regression||Four neural network models are compared with each other.||MLP, RBFNN, GRNN, CCNN||—||ISBSG repository||10-fold CV, MAR|
The CCNN outperforms the other three models
|||2016||Regression||To propose a model based on Bayesian network||GA and PSO||—||COCOMO NASA Dataset||DIR, DRM|
The proposed model is best
|||2016||Classification/regression||A hybrid model using SVM and RBNN compared against previous models||SVM, RBNN||—||Dataset1 = 45 industrial projects|
Dataset2 = 65 educational projects
|LOOCV, MAE, MBRE, MIBRE, SA|
The proposed approach is the best
|||2017||Classification||To estimate software effort by using ML techniques||SVM, KNN||Boosting: kNN and SVM||Desharnais, Maxwell||LOOCV, k-fold CV|
ACC = 91.35% for Desharnais
ACC = 85.48% for Maxwell
Several studies have compared ensemble learning methods with single learning algorithms [45, 46, 48, 49, 51, 60] and examined them on cross-company (CC) and within-company (WC) datasets . The authors observed that ensemble methods obtained by a proper combination of estimation methods achieved better results than single methods. Various ML techniques such as neural network, support vector machine (SVM), and k-nearest neighbor are commonly used as base classifiers for ensemble methods such as bagging and boosting in software effort estimation. Moreover, their results indicate that CC data can increase performance over WC data for estimation techniques .
In addition to the abovementioned studies, researchers have conducted studies without using ensemble techniques. The general approach is to investigate which DM technique has the best effect on performance in software effort estimation. For instance, Subitsha and Rajan  compared five different algorithms—MLP, RBFNN, SVM, ELM, and PSO-SVM—and Nassif et al.  investigated four neural network algorithms—MLP, RBFNN, GRNN, and CCNN. Although neural networks are widely used in this field, missing values and outliers frequently encountered in the training set adversely affect neural network results and cause inaccurate estimations. To overcome this problem, Khatibi et al.  split software projects into several groups based on their similarities. In their studies, the C-means clustering algorithm was used to determine the most similar projects and to decrease the impact of unrelated projects, and then analogy-based estimation (ABE) and NN were applied. Another clustering study by Azzeh and Nassif  combined SVM and bisecting k-medoids clustering algorithms; an estimation model was then built using RBFNN. The proposed method was trained on historical use case points (UCP).
Zare et al.  and Maleki et al.  utilized optimization methods for accurate cost estimation. In the former study, a model was proposed based on Bayesian network with genetic algorithm and particle swarm optimization (PSO). The latter study used GA to optimize the effective factors’ weight, and then trained by ant colony optimization (ACO). Besides conventional effort estimation studies, researchers have utilized machine learning techniques for web applications. Since web-based software projects are different from traditional projects, the effort estimation process for these studies is more complex.
It is observed that PRED(25) and MMRE are the most popular evaluation metrics in effort estimation. MMRE stands for the mean magnitude relative error, and PRED(25) measures prediction accuracy and provides a percentage of predictions within 25% of actual values.
3.3 Data mining in vulnerability analysis
Vulnerability analysis is becoming the focal point of system security to prevent weaknesses in the software system that can be exploited by an attacker. Description of software vulnerability is given in many different resources in different ways . The most popular and widely utilized definition appears in the Common Vulnerabilities and Exposures (CVE) 2017 report as follows:
Vulnerability is a weakness in the computational logic found in software and some hardware components that, when exploited, results in a negative impact to confidentiality, integrity or availability.
Vulnerability analysis may require many different operations to identify defects and vulnerabilities in a software system. Vulnerabilities, which are a special kind of defect, are more critical than other defects because attackers exploit system vulnerabilities to perform unauthorized actions. A defect is a normal problem that can be encountered frequently in the system, easily found by users or developers and fixed promptly, whereas vulnerabilities are subtle mistakes in large codes [62, 63]. Wijayasekara et al. claim that some bugs have been identified as vulnerabilities after being publicly announced in bug databases . These bugs are called “hidden impact vulnerabilities” or “hidden impact bugs.” Therefore, the authors proposed a hidden impact vulnerability identification methodology that utilizes text mining techniques to determine which bugs in bug databases are vulnerabilities. According to the proposed method, a bug report was taken as input, and it produces feature vector after applying text mining. Then, classifier was applied and revealed whether it is a bug or a vulnerability. The results given in  demonstrate that a large proportion of discovered vulnerabilities were first described as hidden impact bugs in public bug databases. While bug reports were taken as input in that study, in many other studies, source code is taken as input. Text mining is a highly preferred technique for obtaining features directly from source codes as in the studies [65, 66, 67, 68, 69]. Several studies [63, 70] have compared text mining-based models and software metrics-based models.
In the security area of software systems, several studies have been conducted related to DM and ML. Some of these studies are compared in Table 3, which shows the data mining task and explanation of the studies, the year they were performed, the algorithms that were used, the type of vulnerability analysis, evaluation metrics, and results. In this table, the best performing algorithms according to the evaluation criteria are shown in bold.
|Ref.||Year||Task||Objective||Algorithms||Type||Dataset description||Evaluation metrics and results|
|||2011||Clustering||Obtaining software vulnerabilities based on RDBC||RDBC||Static||Database is built by RD-Entropy||FNR, FPR|
|||2011||Classification/regression||To predict the time to next vulnerability||LR, LMS, MLP, RBF, SMO||Static||NVD, CPE, CVSS||CC, RMSE, RRSE|
|||2012||Text mining||Analysis of source code as text||RBF, SVM||Static||K9 email client for the Android platform||ACC, PR, recall|
ACC = 0.87, PR = 0.85, recall = 0.88
|||2012||Classification/text mining||To identify vulnerabilities in bug databases||—||Static||Linux kernel MITRE CVE and MySQL bug databases||BDR, TPR, FPR|
32% (Linux) and 62% (MySQL) of vulnerabilities
|||2014||Classification/regression||Combine taint analysis and data mining to obtain vulnerabilities||ID3, C4.5/J48, RF, RT, KNN, NB, Bayes Net, MLP, SVM, LR||Hybrid||A version of WAP to collect the data||10-fold CV, TPD, ACC, PR, KAPPA|
ACC = 90.8%, PR = 92%, KAPPA = 81%
|||2014||Clustering||Identify vulnerabilities from source codes using CPG||—||Static||Neo4J and InfiniteGraph databases||—|
|||2014||Classification||Comparison of software metrics with text mining||RF||Static||Vulnerabilities from open-source web apps (Drupal, Moodle, PHPMyAdmin)||3-fold CV, recall, IR, PR, FPR, ACC.|
Text mining provides benefits overall
|||2014||Classification||To create model in the form of a binary classifier using text mining||NB, RF||Static||Applications from the F-Droid repository and Android||10-fold CV, PR, recall|
PR and recall ≥ 80%
|||2015||Classification||A new approach (VCCFinder) to obtain potentially dangerous codes||SVM-based detection model||—||The database contains 66 GitHub projects||k-fold CV, false alarms <99% at the same level of recall|
|||2015||Ranking/classification||Comparison of text mining and software metrics models||RF||—||Vulnerabilities from open-source web apps (Drupal, Moodle, PHPMyAdmin)||10-fold CV|
Metrics: ER-BCE, ERBPP, ER-AVG
|||2015||Clustering||Search patterns for taint-style vulnerabilities in C code||Hierarchical clustering (complete-linkage)||Static||5 open-source projects: Linux, OpenSSL, Pidgin, VLC, Poppler (Xpdf)||Correct source, correct sanitization, number of traversals, generation time, execution time, reduction, amount of code review <95%|
|||2016||Classification||Static and dynamic features for classification||LR, MLP, RF||Hybrid||Dataset was created by analyzing 1039 test cases from the Debian Bug Tracker||FPR, FNR|
Detect 55% of vulnerable programs
|||2017||Classification||1. Employ a deep neural network|
2. Combine N-gram analysis and feature selection
|Deep neural network||—||Feature extraction from 4 applications (BoardGameGeek, Connectbot, CoolReader, AnkiDroid)||10 times using 5-fold CV|
ACC = 92.87%, PR = 94.71%, recall = 90.17%
|||2017||Text mining||To analyze characteristics of software vulnerability from source files||—||—||CVE, CWE, NVD databases||PR = 70%, recall = 60%|
|||2017||Text mining||Deep learning (LSTM) is used to learn semantic and syntactic features in code||RNN, LSTM, DBN||—||Experiments on 18 Java applications from the Android OS platform||10-fold CV, PR, recall, and F-score|
Deep Belief Network
PR, recall, and F-score > 80%
|||2018||Classification||Identify bugs by extracting text features from C source code||NB, KNN, K-means, NN, SVM, DT, RF||Static||NVD, Cat, Cp, Du, Echo, Head, Kill, Mkdir, Nl, Paste, Rm, Seq, Shuf, Sleep, Sort, Tail, Touch, Tr, Uniq, Wc, Whoami||5-fold CV ACC, TP, TN|
ACC = 74%
|||2018||Regression||A deep learning-based vulnerability detection system (VulDeePecker)||BLSTM NN||Static||NIST: NVD and SAR project||10-fold CV, PR, recall, F-score|
F-score = 80.8%
|||2018||Classification||A mapping between existing requirements and vulnerabilities||LR, SVM, NB||—||Data is gathered from Apache Tomcat, CVE, requirements from Bugzilla, and source code is collected from Github||PR, recall, F-score|
LSI > SVM
Vulnerability analysis can be categorized into three types: static vulnerability analysis, dynamic vulnerability analysis, and hybrid analysis [61, 80]. Many studies have applied the static analysis approach, which detects vulnerabilities from source code without executing software, since it is cost-effective. Few studies have performed the dynamic analysis approach, in which one must execute software and check program behavior. The hybrid analysis approach [72, 76] combines these two approaches.
As revealed in Table 3, in addition to classification and text mining, clustering techniques are also frequently seen in software vulnerability analysis studies. To detect vulnerabilities in an unknown software data repository, entropy-based density clustering  and complete-linkage clustering  were proposed. Yamaguchi et al.  introduced a model to represent a large number of source codes as a graph called control flow graph (CPG), a combination of abstract syntax tree, CFG, and program dependency graph (PDG). This model enabled the discovery of previously unknown (zero-day) vulnerabilities.
To learn the time to next vulnerability, a prediction model was proposed in the study . The result could be a number that refers to days or a bin representing values in a range. The authors used regression and classification techniques for the former and latter cases, respectively.
In vulnerability studies, issue tracking systems like Bugzilla, code repositories like Github, and vulnerability databases such as NVD, CVE, and CWE have been utilized . In addition to these datasets, some studies have used Android [65, 68, 69] or web [63, 70, 72] (PHP source code) datasets. In recent years, researchers have concentrated on deep learning for building binary classifiers , obtaining vulnerability patterns , and learning long-term dependencies in sequential data  and features directly from the source code .
Li et al.  note two difficulties of vulnerability studies: demanding, intense manual labor and high false-negative rates. Thus, the widely used evaluation metrics in vulnerability analysis are false-positive rate and false-negative rate.
3.4 Data mining in design pattern mining
During the past years, software developers have used design patterns to create complex software systems. Thus, researchers have investigated the field of design patterns in many ways [82, 83]. Fowler defines a pattern as follows:
“A pattern is an idea that has been useful in one practical context and will probably be useful in others.” 
Patterns display relationships and interactions between classes or objects. Well-designed object-oriented systems have various design patterns integrated into them. Design patterns can be highly useful for developers when they are used in the right manner and place. Thus, developers avoid recreating methods previously refined by others. The pattern approach was initially presented in 1994 by four authors—namely, Erich Gama, Richard Helm, Ralph Johnson, and John Vlissides—called the Gang of Four (GOF) in 1994 . According to the authors, there are three types of design patterns:
Creational patterns provide an object creation mechanism to create the necessary objects based on predetermined conditions. They allow the system to call appropriate object and add flexibility to the system when objects are created. Some creational design patterns are factory method, abstract factory, builder, and singleton.
Structural patterns focus on the composition of classes and objects to allow the establishment of larger software groups. Some of the structural design patterns are adapter, bridge, composite, and decorator.
Behavioral patterns determine common communication patterns between objects and how multiple classes behave when performing a task. Some behavioral design patterns are command, interpreter, iterator, observer, and visitor.
Many design pattern studies exist in the literature. Table 4 shows some design pattern mining studies related to machine learning and data mining. This table contains the aim of the study, mining task, year, and design patterns selected by the study, input data, dataset, and results of the studies.
|Ref.||Year||Task||Objective||Algorithms||EL||Selected design patterns||Input data||Dataset||Evaluation metrics and results|
|||2012||Text classification||Two-phase method:|
1—text classification to
2—learning design patterns
|NB, KNN, DT, SVM||—||46 security patterns, 34 Douglass patterns, 23 GoF patterns||Documents||Security, Douglass, GoF||PR, recall, EWM|
PR = 0.62, recall = 0.75
|||2013||Regression||An approach is to find a valid instance of a DP or not||ANN||—||Adapter, command, composite, decorator, observer, and proxy||Set of candidate classes||JHotDraw 5.1 open-source application||10 fold CV, PR, recall|
|||2014||Graph mining||Sub-graph mining-based approach||CloseGraph||—||—||Java source code||Open-source project:YARI, Zest, JUnit, JFreeChart, ArgoUML||No any empirical comparison|
|||2015||Classification/clustering||MARPLE-DPD is developed to classify instances whether it is a bad or good instance||SVM, DT, RF, K-means, ZeroR, OneR, NB, JRip, CLOPE.||—||Classification for singleton and adapter|
Classification and clustering for composite, decorator, and factory method
|—||10 open-source software systems|
DPExample, QuickUML 2001, Lexi v0.1.1 alpha, JRefactory v2.6.24, Netbeans v1.0.x, JUnit v3.7, JHotDraw v5.1, MapperXML v1.9.7, Nutch v0.4, PMD v1.8
|10-fold CV, ACC, F-score, AUC|
ACC > =85%
|||2015||Regression||A new method (SVM-PHGS) is proposed||Simple Logistic, C4.5, KNN, SVM, SVM-PHGS||—||Adapter, builder, composite, factory method, iterator, observer||Source code||P-mart repository||PR, recall, F-score, FP|
PR = 0.81, recall =0.81, F-score = 0.81, FP = 0.038
|||2016||Classification||Design pattern recognition using ML algorithms.||LRNN, DT||—||Abstract factory, adapter patterns||Source code||Dataset with 67 OO metrics, extracted by JBuilder tool||5-fold CV, ACC, PR, recall, F-score|
ACC = 100% by LRNN
|||2016||Classification||Three aspects: design patterns, software metrics, and supervised learning methods||Layer Recurrent Neural Network (LRNN)||RF||Abstract factory, adapter, bridge, singleton,|
and template method
|Source code||Dataset with 67 OO metrics, extracted by JBuilder tool||PR, recall, F-score|
F-score = 100% by LRNN and RF
ACC = 100% by RF
|||2017||Classification||1. Creation of metrics-oriented dataset|
2. Detection of software design patterns
|ANN, SVM||RF||Abstract factory, adapter, bridge, composite, and Template||Source code||Metrics extracted from source codes (JHotDraw, QuickUML, and Junit)||5-fold and 10-fold CV, PR, recall, F-score|
ANN, SVM, and RF yielded to 100% PR for JHotDraw
|||2017||Classification||Detection of design motifs based on a set of directed semantic graphs||Strong graph simulation, graph matching||—||All three groups: creational, structural, behavioral||UML class diagrams||—||PR, recall|
High accuracy by the proposed method
|||2017||Text categorization||Selection of more appropriate design patterns||Fuzzy c-means||Ensemble-IG||Various design patterns||Problem definitions of design patterns||DP, GoF, Douglass, Security||F-score|
|||2018||Classification||Finding design pattern and smell pairs which coexist in the code||J48||—||Used patterns: adapter, bridge, Template, singleton||Source code||Eclipse plugin Web of Patterns|
The tool selected for code smell detection is iPlasma
|PR, recall, F-score, PRC, ROC|
Singleton pattern shows no presence of bad smells
In design pattern mining, detecting the design pattern is a frequent study objective. To do so, studies have used machine learning algorithms [87, 89, 90, 91], ensemble learning , deep learning , graph theory , and text mining [86, 95].
In study , the training dataset consists of 67 object-oriented (OO) metrics extracted by using the JBuilder tool. The authors used LRNN and decision tree techniques for pattern detection. Alhusain et al.  generated training datasets from existing pattern detection tools. The ANN algorithm was selected for pattern instances. Chihada et al.  created training data from pattern instances using 45 OO metrics. The authors utilized SVM for classifying patterns accurately. Another metrics-oriented dataset was developed by Dwivedi et al. . To evaluate the results, the authors benefited from three open-source software systems (JHotDraw, QuickUML, and JUnit) and applied three classifiers, SVM, ANN, and RF. The advantage of using random forest is that it does not require linear features and can manage high-dimensional spaces.
To evaluate methods and to find patterns, open-source software projects such as JHotDraw, Junit, and MapperXML have been generally preferred by researchers. For example, Zanoni et al.  developed a tool called MARPLE-DPD by combining graph matching and machine learning techniques. Then, to obtain five design patterns, instances were collected from 10 open-source software projects, as shown in Table 4.
Design patterns and code smells are related issues: Code smell refers to symptoms in code, and if there are code smells in a software, its design pattern is not well constructed. Therefore, Kaur and Singh  checked whether design pattern and smell pairs appear together in a code by using J48 Decision Tree. Their obtained results showed that the singleton pattern had no presence of bad smells.
According to the studies summarized in the table, the most frequently used patterns are abstract factory and adapter. It has recently been observed that studies on ensemble learning in this field are increasing.
3.5 Data mining in refactoring
One of the SE tasks most often used to improve the quality of a software system is refactoring, which Martin Fowler has described as “a technique for restructuring an existing body of code, altering its internal structure without changing its external behavior” . It improves readability and maintainability of the source code and decreases complexity of a software system. Some of the refactoring types are: Add Parameter, Replace Parameter, Extract method, and Inline method .
Code smell and refactoring are closely related to each other: Code smells represent problems due to bad design and can be fixed during refactoring. The main challenge is to obtain which part of the code needs refactoring.
Some of data mining studies related to software refactoring are presented in Table 5. Some studies focus on historical data to predict refactoring  or to obtain both refactoring and software defects  using different data mining algorithms such as LMT, Rip, and J48. Results suggest that when refactoring increases, the number of software defects decreases, and thus refactoring has a positive effect on software quality.
|Ref.||Year||Task||Objective||Algorithms||EL||Dataset||Evaluation metrics and results|
|||2007||Regression||Stages: (1) data understanding, (2) preprocessing, (3) ML, (4) post-processing, (5) analysis of the results||J48, LMT, Rip, NNge||—||ArgoUML, Spring Framework||10-fold CV, PR, recall, F-score|
PR and recall are 0.8 for ArgoUML
|||2008||Classification||Finding the relationship between refactoring and defects||C4.5, LMT, Rip, NNge||—||ArgoUML, JBoss Cache, Liferay Portal, Spring Framework, XDoclet||PR, recall, F-score|
|||2014||Regression||Propose GA-based learning for software refactoring based on ANN||GA, ANN||—||Xerces-J, JFreeChart, GanttProject, AntApache, JHotDraw, and Rhino.||Wilcoxon test with a 99% confidence level (α = 0.01)|
|||2015||Regression||Removing defects with time series in a multi-objective approach||Multi-objective algorithm, based on NSGA-II, ARIMA||FindBugs, JFreeChart, Hibernate, Pixelitor, and JDI-Ford||Wilcoxon rank sum test with a 99% confidence level (α < 1%)|
|||2016||Web mining/clustering||Unsupervised learning approach to detect refactoring opportunities in service-oriented applications||PAM, K-means, COBWEB, X-Means||—||Two datasets of WSDL documents||COBWEB and K-means max. 83.33% and 0%, inter-cluster|
COBWEB and K-means min. 33.33% and 66.66% intra-cluster
|||2017||Clustering||A novel algorithm (HASP) for software refactoring at the package level||Hierarchical clustering algorithm||—||Three open-source case studies||Modularization Quality and Evaluation Metric Function|
|||2017||Classification||A technique to predict refactoring at class level||PCA, SMOTE|
|—||From tera- PROMISE Repository seven open-source software systems||10-fold CV, AUC, and ROC curves|
RBF kernel outperforms linear and polynomial kernel
The mean value of AUC for LS-SVM RBF kernel is 0.96
|||2017||Classification||Exploring the impact of clone refactoring (CR) on the test code size||LR, KNN, NB||RF||data collected from an open-source Java software system (ANT)||PR, recall, accuracy, F-score|
kNN and RF outperform NB
ACC (fitting (98%), LOOCV (95%), and 10 FCV (95%))
|||2017||—||Finding refactoring opportunities in source code||J48, BayesNet, SVM, LR||RF||Ant, ArgoUML, jEdit, jFreeChart, Mylyn||10-fold CV, PR, recall|
86–97% PR and 71–98% recall for proposed tech
|||2018||Classification||A learning-based approach (CREC) to extract refactored and non-refactored clone groups from repositories||C4.5, SMO, NB.||RF, Adaboost||Axis2, Eclipse.jdt.core, Elastic Search, JFreeChart, JRuby, and Lucene||PR, recall, F-score|
F-score = 83% in the within-project
F-score = 76% in the cross-project
|||2018||Clustering||Combination of the use of multi-objective and unsupervised learning to decrease developer’s effort||GMM, EM||—||ArgoUML, JHotDraw, GanttProject, UTest, Apache Ant, Azureus||One-way ANOVA with a 95% confidence level (α = 5%)|
While automated refactoring does not always give the desired result, manual refactoring is time-consuming. Therefore, one study  proposed a clustering-based recommendation tool by combining multi-objective search and unsupervised learning algorithm to reduce the number of refactoring options. At the same time, the number of refactoring that should be selected is decreasing with the help of the developer’s feedback.
Since many SE studies that apply data mining approaches exist in the literature, this article presents only a few of them. However, Figure 4 shows the current number of papers obtained from the Scopus search engine for each year from 2010 to 2019 by using queries in the title/abstract/keywords field. We extracted publications in 2020 since this year has not completed yet. Queries included (“data mining” OR “machine learning”) with (“defect prediction” OR “defect detection” OR “bug prediction” OR “bug detection”) for defect prediction, (“effort estimation” OR “effort prediction” OR “cost estimation”) for effort estimation, (“vulnerab*” AND “software” OR “vulnerability analysis”) for vulnerability analysis, and (“software” AND “refactoring”) for refactoring. As seen in the figure, the number of studies using data mining in SE tasks, especially defect prediction and vulnerability analysis, has increased rapidly. The most stable area in the studies is design pattern mining.
Figure 5 shows the publications studied in classification, clustering, text mining, and association rule mining as a percentage of the total number of papers obtained by a Scopus query for each SE task. For example, in defect prediction, the number of studies is 339 in the field of classification, 64 in clustering, 8 in text mining, and 25 in the field of association rule mining. As can be seen from the pie charts, while clustering is a popular DM technique in refactoring, no study related to text mining is found in this field. In other SE tasks, the preferred technique is classification, and the second is clustering.
Defect prediction generally compares learning algorithms in terms of whether they find defects correctly using classification algorithms. Besides this approach, in some studies, clustering algorithms were used to select futures  or to compare supervised and unsupervised methods . In the text mining area, to extract features from scripts, TF-IDF techniques were generally used [111, 112]. Although many different algorithms have been used in defect prediction, the most popular ones are NB, MLP, and RBF.
Figure 6 shows the number of document types (conference paper, book chapter, article, book) published between the years of 2010 and 2019. It is clearly seen that conference papers and articles are the most preferred research study type. It is clearly seen that there is no review article about data mining studies in design pattern mining.
Table 6 shows popular repositories that contain various datasets and their descriptions, which tasks they are used for, and hyperlinks to download. For example, the PMART repository includes source files of java projects, and the PROMISE repository has different datasets with software metrics such as cyclomatic complexity, design complexity, and lines of code. Since these repositories contain many datasets, no detailed information about them has been provided in this article.
|Nasa MDP||Defect Pred.||NASA’s Metrics Data Program|
|Android Git||Defect Pred.||Android version bug reports|
|PROMISE||Defect Pred. Effort Est.||It includes 20 datasets for defect prediction and cost estimation|
|Software Defect Pred. Data||Defect Pred.||It includes software metrics, # of defects, etc. Eclipse JDT: Eclipse PDE:|
|PMART||Design pattern mining||It has 22 patterns 9 Projects, 139 ins. Format: XML Manually detected and validated|
Refactoring can be applied at different levels; study  predicted refactoring at package level using hierarchical clustering, and another study  applied class-level refactoring using LS-SVM as learning algorithm, SMOTE for handling refactoring, and PCA for feature extraction.
5. Conclusion and future work
Data mining techniques have been applied successfully in many different domains. In software engineering, to improve the quality of a product, it is highly critical to find existing deficits such as bugs, defects, code smells, and vulnerabilities in the early phases of SDLC. Therefore, many data mining studies in the past decade have aimed to deal with such problems. The present paper aims to provide information about previous studies in the field of software engineering. This survey shows how classification, clustering, text mining, and association rule mining can be applied in five SE tasks: defect prediction, effort estimation, vulnerability analysis, design pattern mining, and refactoring. It clearly shows that classification is the most used DM technique. Therefore, new studies can focus on clustering on SE tasks.
|LMT||logistic model trees|
|Rip||repeated incremental pruning|
|NNge||nearest neighbor generalization|
|PCA||principal component analysis|
|PAM||partitioning around medoids|
|LS-SVM||least-squares support vector machines|
|MAE||mean absolute error|
|RBF||radial basis function|
|SMO||sequential minimal optimization|
|GMM||Gaussian mixture model|
|RUS-bal||balanced version of random undersampling|
|RBF||radial basis function|
|ROC||receiver operating characteristic|
|SMOTE||synthetic minority over-sampling technique|