Data Mining and Machine Learning for Software Engineering

Abstract

Software engineering is one of the most utilizable research areas for data mining. Developers have attempted to improve software quality by mining and analyzing software data. In any phase of software development life cycle (SDLC), while huge amount of data is produced, some design, security, or software problems may occur. In the early phases of software development, analyzing software data helps to handle these problems and lead to more accurate and timely delivery of software projects. Various data mining and machine learning studies have been conducted to deal with software engineering tasks such as defect prediction, effort estimation, etc. This study shows the open issues and presents related solutions and recommendations in software engineering, applying data mining and machine learning techniques.

1. Introduction

In recent years, researchers in the software engineering (SE) field have turned their interest to data mining (DM) and machine learning (ML)-based studies since collected SE data can be helpful in obtaining new and significant information. Software engineering presents many subjects for research, and data mining can give further insight to support decision-making related to these subjects.

Figure 1 shows the intersection of three main areas: data mining, software engineering, and statistics/math. A large amount of data is collected from organizations during software development and maintenance activities, such as requirement specifications, design diagrams, source codes, bug reports, program versions, and so on. Data mining enables the discovery of useful knowledge and hidden patterns from SE data. Math provides the elementary functions, and statistics determines probability, relationships, and correlation within collected data. Data science, in the center of the diagram, covers different disciplines such as DM, SE, and statistics.

Figure 1.

The intersection of data mining and software engineering with other areas of the field.

This study presents a comprehensive literature review of existing research and offers an overview of how to approach SE problems using different mining techniques. Up to now, review studies either introduce SE data descriptions [1], explain tools and techniques mostly used by researchers for SE data analysis [2], discuss the role of software engineers [3], or focus only on a specific problem in SE such as defect prediction [4], design pattern [5], or effort estimation [6]. Some existing review articles having the same target [7] are former, and some of them are not comprehensive. In contrast to the previous studies, this article provides a systematic review of several SE tasks, gives a comprehensive list of available studies in the field, clearly states the advantages of mining SE data, and answers “how” and “why” questions in the research area.

The novelties and main contributions of this review paper are fivefold.

• First, it provides a general overview of several SE tasks that have been the focus of studies using DM and ML, namely, defect prediction, effort estimation, vulnerability analysis, refactoring, and design pattern mining.

• Second, it comprehensively discusses existing data mining solutions in software engineering according to various aspects, including methods (clustering, classification, association rule mining, etc.), algorithms (k-nearest neighbor (KNN), neural network (NN), etc.), and performance metrics (accuracy, mean absolute error, etc.).

• Third, it points to several significant research questions that are unanswered in the recent literature as a whole or the answers to which have changed with the technological developments in the field.

• Fourth, some statistics related to the studies between the years of 2010 and 2019 are given from different perspectives: according to their subjects and according to their methods.

• Five, it focuses on different machine learning types: supervised and unsupervised learning, especially on ensemble learning and deep learning.

This paper addresses the following research questions:

RQ1. What kinds of SE problems can ML and DM techniques help to solve?

RQ2. What are the advantages of using DM techniques in SE?

RQ3. Which DM methods and algorithms are commonly used to handle SE tasks?

RQ4. Which performance metrics are generally used to evaluate DM models constructed in SE studies?

RQ5. Which types of machine learning techniques (e.g., ensemble learning, deep learning) are generally preferred for SE problems?

RQ6. Which SE datasets are popular in DM studies?

The remainder of this paper is organized as follows. Section 2 explains the knowledge discovery process that aims to extract interesting, potentially useful, and nontrivial information from software engineering data. Section 3 provides an overview of current work on data mining for software engineering grouped under five tasks: defect prediction, effort estimation, vulnerability analysis, refactoring, and design pattern mining. In addition, some machine learning studies are divided into subgroups, including ensemble learning- and deep learning-based studies. Section 4 gives statistical information about the number of highly validated research conducted in the last decade. Related works considered as fundamental by journals with a highly positive reputation are listed, and the specific methods they used and their categories and purposes are clearly expressed. In addition, widely used datasets related to SE are given. Finally, Section 5 offers concluding remarks and suggests future scientific and practical efforts that might improve the efficiency of SE actions.

2. Knowledge discovery from software engineering data

This section basically explains the consecutive critical steps that should be followed to discover beneficial knowledge from software engineering data. It outlines the order of necessary operations in this process and explains how related data flows among them.

Software development life cycle (SDLC) describes a process to improve the quality of a product in project management. The main phases of SDCL are planning, requirement analysis, designing, coding, testing, and maintenance of a project. In every phase of software development, some software problems (e.g., software bugs, security, or design problems) may occur. Correcting these problems in the early phases leads to more accurate and timely delivery of the project. Therefore, software engineers broadly apply data mining techniques for different SE tasks to solve SE problems and to enhance programming efficiency and quality.

Figure 2 presents the data mining and knowledge discovery process of SE tasks including data collection, data preprocessing, data mining, and evaluation. In the data collection phase, data are obtained from software projects such as bug reports, historical data, version control data, and mailing lists that include various information about the project’s versions, status, or improvement. In the data preprocessing phase, the data are preprocessed after collection by using different methods such as feature selection (dimensionality reduction), feature extraction, missing data elimination, class imbalance analysis, normalization, discretization, and so on. In the next phase, DM techniques such as classification, clustering, and association rule mining are applied to discover useful patterns and relationships in software engineering data and therefore to solve a software engineering problem such as defected or vulnerable systems, reused patterns, or parts of code changes. Mining and obtaining valuable knowledge from such data prevents errors and allows software engineers to deliver the project on time. Finally, in the evaluation phase, validation techniques are used to assess the data mining results such as k-fold cross validation for classification. The commonly used evaluation measures are accuracy, precision, recall, F-score, area under the curve (AUC) for classification, and sum of squared errors (SSE) for clustering.

Figure 2.

KDD process for software engineering.

3. Data mining in software engineering

In this review, we examine data mining studies in various SE tasks and evaluate commonly used algorithms and datasets.

3.1 Data mining in defect prediction

A defect means an error, failure, flaw, or bug that causes incorrect or unexpected results in a system [8]. A software system is expected to be without any defects since software quality represents a capacity of the defect-free percentage of the product [9]. However, software projects often do not have enough time or people working on them to extract errors before a product is released. In such a situation, defect prediction methods can help to detect and remove defects in the initial stages of the SDLC and to improve the quality of the software product. In other words, the goal of defect prediction is to produce robust and effective software systems. Hence, software defect prediction (SDP) is an important topic for software engineering because early prediction of software defects could help to reduce development costs and produce more stable software systems.

Various studies have been conducted on defect prediction using different metrics such as code complexity, history-based metrics, object-oriented metrics, and process metrics to construct prediction models [10, 11]. These models can be considered on a cross-project or within-project basis. In within-project defect prediction (WPDP), a model is constructed and applied on the same project [12]. For within-project strategy, a large amount of historical defect data is needed. Hence, in new projects that do not have enough data to train, cross-project strategy may be preferred [13]. Cross-project defect prediction (CPDP) is a method that involves applying a prediction model from one project to another, meaning that models are prepared by utilizing historical data from other projects [14, 15]. Studies in the field of CPDP have increased in recent years [10, 16]. However, there are some deficiencies in comparisons of prior studies since they cannot be replicated because of the difference in utilizing evaluation metrics or preparation way of training data. Therefore, Herbold et al. [16] tried to replicate different CPDP methods previously proposed and find which approach performed best in terms of metrics such as F-score, area under the curve (AUC), and Matthews correlation coefficient (MCC). Results showed that 7- or 8-year approaches may perform better. Another study [17] replicated prior work to demonstrate whether the determination of classification techniques is important. Both noisy and cleaned datasets were used, and the same results were obtained from the two datasets. However, new dataset gave better results for some classification algorithms. For this reason, authors claimed that the selection of classification techniques affects the performance of the model.

Numerous defect prediction studies have been conducted using DM techniques. In the following subsections, we will explain these studies in terms of whether they apply ensemble learning or not. Some defect prediction studies in SE are compared in Table 1. The objective of the studies, the year they were conducted, algorithms, ensemble learning techniques and datasets in the studies, and the type of data mining tasks are shown in this table. The bold entries in Table 1 have better performance than other algorithms in that study.

Ref.	Year	Task	Objective	Algorithms	Ensemble learning	Dataset	Evaluation metrics and results
[18]	2011	Classification	Comparative study of various ensemble methods to find the most effective one	NB	Bagging, boosting, RT, RF, RS, AdaBoost, Stacking, and Voting	NASA datasets: CM1 JM1 KC1 KC2 KC3 KC4 MC1 MC2 MW1 PC1 PC2 PC3 PC4 PC5	10-fold CV, ACC, and AUC; Vote 88.48% random forest 87.90%
[19]	2013	Classification	Comparative study of class imbalance learning methods and proposed dynamic version of AdaBoost.NC	NB, RUS, RUS-bal, THM, SMB, BNC	RF, SMB, BNC, AdaBoost.NC	NASA and PROMISE repository: MC2, KC2, JM1, KC1, PC4, PC3, CM1, KC3, MW1, PC1	10-fold CV; Balance, G-mean and AUC, PD, PF
[20]	2014	Classification	Comparative study to deal with imbalanced data	Base Classifiers: C4.5, NB; Sampling: ROS, RUS, SMOTE	AdaBoost, Bagging, boosting, RF	NASA datasets: CM1, JM1, KC1, KC2, KC3, MC1, MC2, MW1, PC1, PC2, PC3, PC4, PC5	5 × 5 CV, MCC, ROC, results change according to characteristics of datasets
[17]	2015	Clustering/classification	To show that the selection of classification technique has an impact on the performance of software defect prediction models	Statistical: NB, Simple Logistic; Clustering: KM, EM; Rule based: Ripper, Ridor; NNs: RBF; Nearest neighbor: KNN; DTs: J48, LMT	Bagging, AdaBoost, rotation forest, random subspace	NASA: CM1, JM1, KC1, KC3, KC4, MW1, PC1, PC2, PC3, PC4; PROMISE: Ant 1.7, Camel 1.6, Ivy 1.4, Jedit 4, Log4j 1, Lucene 2.4, Poi 3, Tomcat 6, Xalan 2.6, Xerces 1.3	10 × 10-fold CV; AUC > 0.5; Scott-Knott test α = 0.05, simple logistic, LMT, and RF + base learner outperforms KNN and RBF
[21]	2015	Classification	Average probability ensemble (APE) learning module is proposed by combining feature selection and ensemble learning	APE system combines seven classifiers: SGD, weighted SVMs (W-SVMs), LR, MNB and Bernoulli naive Bayes (BNB)	RF, GB	NASA: CM1, JM1, KC1, KC3, KC4, MW1, PC1, PC2, PC3, PC4; PROMISE (RQ2): Ant 1.7, Camel 1.6, Ivy 1.4, Jedit 4, Log4j 1, Lucene 2.4, Poi 3, Tomcat 6, Xalan 2.6, Xerces 1.3	10 × 10-fold CV, AUC > 0.5; Scott-Knott test α = 0.05, simple logistic, LMT, and RF + base learner outperforms KNN and RBF
[22, 23]	2016	Classification	Comparative study of 18 ML techniques using OO metrics on six releases of Android operating system	LR, NB, BN, MLP, RBF; SVM, VP, CART, J48, ADT, Nnge, DTNB	Bagging, random forest, Logistic model trees, Logit Boost, Ada Boost	6 releases of Android app: Android 2.3.2, Android 2.3.7, Android 4.0.4, Android 4.1.2, Android 4.2.2, Android 4.3.1	10-fold, inter-release validation; AUC for NB, LB, MLP is >0.7
[24]	2016	Classification	Caret has been applied whether parameter settings can have a large impact on the performance of defect prediction models	NB, KNN, LR, partial least squares, NN, LDA, rule based, DT, SVM	Bagging, boosting	Cleaned NASA JM1, PC5; Proprietary from Prop-1 to Prop-5; Apache Camel 1.2, Xalan 2.5–2.6; Eclipse Platform 2.0–2.1–3.0, Debug 3.4, SWT 3.4, JDT, Mylyn, PDE	Out-of-sample bootstrap validation technique, AUC; Caret AUC performance up to 40 percentage points
[25]	2017	Regression	Aim is to validate the source code metrics and identify a suitable set of source code metrics	5 training algorithms: GD, GDM, GDX, NM, LM	Heterogeneous linear and nonlinear ensemble methods	56 open-source Java projects from PROMISE Repository	10-fold CV, t-test, ULR analysis; Neural network with Levenberg Marquardt (LM) is the best
[16]	2017	Classification	Replicate 24 CDPD approaches, and compare on 5 different datasets	DT, LR, NB, SVM	LE, RF, BAG-DT, BAG-NB, BOOST-DT, BOOST-NB	5 available datasets: JURECZKO, NASA MDP, AEEEM, NETGENE, RELINK	Recall, PR, ACC, G-measure, F-score, MCC, AUC
[26]	2017	Classification	Just-in-time defect prediction (TLEL)	NB, SVM, DT, LDA, NN	Bagging, stacking	Bugzilla, Columba, JDT, Platform, Mozilla, and PostgreSQL	10-fold CV, F-score
[13]	2017	Classification	Adaptive Selection of Classifiers in bug prediction (ASCI) method is proposed.	Base classifiers: LOG (binary logistic regression), NB, RBF, MLP, DT	Voting	Ginger Bread (2.3.2 and 2.3.7), Ice Cream Sandwich (4.0.2 and 4.0.4), and JellyBean (4.1.2, 4.2.2 and 4.3.1)	10-fold, inter-release validation; AUC for NB, LB, MLP is >0.7
[27]	2018	Classification	MULTI method for JIT-SDP (just in time software defect prediction)	EALR, SL, RBFNet; Unsupervised: LT, AGE	Bagging, AdaBoost, Rotation Forest, RS	Bugzilla, Columba, Eclipse JDT, Eclipse Platform, Mozilla, PostgreSQ	CV, timewise-CV, ACC, and POPT; MULTI performs significantly better than all the baselines
[28]	2007	Classification	To found pre- and post-release defects for every package and file	LR	—	Eclipse 2.0, 2.1, 3.0	PR, recall, ACC
[8]	2014	Clustering	Cluster ensemble with PSO for clustering the software modules (fault-prone or not fault-prone)	PSO clustering algorithm	KM-E, KM-M, PSO-E, PSO-M and EM	Nasa MDP, PROMISE
[29]	2015	Classification	Defect identification by applying DM algorithms	NB, J48, MLP	—	PROMISE, NASA MDP dataset: CM1, JM1, KC1, KC3, MC1, MC2, MW1, PC1, PC2, PC3	10-fold CV, ACC, PR,; FMLP is the best
[30]	2015	Classification	To show the attributes that predict the defective state of software modules	NB, NN, association rules, DT	Weighted voting rule of the four algorithms	NASA datasets: CM1, JM1, KC1, KC2, PC1	PR, recall, ACC, F-score; NB > NN > DT
[31]	2016	Classification	Authors proposed a model that finds fault-proneness	NB, LR, LivSVM, MLP, SGD, SMO, VP, LR Logit Boost, Decision Stamp, RT, REP Tree	RF	Camel1.6, Tomcat 6.0, Ant 1.7, jEdit4.3, Ivy 2.0, arc, e-learning, berek, forrest 0.8, zuzel, Intercafe, and Nieruchomosci	10-fold CV, AUC; AUC = 0.661
[32]	2016	Classification	GA to select suitable source code metrics	LR, ELM, SVML, SVMR, SVMP	—	30 open-source software projects from PROMISE repository from DS1 to DS30	5-fold CV, F-score, ACC, pairwise t-test
[33]	2016	—	Weighted least-squares twin support vector machine (WLSTSVM) to find misclassification cost of DP	SVM, NB, RF, LR, KNN, BN, cost-sensitive neural network	—	PROMISE repository: CM1, KC1, PC1, PC3, PC4, MC2, KC2, KC3	10-fold CV, PR, recall, F-score, G-mean; Wilcoxon signed rank test
[34]	2016	—	A multi-objective naive Bayes learning techniques MONB, MOBNN	NB, LR, DT, MODT, MOLR, MONB	—	Jureczko datasets obtained from PROMISE repository	AUC, Wilcoxon rank test; CP MO NB (0.72) produces the highest value
[35]	2016	Classification	A software defect prediction model to find faulty components of a software	Hybrid filter approaches; FISHER, MR, ANNIGMA.	—	KC1, KC2, JM1, PC1, PC2, PC3, and PC4 datasets	ACC, ent filters, ACC 90%
[36]	2017	Classification	Propose an hybrid method called TSC-RUS + S	A random undersampling based on two-step cluster (TSC)	Stacking: DT, LR, kNN, NB	NASA MDP: i.e., CM1, KC1, KC3, MC2, MW1, PC1, PC2, PC3, PC4	10-fold CV, AUC, (TSC-RUS + S) is the best
[37]	2017	Classification	Analyze five popular ML algorithms for software defect prediction	ANN, PSO, DT, NB, LC	—	Nasa and PROMISE datasets: CM1, JM1, KC1, KC2, PC1, KC1-LC	10-fold CV; ANN < DT
[38]	2018	Classification	Three well-known ML techniques are compared.	NB, DT, ANN	—	Three different datasets; DS1, DS2, DS3	ACC, PR, recall, F, ROC; ACC 97%; DT > ANN > NB
[10]	2018	Classification	ML algorithms are compared with CODEP	LR, BN, RBF, MLP, alternating decision tree (ADTree), and DT	Max, CODEP, Bagging J48, Bagging NB, Boosting J48, Boosting NB, RF	PROMISE: Ant, Camel, ivy, Jedit, Log4j, Lucene, Poi, Prop, Tomcat, Xalan	F-score, PR, AUC ROC; Max performs better than CODEP

Table 1.

Data mining and machine learning studies on the subject “defect prediction.”

3.1.1 Defect prediction using ensemble learning techniques

Ensemble learning combines several base learning models to obtain better performance than individual models. These base learners can be acquired with:

1. Different learning algorithms

2. Different parameters of the same algorithm

3. Different training sets

The commonly used ensemble techniques bagging, boosting, and stacking are shown in Figure 3 and briefly explained in this part. Bagging (which stands for bootstrap aggregating) is a kind of parallel ensemble. In this method, each model is built independently, and multiple training datasets are generated from the original dataset through random selection of different feature subsets; thus, it aims to decrease variance. It combines the outputs of each ensemble member by a voting mechanism. Boosting can be described as sequential ensemble. First, the same weights are assigned to data instances; after training, the weight of wrong predictions is increased, and this process is repeated as the ensemble size. Finally, it uses a weighted voting scheme, and in this way, it aims to decrease bias. Stacking is a technique that uses predictions from multiple models via a meta-classifier.

Figure 3.

Common ensemble learning methods: (a) Bagging, (b) boosting, (c) stacking.

Some software defect prediction studies have compared ensemble techniques to determine the best performing one [10, 18, 21, 39, 40]. In a study conducted by Wang et al. [18], different ensemble techniques such as bagging, boosting, random tree, random forest, random subspace, stacking, and voting were compared to each other and a single classifier (NB). According to the results, voting and random forest clearly exhibited better performance than others. In a different study [39], ensemble methods were compared with more than one base learner (NB, BN, SMO, PART, J48, RF, random tree, IB1, VFI, DT, NB tree). For boosted SMO, bagging J48, and boosting and bagging RT, performance of base classifiers was lower than that of ensemble learner classifiers.

In study [21], a new method was proposed of mixing feature selection and ensemble learning for defect classification. Results showed that random forests and the proposed algorithm are not affected by poor features, and the proposed algorithm outperforms existing single and ensemble classifiers in terms of classification performance. Another comparative study [10] used seven composite algorithms (Ave, Max, Bagging C4.5, bagging naive Bayes (NB), Boosting J48, Boosting naive Bayes, and RF) and one composite state-of-the art study for cross-project defect prediction. The Max algorithm yielded the best results regarding F-score in terms of classification performance.

Bowes et al. [40] compared RF, NB, Rpart, and SVM algorithms to determine whether these classifiers obtained the same results. The results demonstrated that a unique subset of defects can be discovered by specific classifiers. However, whereas some classifiers are steady in the predictions they make, other classifiers change in their predictions. As a result, ensembles with decision-making without majority voting can perform best.

One of the main problems of SDP is the imbalance between the defect and non-defect classes of the dataset. Generally, the number of defected instances is greater than the number of non-defected instances in the collected data. This situation causes the machine learning algorithms to perform poorly. Wang and Yao [19] compared five class-imbalanced learning methods (RUS, RUS-bal, THM, BNC, SMB) and NB and RF algorithms and proposed the dynamic version of AdaBoost.NC. They utilized balance, G-mean, and AUC measures for comparison. Results showed that AdaBoost.NC and naive Bayes are better than the other seven algorithms in terms of evaluation measures. Dynamic AdaBoost.NC showed better defect detection rate and overall performance than the original AdaBoost.NC. To handle the class imbalance problem, studies [20] have compared different methods (sampling, cost sensitive, hybrid, and ensemble) by taking into account evaluation metrics such as MCC and receiver operating characteristic (ROC).

As shown in Table 1, the most common datasets used in the defect prediction studies [17, 18, 19, 39] are the NASA MDP dataset and PROMISE repository datasets. In addition, some studies utilized open-source projects such as Bugzilla Columba and Eclipse JDT [26, 27], and other studies used Android application data [22, 23].

3.1.2 Defect prediction studies without ensemble learning

Although use of ensemble learning techniques has dramatically increased recently, studies that do not use ensemble learning are still conducted and successful. For example, in study [32], prediction models were created using source code metrics as in ensemble studies but by using different feature selection techniques such as genetic algorithm (GA).

To overcome the class imbalance problem, Tomar and Agarwal [33] proposed a prediction system that assigns lower cost to non-defective data samples and higher cost to defective samples to balance data distribution. In the absence of enough data within a project, required data can be obtained from cross projects; however, in this case, this situation may cause class imbalance. To solve this problem, Ryu and Baik [34] proposed multi-objective naïve Bayes learning for cross-project environments. To obtain significant software metrics on cloud computing environments, Ali et al. used a combination of filter and wrapper approaches [35]. They compared different machine learning algorithms such as NB, DT, and MLP [29, 37, 38, 41].

3.2 Data mining in effort estimation

Software effort estimation (SEE) is critical for a company because hiring more employees than required will cause loss of revenue, while hiring fewer employees than necessary will result in delays in software project delivery. The estimation analysis helps to predict the amount of effort (in person hours) needed to develop a software product. Basic steps of software estimation can be itemized as follows:

• Determine project objectives and requirements.

• Design the activities.

• Estimate product size and complexity.

• Compare and repeat estimates.

SEE contains requirements and testing besides predicting effort estimation [42]. Many research and review studies have been conducted in the field of SEE. Recently, a survey [43] analyzed effort estimation studies that concentrated on ML techniques and compared them with studies focused on non-ML techniques. According to the survey, case-based reasoning (CBR) and artificial neural network (ANN) were the most widely used techniques. In 2014, Dave and Dutta [44] examined existing studies that focus only on neural network.

The current effort estimation studies using DM and ML techniques are available in Table 2. This table summarizes the prominent studies in terms of aspects such as year, data mining task, aim, datasets, and metrics. Table 2 indicates that neural network is the most widely used technique for the effort estimation task.

Ref.	Year	Task	Objective	Algorithms	Ensemble learning	Dataset	Evaluation metrics and results
[45]	2008	Regression	Ensemble of neural networks with associative memory (ENNA)	NN, MLP, KNN	Bagging	NASA, NASA 93, USC, SDR, Desharnais	MMRE, MdMRE and PRED(L); For ENNA PRED(25) = 36.4; For neural network PRED(25) = 8
[46]	2009	Regression	Authors proposed the ensemble of neural networks with associative memory (ENNA)	NN, MLP, KNN	Bagging	NASA, NASA 93, USC, SDR, Desharnais	Random subsampling, t-test; MMRE, MdMRE, and PRED(L); ENNA is the best
[47]	2010	Regression	To show the effectiveness of SVR for SEE	SVR, RBF	—	Tukutuku	LOOCV, MMRE, Pred(25), MEMRE, MdEMRE; SVR outperforms others
[48]	2011	Regression	To evaluate whether readily available ensemble methods enhance SEE	MLP, RBF, RT	Bagging	5 datasets from PROMISE: cocomo81, nasa93, nasa, sdr, and Desharnais; 8 datasets from ISBSG repository	MMRE, MdMRE, PRED(25); RTs and Bagging with MLPs perform similarly
[49]	2012	Regression	To show the measures behave in SEE and to create good ensembles	MLP, RBF, REPTree,	Bagging	cocomo81, nasa93, nasa, cocomo2, desharnais, ISBSG repository	MMRE, PRED(25), LSD, MdMRE, MAE, MdAE; Pareto ensemble for all measures, except LSD.
[50]	2012	Regression	To use cross-company models to create diverse ensembles able to dynamically adapt to changes	WC RTs, CC-DWM	WC-DWM	3 datasets from ISBSG repository (ISBSG2000, ISBSG2001, ISBSG) 2 datasets from PROMISE (CocNasaCoc81 and CocNasaCoc81Nasa93)	MAE, Friedman test; Only DCL could improve upon RT; CC data potentially beneficial for improving SEE
[51]	2012	Regression	To generate estimates from ensembles of multiple prediction methods	CART, NN, LR, PCR, PLSR, SWR, ABE0-1NN, ABE0-5NN	Combining top M solo methods	PROMISE	MAR, MMRE, MdMRE, MMER, MBRE, MIBRE.; Combinations perform better than 83%
[52]	2012	Classification/regression	DM techniques to estimate software effort.	M5, CART, LR, MARS, MLPNN, RBFNN, SVM	—	Coc81, CSC, Desharnais, Cocnasa, Maxwell, USP05	MdMRE, Pred(25), Friedman test; Log + OLS > LMS, BC + OLS, MARS, LS-SVM
[53]	2013	Clustering/classification	Estimation of software development effort	NN, ABE, C-means	—	Maxwell	3-fold CV and LOOCV, RE, MRE, MMRE, PRED
[54]	2014	Regression	ANNs are examined using COCOMO model	MLP, RBFNN, SVM, PSO-SVM Extreme learning Machines	—	COCOMO II Data	MMRE, PRED; PSO-SVM is the best
[55]	2014	—	A hybrid model based on GA And ACO for optimization	GA, ACO	—	NASA datasets	MMRE, the proposed method is the best
[56]	2015	Regression	To display the effect of data preprocessing techniques on ML methods in SEE	CBR, ANN, CART; Preprocessing rech: MDT, LD, MI, FS, CS, FSS, BSS	—	ISBSG, Desharnais, Kitchenham, USPFT	CV, MBRE, PRED (0.25), MdBRE
[57]	2016	Regression	Four neural network models are compared with each other.	MLP, RBFNN, GRNN, CCNN	—	ISBSG repository	10-fold CV, MAR; The CCNN outperforms the other three models
[58]	2016	Regression	To propose a model based on Bayesian network	GA and PSO	—	COCOMO NASA Dataset	DIR, DRM; The proposed model is best
[59]	2016	Classification/regression	A hybrid model using SVM and RBNN compared against previous models	SVM, RBNN	—	Dataset1 = 45 industrial projects; Dataset2 = 65 educational projects	LOOCV, MAE, MBRE, MIBRE, SA; The proposed approach is the best
[60]	2017	Classification	To estimate software effort by using ML techniques	SVM, KNN	Boosting: kNN and SVM	Desharnais, Maxwell	LOOCV, k-fold CV; ACC = 91.35% for Desharnais; ACC = 85.48% for Maxwell

Table 2.

Data mining and machine learning studies on the subject “effort estimation.”

Several studies have compared ensemble learning methods with single learning algorithms [45, 46, 48, 49, 51, 60] and examined them on cross-company (CC) and within-company (WC) datasets [50]. The authors observed that ensemble methods obtained by a proper combination of estimation methods achieved better results than single methods. Various ML techniques such as neural network, support vector machine (SVM), and k-nearest neighbor are commonly used as base classifiers for ensemble methods such as bagging and boosting in software effort estimation. Moreover, their results indicate that CC data can increase performance over WC data for estimation techniques [50].

In addition to the abovementioned studies, researchers have conducted studies without using ensemble techniques. The general approach is to investigate which DM technique has the best effect on performance in software effort estimation. For instance, Subitsha and Rajan [54] compared five different algorithms—MLP, RBFNN, SVM, ELM, and PSO-SVM—and Nassif et al. [57] investigated four neural network algorithms—MLP, RBFNN, GRNN, and CCNN. Although neural networks are widely used in this field, missing values and outliers frequently encountered in the training set adversely affect neural network results and cause inaccurate estimations. To overcome this problem, Khatibi et al. [53] split software projects into several groups based on their similarities. In their studies, the C-means clustering algorithm was used to determine the most similar projects and to decrease the impact of unrelated projects, and then analogy-based estimation (ABE) and NN were applied. Another clustering study by Azzeh and Nassif [59] combined SVM and bisecting k-medoids clustering algorithms; an estimation model was then built using RBFNN. The proposed method was trained on historical use case points (UCP).

Zare et al. [58] and Maleki et al. [55] utilized optimization methods for accurate cost estimation. In the former study, a model was proposed based on Bayesian network with genetic algorithm and particle swarm optimization (PSO). The latter study used GA to optimize the effective factors’ weight, and then trained by ant colony optimization (ACO). Besides conventional effort estimation studies, researchers have utilized machine learning techniques for web applications. Since web-based software projects are different from traditional projects, the effort estimation process for these studies is more complex.

It is observed that PRED(25) and MMRE are the most popular evaluation metrics in effort estimation. MMRE stands for the mean magnitude relative error, and PRED(25) measures prediction accuracy and provides a percentage of predictions within 25% of actual values.

3.3 Data mining in vulnerability analysis

Vulnerability analysis is becoming the focal point of system security to prevent weaknesses in the software system that can be exploited by an attacker. Description of software vulnerability is given in many different resources in different ways [61]. The most popular and widely utilized definition appears in the Common Vulnerabilities and Exposures (CVE) 2017 report as follows:

Vulnerability is a weakness in the computational logic found in software and some hardware components that, when exploited, results in a negative impact to confidentiality, integrity or availability.

Vulnerability analysis may require many different operations to identify defects and vulnerabilities in a software system. Vulnerabilities, which are a special kind of defect, are more critical than other defects because attackers exploit system vulnerabilities to perform unauthorized actions. A defect is a normal problem that can be encountered frequently in the system, easily found by users or developers and fixed promptly, whereas vulnerabilities are subtle mistakes in large codes [62, 63]. Wijayasekara et al. claim that some bugs have been identified as vulnerabilities after being publicly announced in bug databases [64]. These bugs are called “hidden impact vulnerabilities” or “hidden impact bugs.” Therefore, the authors proposed a hidden impact vulnerability identification methodology that utilizes text mining techniques to determine which bugs in bug databases are vulnerabilities. According to the proposed method, a bug report was taken as input, and it produces feature vector after applying text mining. Then, classifier was applied and revealed whether it is a bug or a vulnerability. The results given in [64] demonstrate that a large proportion of discovered vulnerabilities were first described as hidden impact bugs in public bug databases. While bug reports were taken as input in that study, in many other studies, source code is taken as input. Text mining is a highly preferred technique for obtaining features directly from source codes as in the studies [65, 66, 67, 68, 69]. Several studies [63, 70] have compared text mining-based models and software metrics-based models.

In the security area of software systems, several studies have been conducted related to DM and ML. Some of these studies are compared in Table 3, which shows the data mining task and explanation of the studies, the year they were performed, the algorithms that were used, the type of vulnerability analysis, evaluation metrics, and results. In this table, the best performing algorithms according to the evaluation criteria are shown in bold.

Ref.	Year	Task	Objective	Algorithms	Type	Dataset description	Evaluation metrics and results
[71]	2011	Clustering	Obtaining software vulnerabilities based on RDBC	RDBC	Static	Database is built by RD-Entropy	FNR, FPR
[42]	2011	Classification/regression	To predict the time to next vulnerability	LR, LMS, MLP, RBF, SMO	Static	NVD, CPE, CVSS	CC, RMSE, RRSE
[65]	2012	Text mining	Analysis of source code as text	RBF, SVM	Static	K9 email client for the Android platform	ACC, PR, recall; ACC = 0.87, PR = 0.85, recall = 0.88
[64]	2012	Classification/text mining	To identify vulnerabilities in bug databases	—	Static	Linux kernel MITRE CVE and MySQL bug databases	BDR, TPR, FPR; 32% (Linux) and 62% (MySQL) of vulnerabilities
[72]	2014	Classification/regression	Combine taint analysis and data mining to obtain vulnerabilities	ID3, C4.5/J48, RF, RT, KNN, NB, Bayes Net, MLP, SVM, LR	Hybrid	A version of WAP to collect the data	10-fold CV, TPD, ACC, PR, KAPPA; ACC = 90.8%, PR = 92%, KAPPA = 81%
[73]	2014	Clustering	Identify vulnerabilities from source codes using CPG	—	Static	Neo4J and InfiniteGraph databases	—
[63]	2014	Classification	Comparison of software metrics with text mining	RF	Static	Vulnerabilities from open-source web apps (Drupal, Moodle, PHPMyAdmin)	3-fold CV, recall, IR, PR, FPR, ACC.; Text mining provides benefits overall
[69]	2014	Classification	To create model in the form of a binary classifier using text mining	NB, RF	Static	Applications from the F-Droid repository and Android	10-fold CV, PR, recall; PR and recall ≥ 80%
[74]	2015	Classification	A new approach (VCCFinder) to obtain potentially dangerous codes	SVM-based detection model	—	The database contains 66 GitHub projects	k-fold CV, false alarms <99% at the same level of recall
[70]	2015	Ranking/classification	Comparison of text mining and software metrics models	RF	—	Vulnerabilities from open-source web apps (Drupal, Moodle, PHPMyAdmin)	10-fold CV; Metrics: ER-BCE, ERBPP, ER-AVG
[75]	2015	Clustering	Search patterns for taint-style vulnerabilities in C code	Hierarchical clustering (complete-linkage)	Static	5 open-source projects: Linux, OpenSSL, Pidgin, VLC, Poppler (Xpdf)	Correct source, correct sanitization, number of traversals, generation time, execution time, reduction, amount of code review <95%
[76]	2016	Classification	Static and dynamic features for classification	LR, MLP, RF	Hybrid	Dataset was created by analyzing 1039 test cases from the Debian Bug Tracker	FPR, FNR; Detect 55% of vulnerable programs
[77]	2017	Classification	1. Employ a deep neural network; 2. Combine N-gram analysis and feature selection	Deep neural network	—	Feature extraction from 4 applications (BoardGameGeek, Connectbot, CoolReader, AnkiDroid)	10 times using 5-fold CV; ACC = 92.87%, PR = 94.71%, recall = 90.17%
[67]	2017	Text mining	To analyze characteristics of software vulnerability from source files	—	—	CVE, CWE, NVD databases	PR = 70%, recall = 60%
[68]	2017	Text mining	Deep learning (LSTM) is used to learn semantic and syntactic features in code	RNN, LSTM, DBN	—	Experiments on 18 Java applications from the Android OS platform	10-fold CV, PR, recall, and F-score; Deep Belief Network; PR, recall, and F-score > 80%
[66]	2018	Classification	Identify bugs by extracting text features from C source code	NB, KNN, K-means, NN, SVM, DT, RF	Static	NVD, Cat, Cp, Du, Echo, Head, Kill, Mkdir, Nl, Paste, Rm, Seq, Shuf, Sleep, Sort, Tail, Touch, Tr, Uniq, Wc, Whoami	5-fold CV ACC, TP, TN; ACC = 74%
[78]	2018	Regression	A deep learning-based vulnerability detection system (VulDeePecker)	BLSTM NN	Static	NIST: NVD and SAR project	10-fold CV, PR, recall, F-score; F-score = 80.8%
[79]	2018	Classification	A mapping between existing requirements and vulnerabilities	LR, SVM, NB	—	Data is gathered from Apache Tomcat, CVE, requirements from Bugzilla, and source code is collected from Github	PR, recall, F-score; LSI > SVM

Table 3.

Data mining and machine learning studies on the subject “vulnerability analysis.”

Vulnerability analysis can be categorized into three types: static vulnerability analysis, dynamic vulnerability analysis, and hybrid analysis [61, 80]. Many studies have applied the static analysis approach, which detects vulnerabilities from source code without executing software, since it is cost-effective. Few studies have performed the dynamic analysis approach, in which one must execute software and check program behavior. The hybrid analysis approach [72, 76] combines these two approaches.

As revealed in Table 3, in addition to classification and text mining, clustering techniques are also frequently seen in software vulnerability analysis studies. To detect vulnerabilities in an unknown software data repository, entropy-based density clustering [71] and complete-linkage clustering [75] were proposed. Yamaguchi et al. [73] introduced a model to represent a large number of source codes as a graph called control flow graph (CPG), a combination of abstract syntax tree, CFG, and program dependency graph (PDG). This model enabled the discovery of previously unknown (zero-day) vulnerabilities.

To learn the time to next vulnerability, a prediction model was proposed in the study [42]. The result could be a number that refers to days or a bin representing values in a range. The authors used regression and classification techniques for the former and latter cases, respectively.

In vulnerability studies, issue tracking systems like Bugzilla, code repositories like Github, and vulnerability databases such as NVD, CVE, and CWE have been utilized [79]. In addition to these datasets, some studies have used Android [65, 68, 69] or web [63, 70, 72] (PHP source code) datasets. In recent years, researchers have concentrated on deep learning for building binary classifiers [77], obtaining vulnerability patterns [78], and learning long-term dependencies in sequential data [68] and features directly from the source code [81].

Li et al. [78] note two difficulties of vulnerability studies: demanding, intense manual labor and high false-negative rates. Thus, the widely used evaluation metrics in vulnerability analysis are false-positive rate and false-negative rate.

3.4 Data mining in design pattern mining

During the past years, software developers have used design patterns to create complex software systems. Thus, researchers have investigated the field of design patterns in many ways [82, 83]. Fowler defines a pattern as follows:

“A pattern is an idea that has been useful in one practical context and will probably be useful in others.” [84]

Patterns display relationships and interactions between classes or objects. Well-designed object-oriented systems have various design patterns integrated into them. Design patterns can be highly useful for developers when they are used in the right manner and place. Thus, developers avoid recreating methods previously refined by others. The pattern approach was initially presented in 1994 by four authors—namely, Erich Gama, Richard Helm, Ralph Johnson, and John Vlissides—called the Gang of Four (GOF) in 1994 [85]. According to the authors, there are three types of design patterns:

1. Creational patterns provide an object creation mechanism to create the necessary objects based on predetermined conditions. They allow the system to call appropriate object and add flexibility to the system when objects are created. Some creational design patterns are factory method, abstract factory, builder, and singleton.

2. Structural patterns focus on the composition of classes and objects to allow the establishment of larger software groups. Some of the structural design patterns are adapter, bridge, composite, and decorator.

3. Behavioral patterns determine common communication patterns between objects and how multiple classes behave when performing a task. Some behavioral design patterns are command, interpreter, iterator, observer, and visitor.

Many design pattern studies exist in the literature. Table 4 shows some design pattern mining studies related to machine learning and data mining. This table contains the aim of the study, mining task, year, and design patterns selected by the study, input data, dataset, and results of the studies.

Ref.	Year	Task	Objective	Algorithms	EL	Selected design patterns	Input data	Dataset	Evaluation metrics and results
[86]	2012	Text classification	Two-phase method: 1—text classification to; 2—learning design patterns	NB, KNN, DT, SVM	—	46 security patterns, 34 Douglass patterns, 23 GoF patterns	Documents	Security, Douglass, GoF	PR, recall, EWM; PR = 0.62, recall = 0.75
[87]	2013	Regression	An approach is to find a valid instance of a DP or not	ANN	—	Adapter, command, composite, decorator, observer, and proxy	Set of candidate classes	JHotDraw 5.1 open-source application	10 fold CV, PR, recall
[88]	2014	Graph mining	Sub-graph mining-based approach	CloseGraph	—	—	Java source code	Open-source project:YARI, Zest, JUnit, JFreeChart, ArgoUML	No any empirical comparison
[89]	2015	Classification/clustering	MARPLE-DPD is developed to classify instances whether it is a bad or good instance	SVM, DT, RF, K-means, ZeroR, OneR, NB, JRip, CLOPE.	—	Classification for singleton and adapter; Classification and clustering for composite, decorator, and factory method	—	10 open-source software systems; DPExample, QuickUML 2001, Lexi v0.1.1 alpha, JRefactory v2.6.24, Netbeans v1.0.x, JUnit v3.7, JHotDraw v5.1, MapperXML v1.9.7, Nutch v0.4, PMD v1.8	10-fold CV, ACC, F-score, AUC; ACC > =85%
[90]	2015	Regression	A new method (SVM-PHGS) is proposed	Simple Logistic, C4.5, KNN, SVM, SVM-PHGS	—	Adapter, builder, composite, factory method, iterator, observer	Source code	P-mart repository	PR, recall, F-score, FP; PR = 0.81, recall =0.81, F-score = 0.81, FP = 0.038
[91]	2016	Classification	Design pattern recognition using ML algorithms.	LRNN, DT	—	Abstract factory, adapter patterns	Source code	Dataset with 67 OO metrics, extracted by JBuilder tool	5-fold CV, ACC, PR, recall, F-score; ACC = 100% by LRNN
[92]	2016	Classification	Three aspects: design patterns, software metrics, and supervised learning methods	Layer Recurrent Neural Network (LRNN)	RF	Abstract factory, adapter, bridge, singleton,; and template method	Source code	Dataset with 67 OO metrics, extracted by JBuilder tool	PR, recall, F-score; F-score = 100% by LRNN and RF; ACC = 100% by RF
[93]	2017	Classification	1. Creation of metrics-oriented dataset; 2. Detection of software design patterns	ANN, SVM	RF	Abstract factory, adapter, bridge, composite, and Template	Source code	Metrics extracted from source codes (JHotDraw, QuickUML, and Junit)	5-fold and 10-fold CV, PR, recall, F-score; ANN, SVM, and RF yielded to 100% PR for JHotDraw
[94]	2017	Classification	Detection of design motifs based on a set of directed semantic graphs	Strong graph simulation, graph matching	—	All three groups: creational, structural, behavioral	UML class diagrams	—	PR, recall; High accuracy by the proposed method
[95]	2017	Text categorization	Selection of more appropriate design patterns	Fuzzy c-means	Ensemble-IG	Various design patterns	Problem definitions of design patterns	DP, GoF, Douglass, Security	F-score
[96]	2018	Classification	Finding design pattern and smell pairs which coexist in the code	J48	—	Used patterns: adapter, bridge, Template, singleton	Source code	Eclipse plugin Web of Patterns; The tool selected for code smell detection is iPlasma	PR, recall, F-score, PRC, ROC; Singleton pattern shows no presence of bad smells

Table 4.

Data mining and machine learning studies on the subject “design pattern mining.”

In design pattern mining, detecting the design pattern is a frequent study objective. To do so, studies have used machine learning algorithms [87, 89, 90, 91], ensemble learning [95], deep learning [97], graph theory [94], and text mining [86, 95].

In study [91], the training dataset consists of 67 object-oriented (OO) metrics extracted by using the JBuilder tool. The authors used LRNN and decision tree techniques for pattern detection. Alhusain et al. [87] generated training datasets from existing pattern detection tools. The ANN algorithm was selected for pattern instances. Chihada et al. [90] created training data from pattern instances using 45 OO metrics. The authors utilized SVM for classifying patterns accurately. Another metrics-oriented dataset was developed by Dwivedi et al. [93]. To evaluate the results, the authors benefited from three open-source software systems (JHotDraw, QuickUML, and JUnit) and applied three classifiers, SVM, ANN, and RF. The advantage of using random forest is that it does not require linear features and can manage high-dimensional spaces.

To evaluate methods and to find patterns, open-source software projects such as JHotDraw, Junit, and MapperXML have been generally preferred by researchers. For example, Zanoni et al. [89] developed a tool called MARPLE-DPD by combining graph matching and machine learning techniques. Then, to obtain five design patterns, instances were collected from 10 open-source software projects, as shown in Table 4.

Design patterns and code smells are related issues: Code smell refers to symptoms in code, and if there are code smells in a software, its design pattern is not well constructed. Therefore, Kaur and Singh [96] checked whether design pattern and smell pairs appear together in a code by using J48 Decision Tree. Their obtained results showed that the singleton pattern had no presence of bad smells.

According to the studies summarized in the table, the most frequently used patterns are abstract factory and adapter. It has recently been observed that studies on ensemble learning in this field are increasing.

3.5 Data mining in refactoring

One of the SE tasks most often used to improve the quality of a software system is refactoring, which Martin Fowler has described as “a technique for restructuring an existing body of code, altering its internal structure without changing its external behavior” [98]. It improves readability and maintainability of the source code and decreases complexity of a software system. Some of the refactoring types are: Add Parameter, Replace Parameter, Extract method, and Inline method [99].

Code smell and refactoring are closely related to each other: Code smells represent problems due to bad design and can be fixed during refactoring. The main challenge is to obtain which part of the code needs refactoring.

Some of data mining studies related to software refactoring are presented in Table 5. Some studies focus on historical data to predict refactoring [100] or to obtain both refactoring and software defects [101] using different data mining algorithms such as LMT, Rip, and J48. Results suggest that when refactoring increases, the number of software defects decreases, and thus refactoring has a positive effect on software quality.

Ref.	Year	Task	Objective	Algorithms	EL	Dataset	Evaluation metrics and results
[100]	2007	Regression	Stages: (1) data understanding, (2) preprocessing, (3) ML, (4) post-processing, (5) analysis of the results	J48, LMT, Rip, NNge	—	ArgoUML, Spring Framework	10-fold CV, PR, recall, F-score; PR and recall are 0.8 for ArgoUML
[101]	2008	Classification	Finding the relationship between refactoring and defects	C4.5, LMT, Rip, NNge	—	ArgoUML, JBoss Cache, Liferay Portal, Spring Framework, XDoclet	PR, recall, F-score
[102]	2014	Regression	Propose GA-based learning for software refactoring based on ANN	GA, ANN	—	Xerces-J, JFreeChart, GanttProject, AntApache, JHotDraw, and Rhino.	Wilcoxon test with a 99% confidence level (α = 0.01)
[103]	2015	Regression	Removing defects with time series in a multi-objective approach	Multi-objective algorithm, based on NSGA-II, ARIMA		FindBugs, JFreeChart, Hibernate, Pixelitor, and JDI-Ford	Wilcoxon rank sum test with a 99% confidence level (α < 1%)
[104]	2016	Web mining/clustering	Unsupervised learning approach to detect refactoring opportunities in service-oriented applications	PAM, K-means, COBWEB, X-Means	—	Two datasets of WSDL documents	COBWEB and K-means max. 83.33% and 0%, inter-cluster; COBWEB and K-means min. 33.33% and 66.66% intra-cluster
[105]	2017	Clustering	A novel algorithm (HASP) for software refactoring at the package level	Hierarchical clustering algorithm	—	Three open-source case studies	Modularization Quality and Evaluation Metric Function
[99]	2017	Classification	A technique to predict refactoring at class level	PCA, SMOTE; LS-SVM, RBF	—	From tera- PROMISE Repository seven open-source software systems	10-fold CV, AUC, and ROC curves; RBF kernel outperforms linear and polynomial kernel; The mean value of AUC for LS-SVM RBF kernel is 0.96
[106]	2017	Classification	Exploring the impact of clone refactoring (CR) on the test code size	LR, KNN, NB	RF	data collected from an open-source Java software system (ANT)	PR, recall, accuracy, F-score; kNN and RF outperform NB; ACC (fitting (98%), LOOCV (95%), and 10 FCV (95%))
[107]	2017	—	Finding refactoring opportunities in source code	J48, BayesNet, SVM, LR	RF	Ant, ArgoUML, jEdit, jFreeChart, Mylyn	10-fold CV, PR, recall; 86–97% PR and 71–98% recall for proposed tech
[108]	2018	Classification	A learning-based approach (CREC) to extract refactored and non-refactored clone groups from repositories	C4.5, SMO, NB.	RF, Adaboost	Axis2, Eclipse.jdt.core, Elastic Search, JFreeChart, JRuby, and Lucene	PR, recall, F-score; F-score = 83% in the within-project; F-score = 76% in the cross-project
[109]	2018	Clustering	Combination of the use of multi-objective and unsupervised learning to decrease developer’s effort	GMM, EM	—	ArgoUML, JHotDraw, GanttProject, UTest, Apache Ant, Azureus	One-way ANOVA with a 95% confidence level (α = 5%)

Table 5.

Data mining and machine learning studies on the subject “refactoring.”

While automated refactoring does not always give the desired result, manual refactoring is time-consuming. Therefore, one study [109] proposed a clustering-based recommendation tool by combining multi-objective search and unsupervised learning algorithm to reduce the number of refactoring options. At the same time, the number of refactoring that should be selected is decreasing with the help of the developer’s feedback.

4. Discussion

Since many SE studies that apply data mining approaches exist in the literature, this article presents only a few of them. However, Figure 4 shows the current number of papers obtained from the Scopus search engine for each year from 2010 to 2019 by using queries in the title/abstract/keywords field. We extracted publications in 2020 since this year has not completed yet. Queries included (“data mining” OR “machine learning”) with (“defect prediction” OR “defect detection” OR “bug prediction” OR “bug detection”) for defect prediction, (“effort estimation” OR “effort prediction” OR “cost estimation”) for effort estimation, (“vulnerab*” AND “software” OR “vulnerability analysis”) for vulnerability analysis, and (“software” AND “refactoring”) for refactoring. As seen in the figure, the number of studies using data mining in SE tasks, especially defect prediction and vulnerability analysis, has increased rapidly. The most stable area in the studies is design pattern mining.

Figure 4.

Number of publications of data mining studies for SE tasks from Scopus search by their years.

Figure 5 shows the publications studied in classification, clustering, text mining, and association rule mining as a percentage of the total number of papers obtained by a Scopus query for each SE task. For example, in defect prediction, the number of studies is 339 in the field of classification, 64 in clustering, 8 in text mining, and 25 in the field of association rule mining. As can be seen from the pie charts, while clustering is a popular DM technique in refactoring, no study related to text mining is found in this field. In other SE tasks, the preferred technique is classification, and the second is clustering.

Figure 5.

Number of publications of data mining studies for SE tasks from Scopus search by their topics.

Defect prediction generally compares learning algorithms in terms of whether they find defects correctly using classification algorithms. Besides this approach, in some studies, clustering algorithms were used to select futures [110] or to compare supervised and unsupervised methods [27]. In the text mining area, to extract features from scripts, TF-IDF techniques were generally used [111, 112]. Although many different algorithms have been used in defect prediction, the most popular ones are NB, MLP, and RBF.

Figure 6 shows the number of document types (conference paper, book chapter, article, book) published between the years of 2010 and 2019. It is clearly seen that conference papers and articles are the most preferred research study type. It is clearly seen that there is no review article about data mining studies in design pattern mining.

Figure 6.

The number of publications in terms of document type between 2010 and 2019.

Table 6 shows popular repositories that contain various datasets and their descriptions, which tasks they are used for, and hyperlinks to download. For example, the PMART repository includes source files of java projects, and the PROMISE repository has different datasets with software metrics such as cyclomatic complexity, design complexity, and lines of code. Since these repositories contain many datasets, no detailed information about them has been provided in this article.

Repository	Topic	Description	Web link
Nasa MDP	Defect Pred.	NASA’s Metrics Data Program	https://github.com/opensciences/opensciences.github.io/tree/master/repo/defect/mccabehalsted/_posts
Android Git	Defect Pred.	Android version bug reports	https://android.googlesource.com/
PROMISE	Defect Pred. Effort Est.	It includes 20 datasets for defect prediction and cost estimation	http://promise.site.uottawa.ca/SERepository/datasets-page.html
Software Defect Pred. Data	Defect Pred.	It includes software metrics, # of defects, etc. Eclipse JDT: Eclipse PDE:	http://www.seiplab.riteh.uniri.hr/?page_id=834&lang=en
PMART	Design pattern mining	It has 22 patterns 9 Projects, 139 ins. Format: XML Manually detected and validated	http://www.ptidej.net/tools/designpatterns/

Table 6.

Description of popular repositories used in studies.

Refactoring can be applied at different levels; study [105] predicted refactoring at package level using hierarchical clustering, and another study [99] applied class-level refactoring using LS-SVM as learning algorithm, SMOTE for handling refactoring, and PCA for feature extraction.

5. Conclusion and future work

Data mining techniques have been applied successfully in many different domains. In software engineering, to improve the quality of a product, it is highly critical to find existing deficits such as bugs, defects, code smells, and vulnerabilities in the early phases of SDLC. Therefore, many data mining studies in the past decade have aimed to deal with such problems. The present paper aims to provide information about previous studies in the field of software engineering. This survey shows how classification, clustering, text mining, and association rule mining can be applied in five SE tasks: defect prediction, effort estimation, vulnerability analysis, design pattern mining, and refactoring. It clearly shows that classification is the most used DM technique. Therefore, new studies can focus on clustering on SE tasks.

Abbreviations

LMT	logistic model trees
Rip	repeated incremental pruning
NNge	nearest neighbor generalization
PCA	principal component analysis
PAM	partitioning around medoids
LS-SVM	least-squares support vector machines
MAE	mean absolute error
RBF	radial basis function
RUS	random undersampling
SMO	sequential minimal optimization
GMM	Gaussian mixture model
EM	expectation maximizaion
LR	logistic regression
SMB	SMOTEBoost
RUS-bal	balanced version of random undersampling
THM	threshold-moving
BNC	AdaBoost.NC
RF	random forest
RBF	radial basis function
CC	correlation coefficient
ROC	receiver operating characteristic
BayesNet	Bayesian network
SMOTE	synthetic minority over-sampling technique