Development of a Risk Management Model by the Fuzzy DEMATEL Method in the Evaluation of Authorized Certification Bodies

Abstract

The concept of risk, which has been seen as a danger for many years, has started to be seen as an opportunity today, and with this approach, risks have also begun to be considered as opportunities that can facilitate reaching goals. For these reasons, it is gaining importance day by day for businesses to adopt an effective risk management approach, to identify risks, to determine the degree of importance of risks and to define the actions that can be taken against these risks. Within the scope of this section, the possible risks in the activities of the certification bodies authorized by Vocational Qualification Authority operating in Turkey to conduct assessment and certification in order to determine vocational competencies were determined, and the Fuzzy DEMATEL method, which is one of the Fuzzy Multi-Criteria Decision Making Methods, was used to determine the weights of the risks. A new risk management model has been designed to be used in the evaluation and management of possible risks of organizations by using the weights obtained was designed for the assessment and management of risks.

1. Introduction

Determining all risks that businesses are exposed to, evaluating these risks, and planning preventive actions against these risks play an important role in achieving a sustainable competitive advantage and improving business performance. The modern risk management approach has strategic importance as it manages all risks and adopts a holistic approach in the context of the survival of the businesses [1]. Risk management gains special importance in being prepared for changing business conditions, managing change effectively, and minimizing the negative effects of uncertainties on the objectives of the enterprises while increasing their positive effects.

Vocational Qualifications Authority (VQA) is a public institution with administrative and financial autonomy established in Turkey in order to establish and operate a national qualification system compatible with the European Union. The national occupational standards of the occupations performed in Turkey are prepared by the Vocational Qualifications Authority and also national qualification documents that design the assessment and certification processes to be carried out in order to determine the competent individuals in the relevant occupation based on the occupational standards are developed. Both occupational standards and qualifications documents are developed according to needs of sectors in cooperation with sector institutions. Assessment and certification processes are operated through the certification bodies authorized by VQA in accordance with national qualification documents [2].

Certification bodies authorized by VQA are for-profit organizations, and their financial sustainability is among the authorization conditions. These institutions are required to be accredited according to the “ISO 17024 Conformity assessment – General requirements for bodies operating certification of persons” and meet the authorization conditions determined by VQA [3]. Within the scope of the authorization conditions, these institutions regarding risk management evaluate their assessment activities and define, measure, and evaluate their risks in a way to eliminate uncertainties in the realization of their objectives and in the effective implementation of their procedures and to carry out the necessary preventive actions to prevent these risks [3].

In this context, a new risk management model has been designed and proposed by using the fuzzy DEMATEL method, which is one of the multi-criteria decision-making methods, in order for authorized certification bodies to determine their risks, evaluate and measure risks, and plan the necessary preventive and corrective actions according to the results obtained.

2. Risk and risk management

Although the concept of risk appears in the literature in two ways, traditional and new, in the traditional approach, risk is considered a negative concept and is expressed as a threat, danger, damage, or loss [4]. In the traditional approach, risks are handled independently from each other, focused on specific risks, and activities to reduce risk are continued [5].

In classical risk management, each unit in the business focuses on the risks that are directly affected, and in its area of interest, the focused risks are related to the financial dimension and other risks are not taken into account. Independent determination of the risk in other units, without considering the effects on the entire enterprise, prevents the formation of a risk policy adopted both among the units and throughout the enterprise [6].

In the modern approach, risk management is under the coordination of the senior manager, but under the responsibility of all units and employees, and not only limited to the financial dimension but also considers other risks. In this approach, which integrates with all employees and all processes of the enterprise, risk management exhibits an approach that is compatible with all goals and objectives of the enterprise [1].

Risk, which was seen as a danger for many years, can be seen as an opportunity today. Hazard is only the negative aspect of risk that can lead to undesirable consequences. Opportunity, on the other hand, is the probability of an event that positively affects the realization of business objectives, and it is aimed to create value and protect the value created with opportunities. Our age’s risk management approach adopts a risk management approach that transforms risks into opportunities and thus increases value [7].

With the new approach, risks are evaluated by taking into account the entire enterprise, critical risks are primarily focused, the most appropriate response to risks is determined, and all employees take responsibility [5].

For this reason, while the concept of risk was defined as the negative effect of an unexpected event or uncertainty on targets in the early periods [8], with the new approach adopted in recent years, the negative side of risk was not only focused on but also aspects such as opportunity, profit, and gain, which express the positive aspects, were also discussed [4].

In this framework, the concept of risk is considered as threats, negativities that may prevent the realization of the objectives, or opportunities that may facilitate the achievement of the objectives [9, 10].

The Project Management Institute defines the concept of risk as “an event or condition with uncertainty that, if realized, could have a positive or negative impact on the objectives of the organization.” According to the ISO 31000:2009 Principles and Principles standard risk, it is explained as the effect of uncertainty on the targets, and with the effect expressed here, positive or negative deviations from the expected situation are expressed [11].

The concept of risk management was first used in the insurance field in the early 1950s. The first principles of risk management were developed in the early 1960s, and in this context, it was emphasized that risks should not be contented with only insurance, but all risks should be managed. In parallel with this, risk management started to play an active role in political, economic, military, scientific, and technological fields in the following years [7].

Risk management, which was applied only for insurable risks in the past, has gained a different dimension today. Businesses have started to implement risk management in a way that takes into account strategic, operational, and financial risks [12].

As external factors, while it is expressed as economic events, natural environmental events, political events, social events, and technological events, it is classified as infrastructure-related events, personnel-related events, process-related events, and technology-related events as internal factors [13].

While the risks faced by businesses are generally classified as being from strategic, financial, operational, and external environments, the classification system based on internal and external factors by COSO (Committee of Sponsored Organizations), which offers a widely accepted risk management framework, is one of the comprehensive classifications [13].

The activities for businesses to define their risks and evaluate and reduce their risks appear as risk management. According to ISO 31000, the risk management process includes communication, negotiation, scoping, assessing risks, responding to risks, monitoring, reviewing, recording, and reporting [14].

All activities carried out on this basis, with the identification and evaluation of events or situations that are likely to occur and which are considered to affect the achievement of the administration’s goals and objectives, constitute the subject of risk management [15].

In summary, risk management exhibits a proactive approach that reduces uncertainties and the negative effects of uncertainty to a more acceptable level and prevents problems before they arise. In addition, it aims to lead the way in which opportunities are recognized in advance and turn them into advantages for the business.

Thanks to risk management, businesses identify the risks involved in the activities they carry out, evaluate the possibility of the risks to occur and the effect they will have when they occur, plan the necessary preventive actions, and thus turn the threat or danger element posed by the risks into an advantage [12]. With risk management, it is aimed not to completely eliminate risks, but to enable businesses to better understand their risks and manage them at a level they can control [16].

3. Certification bodies and risk factors

3.1 Certification bodies authorized by the vocational qualifications authority

Within the scope of our study, the risk factors of certification bodies authorized by the Vocational Qualifications Authority, a public institution in Turkey, are evaluated.

Vocational Qualifications Authority (VQA) is a public institution with a public legal personality, administrative and financial autonomy, established to establish and operate a national qualification system compatible with the European Union. The establishment purpose of the institution, as stated above, is to establish and operate a national qualification system compatible with the European Union [2].

In this context, VQA carries out work and procedures related to the preparation of national occupational standards, the development of national qualifications based on national or international occupational standards, the execution of activities for assessment and certification within the framework of national qualifications, and the regulation of the Turkish Qualifications Framework [2].

Within the scope of national occupational standards, the knowledge, skills, attitudes, and behaviors that must be possessed in order to perform a profession successfully, and the tasks, duties, and performance criteria that must be exhibited are defined. Within the scope of national qualifications prepared on the basis of national occupational standards, the procedures and principles for assessment and certification activities are determined [17].

Assessment and certification activities according to national qualifications are carried out by certification bodies authorized by VQA. The most basic condition of being a certification body authorized by VQA is to be accredited according to the international personnel certification standard called “TS EN ISO/IEC 17024:2012 Conformity Assessment - General Conditions for Personnel Certification Bodies”. After accreditation, compliance with the conditions determined in VQA legislation and regulations is examined, audited, and evaluated. Institutions and organizations that meet the requirements are authorized by VQA and carry out assessment and certification activities in relevant national qualifications. These institutions and organizations are regularly audited through both programmed and unscheduled audits [18].

The criteria that authorized certification bodies must meet are defined in the scope of “Authorization Criteria and Implementation Guide for Certification Bodies”. The criteria are grouped under 13 main headings. Each main criterion under these 13 main headings and sub-criteria related to this criterion define the conditions that must be met [3].

According to this guide, certification bodies must meet the conditions determined within the scope of legal status and organizational structure of organizations; human resources and management; physical, technical, and financial resources and management; examination materials, measurement, evaluation, and certification activities; internal and external verification; objections and complaints; information sharing; communication and guidance; internal and external audit activities; management of objectivity; policy, and objectives; and management of documents and records [3].

Situations or events that may prevent the realization of these conditions appear as risks. In this respect, organizations are expected to evaluate all their procedures, including the steps to be followed in fulfilling these conditions, to identify and evaluate possible risks that may prevent the effective implementation of their procedures, and to implement the necessary preventive actions to prevent risks [3].

3.2 Risks in assessment and certification activities of certification bodies

Within the scope of the study, the risks used in the design of the model are considered as the risks arising from the assessment and certification activities of the organizations, the human resources, physical and technical resources used in these activities, internal verification activities, assessment materials, the impartiality and reliability of the assessment, and certification activities.

While determining the risks, they are defined as situations or events that may cause significant or major noncompliance if they occur within the organizations, and that may cause the suspension or cancelation of the authority of the institutions. The identified risks were also confirmed by an expert group consisting of lead auditors appointed by VQA to take part in the audits of the organizations. Risks have been determined under the main headings and the risks are listed in Table 1 under the main headings.

Main risk group	Sub-risks
Human resources	Insufficient employment of assessors and internal verifiers.
	Failure of the assessor and internal verifiers to meet the assessor criteria.
	Assessors and internal verifiers do not have sufficient knowledge and experience.
	Lack of awareness of the assessor and internal verifiers about the system
Assessment and certification activities	The method used in theoretical and performance-based exams is not compatible with the qualifications.
	Failure to conduct theoretical and performance-based exams in accordance with the guidelines
	Assessor’s failure to conduct exams in accordance with scenarios, checklists, and national qualifications
	Failure to perform assessment activities accurately, consistently, and reliably
	Failure to make correct, consistent, fair, and reliable certification decisions
Internal verification activities	Failure to operate internal verification activity for each national qualification, qualification unit, and assessor
	Failure to perform internal verification activities in accordance with national qualifications
	Failure of internal verifiers to make accurate, consistent, and fair assessments
	Inadequate creation of the sampling plan in internal verification activities
	Failure to take corrective actions for detected nonconformities within the scope of internal verification
Assessment material	Not creating enough questions to meet the knowledge statements in the annex of the qualification units
	The question booklets do not contain a sufficient number and quality of questions to meet the knowledge statements.
	Scenarios and checklists do not meet the skills and competencies in the annex of the qualification units
	Failure to verify the suitability of materials used in assessment processes
Physical and technical facilities	Inadequate physical environments to measure skills and competencies
	Failure to take adequate OHS measures in the areas where performance-based assessments are held
	Equipment and materials are not suitable for measuring skills and competencies
	Failure to take adequate measures to ensure the reliability of equipment
Impartiality and Reliability	Lack of awareness of assessor and internal verifiers for a consistent and fair assessment.
	Possible conflicts of interest between assessors and candidates
	The internal verifier has a conflict of interest with the candidate or assessor
	Failure to take adequate precautions for reliable assessment

Table 1.

Main risk group and sub-risks.

The model designed in this study was used to evaluate the risks (Table 1) and it was proposed as a new risk assessment method.

4. Fuzzy DEMATEL (fuzzy decision-making trial and evaluation laboratory) method

The DEMATEL method is a multi-criteria decision-making method and is used to solve many complex problems. With this method, the relationships between the variables are evaluated and these relationships are visualized through diagrams showing cause-and-effect relationships. Thanks to this method, all variables are determined as influencing and affected variables, or in other words, cause-effect relationships and the structural relationship between the variables is revealed [19]. The DEMATEL method has a superior feature compared to other multi-criteria decision-making methods as it deals with the interrelationships between variables.

Fuzzy logic was first introduced by Lotfi A. Zadeh in 1965. Fuzzy logic is an approach that is based on thinking like a human and adopts that the key elements of human thought are linguistic variables [20]. The differences in perception arising from the way of thinking of people and the uncertainties in their subjective behaviors and goals are explained by the concept of blurriness, and in this respect, it is defined as the application of fuzzy mathematics to the real world. In fuzzy logic, variables are classified without precise evaluations. Unlike classical logic, it models the data by using linguistic variables such as “very little, little, medium, high, very high” instead of definite propositions such as true-false or yes-no. Afterward, these expressions are converted into fuzzy numbers and more realistic solutions are obtained [21].

The DEMATEL method reveals the relationship between variables in complex systems and it is not always possible to evaluate these variables with definite propositions. At this point, fuzzy logic is used and expert opinions about the variables are converted into fuzzy numbers. In summary, the Fuzzy DEMATEL method is obtained by transferring the DEMATEL method to the fuzzy environment [22].

When the studies in the literature with the fuzzy DEMATEL method are examined, the fuzzy DEMATEL method was used to investigate the factors affecting the adoption of new technology and to determine the relationship between the factors in the study conducted by Zargar et al. [23]. In the study by Chang et al., fuzzy DEMATEL method was used to determine supplier selection criteria [24]. In the study conducted by Chou et al., fuzzy AHP and fuzzy DEMATEL methods were applied integrated in order to evaluate human resources in the field of science and technology [25]. Çelik and Akyüz used the fuzzy DEMATEL method to evaluate the critical hazards in the gas release process in oil tankers [26]. Seker and Zavadskas used the fuzzy DEMATEL method in the analysis of occupational risks in the construction industry [27]. Mahmoudi et al. used the fuzzy DEMATEL method to determine the critical success factors for the self-care process in heart failure [28]. Feng and Ma determined the factors affecting service innovation in the manufacturing sector with fuzzy DEMATEL [29].

4.1 Steps of the method of fuzzy DEMATEL

Although the steps of the fuzzy DEMATEL method are similar to the steps of the DEMATEL method, fuzzy numbers are used in this method and these numbers need to be defuzzification in order to convert them into definite results. At this point, unlike the DEMATEL method, the defuzzification process is integrated into the steps of the method. Although various methods are used in defuzzification, the CFCS (Converting Fuzzy Data into Crisp Scores) method used in a study by Opricovic and Tzeng in 2003 was used within the scope of our study [30].

Zhou et al. used the fuzzy DEMATEL method to determine critical success factors in emergency management in 2011. The steps followed in the study by Zhou et al. are listed below [31]. In this study, Zhou et al. used the CFCS method, developed by Opricovic and Tzeng [30], which is used to defuzzifying fuzzy numbers. The steps and demonstrations presented within the scope of Zhou et al.’s work were also used in our study [31];

Step 1: Determine the initial direct-relation matrix.

At this stage, a group of experts is formed in order to determine the relationships between variables, criteria, or factors. Linguistic variables and fuzzy numbers in Table 2 are used when group members make pairwise comparisons.

Definition	Triangular fuzzy numbers
No influence	(0, 0, 0.25)
Very low influence	(0, 0.25, 0.50)
Low influence	(0.25, 0.50, 0.75)
High influence	(0.50, 0.75, 1.00)
Very high influence	(0.75, 1.00, 1.00)

Table 2.

Triangular fuzzy numbers according to the degree of effect.

At this stage, the relations between the criteria or factors are evaluated by experts by making pairwise comparisons. As a result of the evaluation, an initial direct matrix consisting of triangular fuzzy numbers is obtained. Defuzzification processes are applied to obtain the initial direct matrix with the crisp values.

Step 2: Defuzzification.

In this study, CFCS (converting fuzzy data into crisp scores) method was used in order to convert fuzzy numbers into crisp values.

$$z_{\mathit{ij}}^k=\left(l_{\mathit{ij}},m_{\mathit{ij}},r_{\mathit{ij}}\right)$$E1

1 ≤ k ≤ K.

K: Number of experts.

$z_{\mathit{ij}}^k$: Evaluation of the effect of the i criterion on the j criterion by the kth expert in a fuzzy environment.

The following formulas are used for normalization, calculation of left and right normalized value, calculation of total normalized value, and calculation and integration of crisp value for defuzzification operations.

4.1.1 Normalization

$${\mathit{xl}}_{\mathit{ij}}^k=\left(l_{\mathit{ij}}^k-\underset{1\le k\le K}{min}{l}_{\mathit{ij}}^k\right)/{∆}_{\mathit{min}}^{\mathit{max}.}$$E2

$${\mathit{xm}}_{\mathit{ij}}^k=\left(m_{\mathit{ij}}^k-\underset{1\le k\le K}{min}{l}_{\mathit{ij}}^k\right)/{∆}_{\mathit{min}}^{\mathit{max}}$$E3

$${\mathit{xr}}_{\mathit{ij}}^k=\left(r_{\mathit{ij}}^k-\underset{1\le k\le K}{min}{l}_{\mathit{ij}}^k\right)/{∆}_{\mathit{min}}^{\mathit{max}}$$E4

$${∆}_{\mathit{min}}^{\mathit{max}}={\mathit{\text{maxr}}}_{\mathit{ij}}^k-min{l}_{\mathit{ij}}^k$$E5

4.1.2 Computing of left (ls) and right (rs) normalized values

$${\mathit{xls}}_{\mathit{ij}}^k=x{m}_{\mathit{ij}}^k/\left(1+x{m}_{\mathit{ij}}^k-x{l}_{\mathit{ij}}^k\right)$$E6

$${\mathit{xrs}}_{\mathit{ij}}^k=x{r}_{\mathit{ij}}^k/\left(1+x{r}_{\mathit{ij}}^k-x{m}_{\mathit{ij}}^k\right)$$E7

4.1.3 Computing total normalized crisp values

$$x_{\mathit{ij}}^k=\left[x{\mathit{ls}}_{\mathit{ij}}^k\left(1-x{\mathit{ls}}_{\mathit{ij}}^k\right)+x{\mathit{rs}}_{\mathit{ij}}^{k}x{\mathit{rs}}_{\mathit{ij}}^k\right]/\left(1+x{\mathit{rs}}_{\mathit{ij}}^k-x{\mathit{ls}}_{\mathit{ij}}^k\right)$$E8

4.1.4 Computing crisp values

$${\mathit{BNP}}_{\mathit{ij}}^k={\mathit{\text{minl}}}_{\mathit{ij}}^k+x_{\mathit{ij}}^k{∆}_{\mathit{min}}^{\mathit{max}}$$E9

4.1.5 Integrating crisp values

$$a_{\mathit{ij}}^k=\frac{1}{K}{\sum }_k^{1\le k\le K}{\mathit{BNP}}_{\mathit{ij}}^k$$E10

As a result of the operations performed, the initial direct-relation matrix is obtained.

Step 3: Obtaining the normalized direct-relation matrix

By means of the formula below, the normalized direct-relation matrix is obtained.

$$\mathrm{D}=\mathrm{A}/\mathrm{s}$$E11

s = max (max $\sum _{j=1}^n\mathit{aij}$, max $\sum _{i=1}^n\mathit{aij}$)

i,j = 1,2,……n

Step 4: Obtaining the total-relation matrix

When the normalized direct-relation matrix D is obtained, the total-relation matrix T is calculated using the formula below. “I” stands for the unit matrix

$$\mathrm{T}=\mathrm{D}+{\mathrm{D}}^2+{\mathrm{D}}^3+\dots \dots ..{}^{=}{\sum }_{i=1}^{\infty }{D}^{\mathrm{i}}=\mathrm{D}{\left(\mathrm{I}-\mathrm{D}\right)}^{-1}$$E12

Step 5: Identifying cause and effect groups

The sum of the rows in the T matrix is determined by ri and the sum of the columns by cj. Cause and effect groups are determined by calculating “ri – cj” and “ri + cj” values.

The “r” obtained as a result of row sums shows the effect of the ith factor on other factors. The sum of the columns “cj” shows the effect of other factors on the ith factor. “ri + cj” values show the total effect and the effective value of the relevant factor, in other words, the degree of relations with other criteria.

Among the “ri – cj” values, those with positive values express those that affect other criteria, while those with negative values express those who are affected by other criteria. In other words, the value of “ri – cj” expresses the effect of that criterion on the system [32].

Step 6: Producing diagrams of cause and effect groups

Diagrams are obtained by showing “ri + cj” values on the horizontal axis and “ri – cj” values on the vertical axis on the coordinate plane. If the (ri − cj) axis is positive, the factor is in the cause group. Otherwise, if the (ri − cj) axis is negative, the factor is in the effect group.

A threshold value is determined in order to get rid of the complexity of the criteria with a small effect level. The threshold value is determined by averaging the values in the total correlation matrix or by an expert group. The criteria below the threshold value are determined as the affected (effect) criteria, and the criteria above the threshold value are determined as the affecting (cause) criteria [33].

Step 7: Calculating criterion weights

The following formula was used to calculate the criterion weights [34].

$$w_i=\sqrt{\left[\left(r_i+c_j\right)\right]2+\left[\left(r_i-c_j\right)\right]2}$$E13

$$\mathrm{Wi}=\frac{\mathit{wi}}{{\sum }_{i=1}^n\mathit{wi}}$$

Step 8: Operating the steps for the main criteria

All the steps described above are operated to determine the main criterion weights.

Step 9: Operating the steps for sub-criteria.

All the steps described above are operated for the sub-criteria under each main criterion group in order to calculate the sub-criteria weights, and as a result, the sub-criteria weights are calculated.

Step 10: Integrating main criterion and sub-criteria weights

The final weights are calculated by multiplying the weights of the main criteria with the weights of the sub-criteria.

5. Weighting the risks of authorized certification bodies by fuzzy DEMATEL method

Within the scope of the study, the main risks that may be encountered in certification bodies authorized by VQA and sub-risks related to these risks were determined, and these risks were confirmed by the lead auditors in the audit of certification bodies. The weights of the risks were calculated using the “Fuzzy DEMATEL Method” introduced in the previous section.

Step 1: Demonstrating the relationship between risks

The network structure of the model is presented in the Figure below (Figure 1). The relations between the main criteria and the sub-criteria are shown in the network structure of the model. As a result of the evaluation made with the expert group, it was evaluated that all the criteria were in interaction with each other.

Figure 1.

Relations between main and sub-risk criteria.

Step 2: Designing the questionnaire

A questionnaire consisting of two parts was designed for the application of the fuzzy DEMATEL method. In the first part of the questionnaire, in order to determine the relations between the main criteria, and in the second part, in order to determine the relations between the sub-criteria under each main criterion group, matrices were designed to allow pairwise comparison. Questionnaires were asked to make pairwise comparisons using these matrices and to determine whether the risks affect each other. The questionnaire was administered to a group of experts consisting of 12 people. The expert group was selected from people who are in charge as lead auditors in VQA audits and had sufficient knowledge and experience in assessment and certification and audit activities.

Step 3: Calculating the inconsistency rate of the questionnaire results

The inconsistency rate of the obtained data was determined in accordance with the formula for the calculation of the inconsistency rate presented within the scope of a study conducted by Wang and Tzeng in 2012 [35]. The formula is presented below;

$$\text{Inconsistency rate}=\frac{1}{n\left(n-1\right)}\sum _{i=1}^n\sum _{j=1}^n\frac{\left|a_{\mathit{ij}}^p-a_{\mathit{ij}}^{p-1}\right|}{a_{\mathit{ij}}^p}\times 100\%$$E14

n = Number of criteria

p = Number of experts

$${\mathrm{a}}_{\mathrm{ij}}^{\mathrm{p}}=\text{average of data from}\mathrm{p}\text{experts for each pairwise comparison}$$

$${\mathrm{a}}_{\mathrm{ij}}^{\mathrm{p}-1}=\text{average of data from}\mathrm{p}-1\text{experts for each pairwise comparison}$$

If the inconsistency rate is <5%, the obtained data is determined to be consistent.

In line with the formula presented above, it has been determined that the data obtained as a result of the calculations made for the main criteria and the sub-criteria defined under the main criteria are consistent. Consistency rates are presented in Table 3. Since the consistency ratios of all criteria are less than 0.05, it is seen that the data are consistent.

Criteria	Consistency rate	Evaluation
Main criteria	0,03	Consistent
Human resources	0,04	Consistent
Assessment and certification activities	0,03	Consistent
Internal verification activities	0,04	Consistent
Assessment material	0,04	Consistent
Physical and technical facilities	0,03	Consistent
Impartiality and reliability	0,04	Consistent

Table 3.

Consistency rate of the data obtained from the expert group for the main criteria and sub-criteria.

Step 4: Conversion of survey data to fuzzy numbers

The data obtained as a result of pairwise comparisons made by each member of the expert group for the main criteria and sub-criteria were converted into fuzzy numbers. The triangular fuzzy values in Table 1 were used to transform the data into fuzzy numbers.

Step 5: Utilizing CFCS (Converting Fuzzy Data into Crisp Scores) defuzzification method to defuzzifying fuzzy numbers and creating the initial matrix.

The normalization process was carried out by using the CFCS method steps presented in Eqs. (1)–(10). As a result of the calculations, xls and xrs matrices were obtained for both the main risk criteria group and the sub-risk criteria groups under the main risk criteria group. By using these matrices, the total normalized value and the crisp value were calculated.

After obtaining the crisp values, the initial direct-relation matrices were calculated using Eq. (10). The initial direct-relation matrix obtained for the main risk criteria is presented in Table 4 for illustrative purposes. The same calculations were made for the sub-risk criteria groups.

A matrix	1	2	3	4	5	6
1	0,045	0,980	0,898	0,568	0,431	0,841
2	0,586	0,045	0,630	0,668	0,508	0,768
3	0,610	0,790	0,045	0,610	0,553	0,648
4	0,357	0,941	0,633	0,045	0,567	0,645
5	0,240	0,770	0,594	0,568	0,045	0,513
6	0,703	0,907	0,785	0,575	0,497	0,045

Table 4.

Initial direct-relation matrix for main risk criteria (A).

Step 6: Obtaining the normalized direct-relation matrix

Normalized direct-relation matrices were obtained by using Eq. (11). The normalized matrix obtained for the main risk criteria is presented in Table 5 for illustrative purposes. The same calculations were made for the sub-risk criteria groups.

D matrix	1	2	3	4	5	6
1	0,010	0,221	0,203	0,128	0,097	0,190
2	0,132	0,010	0,142	0,151	0,115	0,173
3	0,138	0,178	0,010	0,138	0,125	0,154
4	0,080	0,212	0,143	0,010	0,128	0,146
5	0,054	0,174	0,134	0,128	0,010	0,116
6	0,159	0,204	0,177	0,130	0,112	0,010

Table 5.

Normalized direct-relation matrix for main risk criteria (D).

Step 7: Obtaining the total-relation matrices

Using Eq. (12), the total-relation matrices were calculated. The total relation matrix obtained for the main risk criteria is presented in Table 6. Total relation matrices were also obtained for the sub-risk criteria groups.

T matrix	1	2	3	4	5	6
1	0,014	0,160	0,126	0,065	0,042	0,115
2	0,055	0,015	0,073	0,071	0,046	0,092
3	0,058	0,112	0,014	0,065	0,051	0,081
4	0,030	0,135	0,072	0,014	0,052	0,074
5	0,017	0,095	0,060	0,052	0,013	0,050
6	0,073	0,139	0,102	0,063	0,047	0,015

Table 6.

Total relation matrix for main risk criteria (T).

Step 8: Identifying cause and effect groups

The sum of the rows in the T matrix is shown with ri and the sum of the columns with cj, and the cause and effect groups are determined by calculating the values of “ri – cj” and “ri + cj”. The cause and effect groups calculated for the main risk criteria are presented in Table 7. Similarly, cause and effect groups were calculated for the sub-risk criteria groups.

Criteria	“ri – cj”	“ri + cj”	Group definiton	Criteria	“ri – cj”	“ri + cj”	Group definiton
1	0,28	0,77	Cause	4	0,05	0,71	Cause
2	−0,30	1,01	Effect	5	0,04	0,54	Cause
3	−0,06	0,83	Effect	6	0,01	0,87	Cause

Table 7.

Cause and effect groups for main risk criteria.

A threshold value has been determined in order to avoid the complexity of the criteria with a small effect level. The threshold value was calculated by averaging the values in the total relationship matrix and 0.07 was obtained for the main risk criterion total relationship matrix.

Criteria below the threshold value were determined as affected (effect) criteria, and criteria above the threshold value were determined as affecting (cause) criteria [33].

Values below the threshold value of 0.07 for the main risk criterion total relationship matrix are shown with “-” and presented in Table 8. Similarly, threshold values were calculated for the sub-risk criteria groups.

T matrisi	1	2	3	4	5	6
1	—	0,16	0,126	—	—	0,115
2	—	—	0,073	0,071	—	0,092
3	—	0,112	—	—	—	0,081
4	—	0,135	0,072	—	—	0,074
5	—	0,095	—	—	—	—
6	0,073	0,139	0,102	—	—	—

Table 8.

Illustration of values above and below the threshold value.

According to the values in Table 8, a cause and effect diagram was produced for the main risk criterion matrix, which is shown in Figure 2. Similarly, cause and effect diagrams were produced for the sub-risk criteria groups.

Figure 2.

Cause and effect diagram for the main risk criterion matrix.

Step 9: Calculation of criterion weights

Main risk criteria weights and sub-risk criteria weights were calculated using Eq. (13) and the results are presented in Table 9. The final weights were obtained by multiplying the weights of the main criteria and the weights of the sub-criteria. The final weights are shown in Table 9.

Main criteria	Main criterion weight	Sub criteria	Weights	Final weights
Human resources	0,17	1	0,10	0,018
		2	0,30	0,050
		3	0,32	0,054
		4	0,28	0,048
Assessment and certification activities	0,22	1	0,20	0,044
		2	0,19	0,041
		3	0,21	0,045
		4	0,21	0,046
		5	0,20	0,044
İnternal verification activities	0,17	1	0,23	0,038
		2	0,23	0,039
		3	0,22	0,037
		4	0,21	0,035
		5	0,17	0,029
Assessment material	0,15	1	0,26	0,039
		2	0,25	0,039
		3	0,21	0,039
		4	0,28	0,039
Physical and technical facilities	0,11	1	0,25	0,027
		2	0,24	0,027
		3	0,24	0,026
		4	0,27	0,029
Impartiality and reliability	0,18	1	0,24	0,043
		2	0,23	0,042
		3	0,23	0,042
		4	0,30	0,053

Table 9.

Table showing main criterion weights, sub-criteria weights, and final weights.

Step 10: Classification of criteria

Based on the criteria weights, the criteria were classified as high-, moderate-, and low-risk groups together with the expert group. While making this classification, risks with a value between 0 and 0.035 were included in the low-risk group, risks with a value between 0.035 and 0.045 were included in the moderate-risk group, and finally risks with a value above 0.045 were included in the high-risk group. The risk groups according to the weights of the criteria are shown in Table 10.

No	Sub criteria	Definition of risk criteria	Final weight	Risk group
1	A3	Assessors and internal verifiers do not have sufficient knowledge and experience.	0,054	High
2	F4	Failure to take adequate precautions for reliable assessment	0,053	High
3	A2	Failure of the assessor and internal verifiers to meet the assessor criteria	0,050	High
4	A4	Lack of awareness of the assessor and internal verifiers about the system	0,048	High
5	B4	Failure to perform assessment activities accurately, consistently, and reliably	0,046	High
6	B3	Assessor’s failure to conduct exams in accordance with scenarios, checklists, and national qualifications	0,045	High
7	B1	The method used in theoretical and performance-based exams is not compatible with the qualifications.	0,044	Moderate
8	B5	Failure to make correct, consistent, fair, and reliable certification decisions	0,044	Moderate
9	F1	Lack of awareness of assessor and internal verifiers for consistent and fair assessment.	0,043	Moderate
10	F2	Possible conflicts of interest between assessors and candidates	0,042	Moderate
11	F3	The internal verifier has a conflict of interest with the candidate or assessor	0,042	Moderate
12	B2	Failure to conduct theoretical and performance-based exams in accordance with the guidelines	0,041	Moderate
13	D1	Not creating enough questions to meet the knowledge statements in the annex of the qualification units	0,039	Moderate
14	D2	The question booklets do not contain sufficient numbers and quality of questions to meet the knowledge statements.	0,039	Moderate
15	D3	Scenarios and checklists do not meet the skills and competencies in the annex of the qualification units	0,039	Moderate
16	D4	Failure to verify the suitability of materials used in assessment processes	0,039	Moderate
17	C2	Failure to perform internal verification activities in accordance with national qualifications	0,039	Moderate
18	C1	Failure to operate internal verification activity for each national qualification, qualification unit, and assessor	0,038	Moderate
19	C3	Failure of internal verifiers to make accurate, consistent, and fair assessments	0,037	Moderate
20	C4	Inadequate creation of the sampling plan in internal verification activities	0,035	Moderate
21	E4	Failure to take adequate measures to ensure the reliability of equipment	0,029	Low
22	C5	Failure to take corrective actions for detected nonconformities within the scope of internal verification	0,029	Low
23	E1	Inadequate physical environments to measure skills and competencies	0,027	Low
24	E2	Failure to take adequate OHS measures in the areas where performance-based assessments are held	0,027	Low
25	E3	Equipment and materials are not suitable for measuring skills and competencies	0,026	Low
26	A1	Insufficient employment of assessors and internal verifiers	0,018	Low

Table 10.

Risk groups to which the criteria belong.

6. Development of a new risk management model

The weights of the risk criteria were calculated by the fuzzy DEMATEL method and classified as high, moderate, and low-risk groups according to the data obtained. The effects that the risks will create in case of occurrence are classified as shown in Table 11.

Impact level	Numerical value
Very low	1
Low	2
Moderate	3
High	4
Very high	5

Table 11.

The effects that the risk will create in case of occurrence.

The “Risk Decision Matrix” in Appendix Table A1 was created by utilizing the risk criterion weights and impact scores. It was obtained by dividing the weights of the risk criteria (importance of the risks) into the matrix depending on the effective value of the risks in case of occurrence.

The five-point value scale developed by Liberatore was used to rank the impact values of the risks. This scale consists of excellent, good, moderate, mediocre, and weak points [36]. This scale is adapted to classify effect values as very low, low, medium, high, and very high.

For example, the weight of the risk of “the assessor and internal verifiers do not have sufficient knowledge and experience” was determined as 0.054. The resulting weight was multiplied by “1000.” The effects that the risk will create in case of occurrence and their scores are listed in the “Risk Decision Matrix” in Appendix Table A1. Accordingly, the score of the risk in case of very high impact is 0.054, in case of high impact “(4*0.054)/5”, in case of medium impact “(3*0.054)/5”, in case of low impact “(2*0.054)/5,” and in case of very low impact, it was calculated as “(1*0.054)/5.” Similar calculations were made for all sub-risk criteria.

The actions to be taken in case of occurrence of risks according to the risk decision matrix are shown in Table 12. The actions to be taken by Vocational Qualification Authority in case of occurrence of the risk are determined together with the expert group.

Numerical value	Color of the region	Preventive actions to be taken by the organization depending on the risk value	Action to be taken by VQA in case of occurrence of risk
0–10	Light gray	Initiation of preventive action	Nonconformity to be corrected
11–20	Gray	Not taking assessment/postponing assessment before the preventive action is completed	Suspension of assesments
21–30	Blue	Not accepting the candidate application before the preventive action is completed	Cancelation of assesments
31–40	Light red	Not accepting the candidate application without internal verification and reverification of all processes and elements related to assessment activities	Suspension of authority
41 and above	Red		Withdrawal of authority

Table 12.

Precautions to be taken against risk and actions to be taken in case of occurrence.

As a result, a new risk management model has been proposed for organizations to evaluate risks related to measurement and evaluation activities and to take necessary precautions. The flow of the new risk management model is presented in Figure 3.

Figure 3.

The flow of the new risk management model.

7. Conclusion

Risk management is defined as the activities aimed at identifying the risks that businesses may encounter and evaluating and reducing these risks. The subject of risk management is to define, evaluate, and respond appropriately to events or situations that are likely to occur and are considered to affect the achievement of the goals and objectives of the administration when they occur [15].

Thanks to risk management, businesses identify the risks involved in the activities they perform, evaluate the possibility of the risks to occur and the effect when they occur, plan the necessary preventive actions, and thus turn the threat or danger element posed by the risks into an advantage [12].

Within the scope of the study, the possible risks to be encountered in the assessment and certification activities of the certification bodies authorized by the Vocational Qualifications Authority, a public institution operating in Turkey, were evaluated. The possible risks in the assessment activities of the organizations were determined, and the fuzzy DEMATEL method, which is one of the Fuzzy Multi-Criteria Decision Making Methods, was used to evaluate the risks.

For the application of the method, a questionnaire questioning the effect status among the risk criteria was designed. The designed questionnaire was applied to a group of experts consisting of 12 people. At this stage, the CFCS (Converting Fuzzy Data into Crisp Scores) defuzzification method, which was developed by Opricovic and Tzeng (2003) for the application of fuzzy DEMATEL, was used [30, 31].

The answers given to the questionnaires by the expert group were converted into fuzzy numbers and these numbers were clarified by using the CFCS method. The relations and risk weights between the main risk groups, the relations between the sub-risks under the main risk groups, and the sub-risk weights were determined. The final weights were obtained by integrating the main risk criteria weights and sub-risk criteria weights. Depending on the criterion weights, the risks are classified as low, moderate, and high.

As a result of the calculations made within the scope of fuzzy DEMATEL, it was determined that the criterion with the highest priority among the main risk criteria groups was “Assessment and Certification Activities” with a score of 0.22. This criterion was followed by “Impartiality and Reliability” with 0.18 points, “Internal Verification Activities” with 0.17 and “Human Resources” with 0.17 points. While the main criteria of internal verification and human resources were of equal importance, “Assessment Material” with 0.15 points and “Physical and Technical Facilities” with 0.11 points followed these criteria.

When the sub-risk criteria are examined within the scope of the main criteria, it has been determined that the criteria with the highest priority and accordingly the high-risk group are generally sub-criteria within the scope of the human resources main criterion.

It was determined that the criteria defined under assessment and certification activities came in second place, and the criteria within the scope of impartiality and reliability took the third place. While the criteria for impartiality and reliability, assessment material, and internal verification are generally in the moderate-risk group, the criteria for physical and technical facilities are in the low-risk group.

It has been observed that the distribution of the criteria to risk groups (low, moderate, and high) is homogeneous. Validity and reliability of assessment and certification activities depend on the competence of the assessors and internal verifiers, and the risks that may arise from the assessors and internal verifiers pose a high risk in terms of the validity and reliability of the related activities. For this reason, the fact that the sub-risk criteria weights due to human resources are high and they are included in the high-risk group have been evaluated as a suitable result by the expert group.

The weights of the risks have been determined, the effects to be created in the assessment and certification activities in case of occurrence of the risks have been calculated, and the activities to be carried out according to the obtained results have been determined. The effects of the risks in case of occurrence are classified as very low, low, moderate, high, and very high. Impact values were assigned as one for very low, two for low, three for moderate, four for high, and five for very high.

By making use of the five-point value scale, the risk criteria weights were distributed to the matrix depending on the impact value of the risks in case of occurrence, and the “Risk Decision Matrix” was obtained. In line with the results obtained from the matrix, the precautions that should be applied by the institutions and the sanctions to be applied in case the risks occur in the systems of the institutions were determined.

As a result, a new risk management model was designed for the assessment and management of risks. The new model designed both offers a new approach and guides the institutions in the management of the risks in the assessment activities of the assessment and certification bodies. It is evaluated that the stages defined in the new risk management model designed can be used in many different fields of activity, and thus, businesses operating in various sectors and fields can identify and measure the possible risks in their system. According to risk value obtained by using this model, they can determine the necessary precautions and sanctions.

Weights				Impact value
Group of risks	Number of sub-criteria	Final weights	Final weights *1000	Very low (1)	Low (2)	Moderate (3)	High (4)	Very high (5)
High	A3	0,054	54	11	22	32	43	54
High	F4	0,053	53	11	21	32	42	53
High	A2	0,050	50	10	20	30	40	50
High	A4	0,048	48	10	19	29	38	48
High	B4	0,046	46	9	18	28	37	46
High	B3	0,045	45	9	18	27	36	45
Moderate	B1	0,044	44	9	18	26	35	44
Moderate	B5	0,044	44	9	18	26	35	44
Moderate	F1	0,043	43	9	17	26	34	43
Moderate	F2	0,042	42	8	17	25	34	42
Moderate	F3	0,042	42	8	17	25	34	42
Moderate	B2	0,041	41	8	16	25	33	41
Moderate	D1	0,039	39	8	16	23	31	39
Moderate	D2	0,039	39	8	16	23	31	39
Moderate	D3	0,039	39	8	16	23	31	39
Moderate	D4	0,039	39	8	16	23	31	39
Moderate	C2	0,039	39	8	16	23	31	39
Moderate	C1	0,038	38	8	15	23	30	38
Moderate	C3	0,037	37	7	15	22	30	37
Moderate	C4	0,035	35	7	14	21	28	35
Low	E4	0,029	29	6	12	17	23	29
Low	C5	0,029	29	6	12	17	23	29
Low	E1	0,027	27	5	11	16	22	27
Low	E2	0,027	27	5	11	16	22	27
Low	E3	0,026	26	5	10	16	21	26
Low	A1	0,018	18	4	7	11	14	18

Table A1.

Risk decision matrix.

Thanks

The author of the chapter would like to thank Prof. Dr. Abdullah Süreyya ERSOY and also the managers and employees of the Vocational Qualifications Authority who supported the necessary work for the writing of the chapter.