DANP-Evaluation of AHP-DSS

Abstract

The analytic hierarchy process (AHP) and the analytic network process (ANP) are important multi-criteria decision-making (MCDM) methods for solving strategic decision problems. In the field of the research and teaching projects of a university’s Management Science Department, the use of adequate decision support systems (DSS) enables an appropriate application and acceptance of these methods. By reason of the great variety of AHP-DSS, the aim of this paper is the selection of AHP-supporting software. Owing to the interdependencies of the software quality criteria, these influences can be evaluated appropriately by the ANP. As for the various requirements of the different department members, the ANP procedure is linked with the DEMATEL approach. Within such a combined framework (DANP), the alternate software products and their quality selection criteria are transparently analysed and evaluated from a multi-personal point of view. The described procedure is an object of reference to solve such structuring and evaluation problems by support of parallel and/or distributed computing architecture.

1. Introduction

In the field of academic teaching and research, the multi-criteria decision-making (MCDM) methods gain ongoing increasing importance. In many lectures, seminaries, exercises, tutorials, papers, etc., these methods are presented, and decision support systems (DSS) are applied. The same applies to academic research and paper production. Within this context, the analytic hierarchy process (AHP) and the analytic network process (ANP) are regarded as important MCDM methods to solve strategic decision problems. Additionally, the impact of these two methods in the literature is continuously growing. Thereby, working with these methods and embedding them into the systematic environment of an adequate DSS becomes undeniable for students and academic staff.

Despite the comparatively less sophisticated mathematical computation of the more popular AHP, it is necessary to secure an efficient application of this method by a suitable DSS. By this, a correct implementation of this method is brought forward and the acceptance of academic staff and students can be boosted. On this background, the paper presents five substantially varying AHP-DSS, evaluated by five members of a Management Science Department of a medium-sized university. These persons have different profiles in academic teaching and research experiences, requirements and preferences. Based on standard criteria of ISO/IEC 25010 to evaluate the quality of software products, modified criteria were customized to the specificity of AHP software products and the demands of a Management Science Department. To cope with (inter-)dependencies of the evaluation criteria, the ANP is used as evaluation method and supported by DEMATEL to reconsider the wide range of requirements of the different department members. As a contribution to the field of ANP application, the DANP procedure is transparently shown. Furthermore, the implications of more network complexity and of an enhancing number of experts with diverging software quality requirements regarding a demand for parallel and/or distributed computing architectures are subsequently focused.

The remainder of the chapter is organized as follows: Section 2 provides a critical overview of AHP’s and ANP’s conceptual foundations in the field of discrete strategic decision problems. Furthermore, the necessity of using DSS is pointed out. Section 3 is devoted to the research framework and the evaluation of AHP-DSS with DANP followed by considerations on a possible support by parallel and distributed computing (Section 4). The chapter ends with a summary of the main results of the study as well as with concluding remarks and future prospects (Section 5).

2. Applying AHP and ANP in a Management Science Department

One of the fundamental tasks of Management Science is the support of complex managerial decision problems. For this purpose, information and supporting methodology relevant to the decision-making process must be made available. This applies particularly to the field of future-oriented strategic decision settings which require top management, involve the allocation of a large amount of resources, are likely to have a significant impact on the long-term prosperity of the company with major consequences and necessitate to consider internal and external environmental factors [1]. Academic teaching and research in business has to take these requirements into account and to embed these decision problems in a multidimensional decision system with diverging goals. This task can be induced by multiple top-goals (to be sufficed in a not-for-profit organization) or an analogous structure of the relationship between multiple causes and one intended financial effect in the context of the steering tasks of a traditional entrepreneurial organization and its cause-and-effect structure relevant for financial performance generation.

The analysis of multiple goal decision problems has evolved continuously over recent decades, primarily in the field of operations research (OR) beginning with goal programming [2–4]. In business, OR is an important support for decision-making by available adequate MCDM methods. Such methods are becoming increasingly important for decision support functions. The results of a performed MCDM related bibliometric study [5] revealed that AHP [6–9] and ANP [10–13] are two of the most adequate decision support methods and the most important MADM approaches for solving complex discrete decision problems. Comparative advantages of both methods are for instance that they are able to cope with quantitative and qualitative criteria by the possibility of considering ordinal and cardinal judgements, with the involvement of more than one decision maker and the ongoing development of efficient software support.

Within the AHP, decision problems have to be structured in a clear and unambiguous hierarchy with an overall goal, sub-goals, criteria and alternatives. The ANP—as a more general form of the AHP—exceeds the AHP by the possibility to consider dependence and feedback between criteria referring to the problem.

With respect to the complexity of strategic management decisions which can be disassembled into variety (number and type of elements) and connectivity (number and type of relations between the elements), there can be distinguished between managerial decision problems with a lower and a higher degree of complexity. As the variety of elements is not influencing the choice between AHP and ANP due to the fact that both approaches can handle a lot of different decision elements at the same time, the focus lies on the connectivity aspect of a decision environment. If there is a lower level of complexity with a manageable amount of dependencies in a hierarchic structure, the AHP can be used. In the case of higher complexity (increasing connectivity) with more horizontal dependencies, the ANP is the adequate decision support technique.

Even though many complex strategic decision settings can be depicted through a network structure, an ANP model must not yield better results than using the AHP [14]. Using hierarchies (as structural characteristic of the AHP) has furthermore the advantage that this system can be used to describe changes in priority on higher levels affect the priority of elements on lower levels. Constraints of the elements on a level are represented on the next higher level to ensure that they are met. Moreover, hierarchies are stable and flexible which means that small changes cause small effects and that additions to a well-structured hierarchy do not disrupt the performance [6, 7].

The AHP can support complex strategic decisions, e.g. the selection of new suppliers, locations of production plants or capital goods of any kind [5]. By contrast, the ANP should be used in case of interdependencies between criteria, for instance, to be reconsidered in means-end-relationships for organizational policy on the basis of the cause-and-effect structure of the financial performance generation. In comparison to ANP, the AHP is furthermore more popular because of less complexity during the modelling process and on the other hand due to less sophisticated mathematical requirements. But nevertheless, AHP-DSS are necessary for an adequate application of the method and its acceptance by scholars and managers. As DSS comprise a wide spectrum of characteristics, it is important to select a product adequate to requirements which will vary within a Management Science Department according to different persons and their functions. Therefore, our aim is a transparent evaluation of selected AHP-DSS in a multi-personally organized process.

As the relevance of the AHP for scientists and practitioners is proved in bibliometric studies [5, 15], a need for AHP-adequate software support for these groups of persons is comprehensible. The question is first of all, if such software products should be evaluated and selected by AHP or by ANP?

Even though AHP-based evaluations of AHP-software exist [16–18], it is to be considered that some criteria relevant for quality estimation of AHP-oriented software products in academic departments do not seem to be independent from each other. Therefore, it seems to be an appropriate option to use ANP for our evaluation.

As there is no ANP-based evaluation of AHP-software to be found in the literature, which might support our software evaluation problem, an own tailor-made process of selecting AHP software, adequate to the research and teaching requirements and demands of a Management Science Department was developed.

3. DANP-evaluation of AHP-DSS

3.1. Selection of AHP-DSS

Owing to the wide range of software solutions supporting AHP application [17], the selection of software has to be conducted first, whereby initially all products are relevant which are freeware or are ensuring a free trial access for evaluation purposes (at least for a limited time period). Regarding the fact that Questfox is a Software as a Service (SaaS) product which has to be accessed by users via a web browser, this solution was excluded from our list of potential evaluation products. For a mutual evaluation of software solutions, it is important to determine a manageable number of evaluation alternatives. By random selection, MakeItRational, Qualica Decision Suite, SelectPro, easy-mind and SuperDecisions have been determined for our evaluation.

3.2. Evaluation criteria derived from international standard norm

Evaluating software demands to consider commonly understood quality criteria. The first part of the international standard norm ISO/IEC 25010-1:2011 “Systems and software engineering. Systems and software Quality Requirements and Evaluation (SQuaRE). System and software quality models” provides a first foundation for our software evaluation. Scope of this international standard is the definition of a product quality model composed of eight characteristics that relate to static properties of software and dynamic properties of the computer system.

With respect to the evaluation of AHP software products and the underlying standard norm, the focus of interest lies on the product quality from a user’s point of view. The quality criteria provided by ISO/IEC are postulated for software evaluations in general. But on account of a certain lack of concreteness with respect to AHP software products, these criteria had to be customized for the teaching and research requirements of the members of the Management Science Department. Thereby, this standard norm is used as a starting point to develop relevant criteria for the evaluation of AHP software products. The results of the transition process are shown in Figure 1.

Figure 1.

Transition process for criteria identification.

Partial pretests suggested that all software alternatives worked efficiently and secure enough for the department’s purposes. Furthermore, there was no need for us to modify the software. Therefore, the criteria performance efficiency, security and maintainability could be disregarded in our model in order to avoid an unnecessarily high level of complexity. Moreover, we added costs and advanced functions as clusters to our model. Within costs, the initial investment was regarded exclusively. The criteria within advanced functions are AHP/ANP-specific as they are derived from special requirements towards Group decision making [19–21], Transparency, Benefits-Opportunities-Costs-Risks (BOCR) modelling [22, 23] and more general AHP advancements. Group decision modelling is an important characteristic as decisions with uncertain attributes often have to be solved in a group context to achieve a broader base of intersubjectivation or objectivity. Furthermore, transparency is necessary for performing sensitivity analysis as well as for the interpretation of the results. In this context, the possibility of BOCR modelling is inevitable for structuring complex strategic decision settings. Apart from structuring, it is moreover possible to cope with scale incommensurability [22, 24–26]. The possibilities of considering horizontal (inter-)dependencies (ANP-extension) or multiplicative AHP [27] are subsumed under AHP advancements.

3.3. DANP-evaluation framework

3.3.1. Application of DANP in literature

As (inter-)dependencies can be assumed between the derived criteria in Figure 1, an approach has to be used being able to cope with this kind of criteria structure. So, the ANP moves into focus and is therefore considered to be an adequate evaluation method.

In order to meet our group decision requirements, we additionally use the DEMATEL approach [28–30] for identifying criteria (inter-)dependencies within the ANP evaluation model. Regarding its frequency ranked in the literature [31] shortly beyond fuzzy set theory, DEMATEL belongs to the most common auxiliary tools of ANP—as such denoted by DANP.

In order to highlight the importance of DEMATEL in the field of the ANP literature, we analysed the bibliometric study of Kaspar [15], who used for his study the leading databases EBSCOhost Business Source® Complete, SciVerse® ScienceDirect and Thomson Reuters Web of Knowledge and thus exceeded other bibliometric studies on the ANP [32, 33] to achieve a maximum scope of the literature. The procedure covers three databases and a time horizon from 1998 to 2012 (database accesses for 1998–2011: July 16, 2012 and for 2012: January 28, 2013 [15]). In total, Kaspar found 4187 AHP publications and 613 ANP publications within the databases using the keywords “analytic hierarchy process” and “analytical hierarchy process”, respectively, “analytic network process” and “analytical network process” in titles, key words or abstracts [15]. About 52 publications dealt with DEMATEL [15].

Figure 2 [15] gives an overview of AHP and ANP publications from 1998 to 2012. Both methods show a clear upward trend in the timeline, especially rising with the last few years. Although there is a clear growth of ANP related publications, the comparison of total numbers of the publications points out that AHP seems to be more popular in research and practice. This might be due to a lack of software support for ANP and its more complex cognitive requirements. Thereby, evaluating and selecting adequate DSS is an important task to improve the chances of these MCDM methods to be accepted and implemented in a real multi-personal decision contexts. As existing evaluations [16, 18, 34] do not work with advanced approaches as the ANP and/or DEMATEL, and since the 52 publications using DEMATEL within an ANP procedure did not supply any evaluation of AHP software, a new study on this was motivated.

Figure 2.

Overview of AHP and ANP publications (1998–2012).

3.3.2. Formal description of DEMATEL for ANP

In order to achieve a better understanding of our evaluation, we start with a short formal explanation of the DEMATEL approach [28]. The initial step within DEMATEL is the determination of the influence values ${\propto }_{\text{ij}}^{\text{r}}$ for each decision maker DM_r (r = 1, …, R) for all criteria elements (software alternatives are excluded). The influence values are on an ordinal scale from “0” (no influence) to “4” (extreme strong influence). For synthesizing the group, the next step is to calculate the average matrix ${\displaystyle \widehat{\text{F}}}$ :

$\hat{\text{F}}=\frac{1}{\text{R}}\sum _{\text{r}=1}^{\text{R}}{\left[{\propto }_{\text{i}.\text{j}}^{\text{r}}\right]}_{\text{m}\times \text{m}}.$ E1

which has to be normalized by scalar $â$, in the following way:

${{\displaystyle \widehat{\text{F}}}}_{\text{norm }}=\frac{{\displaystyle \widehat{\text{F}}}}{\text{â}}$ E2

$\text{â}=\max ({\max }_{1\le \text{i}\le \text{m}}{\sum }_{\text{j=1}}^{\text{m}}{\propto }_{\text{i}.\text{j}},{\max }_{1\le \text{j}\le \text{m}}{\sum }_{\text{i=1}}^{\text{m}}{\propto }_{\text{i}.\text{j}}).$ E3

As a next step, the total-influence matrix ${\displaystyle \widehat{\text{T}}}$ is to be calculated as follows in order to consider the indirect effects:

${\text{where lim}}_{\text{k}\to \infty }{{\displaystyle \widehat{\text{F}}}}_{\text{norm}}{}^{\text{k}} = {[{\displaystyle \widehat{0}}]}_{\text{m}\times \text{m}}$ E4

${\text{and lim}}_{\text{k}\to \mathrm{\infty }}(\hat{\text{E}}{\mathrm{+}\hat{\text{F}}}_{\text{norm}}+{\hat{\text{F}}}_{\text{norm}}{}^2+\dots +{\mathrm{}\hat{\text{F}}}_{\text{norm}}{}^{\text{k}})={(\hat{\text{E}}-{\hat{\text{F}}}_{\text{norm}})}^{-1},$ E5

$\begin{array}{ccc}\text{and}\hat{\text{T}}=\hat{\text{E}}-{\hat{\text{F}}}_{\text{norm}}{}^{\text{k}}& \text{or}& \hat{\text{T}}={\hat{\text{F}}}_{\text{norm}}{(\hat{\text{E}}-{\hat{\text{F}}}_{\text{norm}})}^{-1}.\end{array}$ E6

Having constructed ${\displaystyle \widehat{\text{T}}}$ , it is necessary to set a threshold value of the required influence level. Only some elements, of which the influence level in matrix F is higher than the threshold value, can be chosen and converted into the impact-digraph-map respectively into the ANP network model [28, 29].

According to the setting of a threshold value, there is no fixed determination rule. It can be decided by experts through discussions [35] or brainstorming [28]. Another possibility would be the exogenous determination by a meta-decision maker. Regardless of the variant of determination, the threshold value should not be too high (too low), as only a few (too many) dependencies would be considered within the ANP model [36].

3.3.3. Construction of the evaluation network model: clustering and dependencies

Before identifying the dependencies within the model, all evaluation criteria have to be assigned to clusters. To cope with scale incommensurability [26], our evaluation model fundamentally consists of the two subnets Benefits (with performance quality criteria which should be assessed by ordinal judgements) and Costs (priorities are derived from monetary values). The overlapping alternatives-cluster consisting of the five AHP-DSS A₁ to A₅ is an integral part of both subnets. The subnet costs contain only one cluster with the element initial investment. The subnet benefits further contains the clusters usability (US), compatibility (CO), functional suitability (FS), reliability (RE) and advanced functions (AF) which are further explained in the evaluation process. All suggested characteristics have to be individually specified to ensure the principle of preferential independence.

Having derived and clustered the relevant criteria and (inter-)dependencies with DEMATEL, the evaluation model is created in a network structure. In order to reach a result representing the different needs of the department’s members, the estimations should be the output of a multi-personal estimation, reconsidering different personal requirements with weighed and assessed estimations thus caring for a higher level of intersubjectivity/objectivity. The members of this group—an advanced Master-student (Tutor), academic lecturers and professors—had graduations in Management Science/Business, Informatics and Mathematics, and rated the strength of the dependencies between all model elements within benefits subnet. For assessing the strength of model influences the DEMATEL standard scale with “0 = no interdependency”, “1 = low influence”, “2 = medium influence”, “3 = high influence” and “4 = very high influence” is used. The results in form of ${\displaystyle \widehat{\text{T}}}$ are shown in Table 1 (see the Appendix for individual direct relation matrices ${\left[{\propto }_{\text{i}.\text{j}}^{\text{r}}\right]}_{\text{m}\times \text{m}}$ ), whereby the number in each cell indicates the influence of the row element on the column element. Following Ou Yang [28] we interpret an influence as essential and considerable if it exceeds a threshold value of 0.1. Such an exceedance characterises an influence as significant and therefore to be subsequently considered in the model of (inter-)dependencies, whereas influences not surpassing this threshold are to be neglected in the model as insignificant. Influences regarded as significant and therefore to be considered are highlighted in the Table by bold numbers. Thereupon the influences are transferred as (inter-)dependencies to the evaluation model to complete the network structure. For improving the overall view of the model, within the ANP approach, the (inter-)dependencies are aggregated by clusters and then visualized by directional arrows. Arrows with double tips are used for representing interdependencies. Figure 3 shows the final evaluation model for AHP software.

	FS1	FS2	US1	US2	US3	US4	US5	RE1	RE2	RE3	CO1	CO2	AF1	AF2	AF3	AF4
FS1	0.0276	0.0525	0.0743	0.1267	0.1620	0.0453	0.0279	0.0600	0.0485	0.0415	0.0830	0.0385	0.0882	0.1154	0.0308	0.1066
FS2	0.0130	0.0205	0.0240	0.0990	0.1013	0.0456	0.0365	0.1271	0.1128	0.0304	0.0291	0.0036	0.0100	0.0676	0.0210	0.0126
US1	0.0014	0.0025	0.0050	0.0347	0.0318	0.0095	0.0118	0.0018	0.0013	0.0003	0.0007	0.0006	0.0019	0.0362	0.0006	0.0015
US2	0.0025	0.0024	0.0406	0.0203	0.0753	0.0016	0.0383	0.0026	0.0009	0.0004	0.0016	0.0014	0.0026	0.0387	0.0011	0.0022
US3	0.0200	0.0046	0.0377	0.1575	0.0255	0.0031	0.0494	0.0285	0.0049	0.0029	0.0192	0.0182	0.0051	0.0292	0.0027	0.0038
US4	0.0083	0.0532	0.0104	0.0870	0.0940	0.0095	0.0102	0.0669	0.0884	0.0094	0.0043	0.0025	0.0216	0.0389	0.0028	0.0027
US5	0.0164	0.0054	0.0741	0.1602	0.1079	0.0036	0.0129	0.0060	0.0034	0.0014	0.0034	0.0024	0.0290	0.0571	0.0179	0.0203
RE1	0.0367	0.0806	0.0142	0.0746	0.1070	0.0461	0.0196	0.0300	0.1267	0.0761	0.0080	0.0116	0.0252	0.0451	0.0203	0.0061
RE2	0.0340	0.0630	0.0121	0.0769	0.1080	0.0511	0.0112	0.1011	0.0244	0.0514	0.0145	0.0040	0.0240	0.0350	0.0043	0.0052
RE3	0.0211	0.0084	0.0056	0.0425	0.0581	0.0219	0.0052	0.0502	0.0501	0.0063	0.0032	0.0021	0.0044	0.0164	0.0018	0.0028
CO1	0.0246	0.0066	0.0436	0.0347	0.1020	0.0047	0.0160	0.0329	0.0230	0.0043	0.0045	0.0194	0.0296	0.0285	0.0187	0.0037
CO2	0.0228	0.0044	0.0327	0.0262	0.0733	0.0031	0.0058	0.0305	0.0061	0.0033	0.0362	0.0028	0.0205	0.0095	0.0185	0.0027
AF1	0.0911	0.0315	0.0701	0.0881	0.1300	0.0274	0.0305	0.0407	0.0379	0.0162	0.0103	0.0057	0.0136	0.1131	0.0052	0.0135
AF2	0.0179	0.0555	0.0723	0.1054	0.0863	0.0299	0.0356	0.0134	0.0114	0.0032	0.0046	0.0022	0.0377	0.0193	0.0107	0.0365
AF3	0.0808	0.0254	0.0637	0.0738	0.1023	0.0075	0.0185	0.0283	0.0256	0.0062	0.0088	0.0047	0.0177	0.0433	0.0038	0.0100
AF4	0.0908	0.0362	0.0659	0.0847	0.0926	0.0095	0.0195	0.0395	0.0370	0.0080	0.0180	0.0052	0.0278	0.0549	0.0047	0.0114

Table 1.

Total influence matrix with final model influences (threshold value = 0.1).

Figure 3.

Final DANP-evaluation model for AHP software.

3.4. Assessments and results

3.4.1. Preliminary remarks

Users’ individual requirements towards adequate/appropriate software solutions can vary strongly. Therefore, our aim is not primarily to determine a “best DSS” as a general recommendation from the point of view of the members of a Management Science Department. Instead, our evaluation focuses on a transparent confrontation with the 5 heterogeneous products displaying their dependencies and interdependencies within the network of our evaluation criteria.

At an expert workshop, the pairwise comparisons as for the software alternatives’ fulfilment of the quality criteria were performed by the authors’ mutual agreement to derive a consensus [37]. So, there was no necessity to aggregate the results with the support of a group decision rule. But within a greater department with more experts sharing the evaluation procedure such a rule might have made sense. To evaluate the alternate software products, we constructed a multi-criteria standard problem to be handled by the different DSS. The matrices representing the judgements on Saaty’s 1 (“equal importance”)-to-9 (“extreme importance”) scale [13] are listed below. In the tables, C.R. stands for consistency ratio. The more inconsistent the pairwise judgements, the higher the consistency ratio. Theory suggests that if the consistency ratio for the matrix is not smaller than 0.1, the ratios should be adjusted to make them more consistent. In our evaluation, there was no need for additional adjustments.

The local priorities of the criteria are shown at the bottom of each table, providing evidence of their importance. The priorities in our study are derived by Saaty’s principal eigenvalue method [6, 38, 39].

The next subsection deals with the direct assessments for alternatives’ clusters, followed by the presentation of subnets benefits’ unweighted supermatrix ${{\displaystyle \widehat{\text{S}}}}^{\text{U}}$ and the clustermatrix ${\displaystyle \widehat{\text{C}}}$ showing the indirect influences.

3.4.2. Assessments for alternatives cluster

Subsequently, the pairwise comparisons of the DSS alternatives are represented by clusters. Regarding the functional completeness (FS₁, see Table 2), all software alternatives except Qualica Decision Suite—which cannot handle different alternatives—are equipped with a large and detailed set of functions to handle problems by support of AHP, including dynamic sensitivity analysis, direct data entry and consistency calculation. With respect to the hierarchy to be modelled, the number of criteria, levels and alternatives are not limited in all software products. Highlighting distinctive features, it can be pointed out that MakeItRational contains a special alert feature which proposes steps for trouble-shooting when inconsistency reaches 0.1. SuperDecisions and SelectPro comparatively provide the best sensitivity analysis and the largest set of functions according to direct data entry possibilities. In addition, SuperDecisions provides a broader range of rating possibilities (direct priorities, graphically or numbers on 1–9 scale).

Functional suitability	easy-mind	MakeItRational	Qualica D S	SelectPro	SuperDecisions
FS1: Functional completeness
C.R. = 0.05793
easy-mind	1	1/2	7	1/3	1/3
Make ItRational	2	1	8	1/3	1/3
Qualica D S	1/7	1/8	1	1/8	1/9
SelectPro	3	3	8	1	1/2
SuperDecisions	3	3	9	2	1
Local priority	0.11872	0.16196	0.02774	0.29620	0.39538
FS2: Functional correctness
C.R. = 0.06412
easy-mind	1	1/5	7	1/4	1/3
MakeItRational	5	1	9	2	3
Qualica D S	1/7	1/9	1	1/8	1/8
SelectPro	4	1/2	8	1	2
SuperDecisions	3	1/3	8	1/2	1
Local priority	0.09452	0.42264	0.02738	0.27347	0.18199

Table 2.

Judgements of the alternatives I (FS).

Within all DSS, the provided functions work correctly (functional correctness, FS₂, see Table 2). Due to Qualica Decision Suite’s inability regarding to the handling of alternatives, it is not possible to achieve a final AHP calculation result. MakeItRational, SuperDecisions and SelectPro provide a large number of possibilities to show the results either in a graphical way or as data tables. Unfortunately, the given results of SuperDecisions and SelectPro are sometimes not exact in fourth or fifth decimal place (SuperDecisions sometimes curiously displays later on, e.g. 3.0003 instead of the original value of 3.0 inside the evaluation matrices). Furthermore, MakeItRational has slight advantages because of its advanced visualization possibilities for the results including alternatives, criteria and ranking comparisons as well as handling of local and global weights.

With respect to the appropriateness [7] of recognizability (US₁, see Table 3), MakeItRational and SuperDecisions provide the best information about the supported functions on their website as well as free trials equipped with the full set of functions, even if MakeItRational is slightly more detailed, whereas SuperDecisions needs registration to download the trial version. SelectPro offers a 30-day and fully functional demo-version without registration, but does not inform about the functions, while easy-mind’s provided information about the functions is not structured helpfully and partly overhauled. In addition, easy-mind’s trial is hardly limited in its use of functions. Qualica Decision Suite finally allows to download a 30-day trial with all features, but it is absolutely not evident, which functions the software offers or if it supports AHP at all.

Usability	easy-mind	MakeItRational	Qualica D S	SelectPro	SuperDecisions
US1: Appropriateness recognizability
C.R. = 0.04383
easy-mind	1	1/4	7	1/2	1/3
MakeItRational	4	1	9	3	2
Qualica D S	1/7	1/9	1	1/8	1(8
SelectPro	2	1/3	8	1	1/2
SuperDecisions	3	1/2	8	2	1
Local priority	0.11351	0.41896	0.02805	0.17265	0.26683
US2: Learnability
C.R. = 0.01875
easy-mind	1	1/5	3	1/3	3
MakeItRational	5	1	9	3	9
Qualica D S	1/3	1/9	1	1/5	1
SelectPro	3	1/3	5	1	5
SuperDecisions	1/3	1/9	1	1/5	1
Local priority	0.11737	0.53822	0.04817	0.24808	0.04817
US3: Operability
C.R. = 0.03498
easy-mind	1	1/4	4	1/4	4
MakeItRational	4	1	6	1	6
Qualica D S	1/4	1/6	1	1/6	1
SelectPro	4	1	6	1	6
SuperDecisions	1/4	1/6	1	1/6	1
Local priority	0.14386	0.37688	0.05119	0.37688	0.05119
US4: User error protection
C.R. = 0.00385
easy-mind	1	1/3	2	1/2	1
MakeItRational	3	1	5	2	3
Qualica D S	1/2	1/5	1	1/3	1/2
SelectPro	2	1/2	3	1	2
SuperDecisions	1	1/3	2	1/2	1
Local priority	0.13500	0.41428	0.07427	0.24145	0.13500
US5: User interface aesthetics
C.R. = 0.01131
easy-mind	1	1/5	1/5	1/8	1/3
MakeItRational	5	1	1	1/2	3
Qualica D S	5	1	1	1/2	3
SelectPro	8	2	2	1	5
SuperDecisions	3	1/3	1/3	1/5	1
Local priority	0.04249	0.22570	0.22570	0.41691	0.08920

Table 3.

Judgements of the alternatives II (US).

With regard to the criterion learnability (US₂, see Table 3), MakeItRational, SelectPro and easy-mind are much more intuitively to handle and more advanced in providing assistants, helping hints and tools, online guides and tutorials as well as examples. The other two software alternatives were more difficult to understand with respect to a convenient handling and had a less number of helpful tools.

Operability (US₃, see Table 3) was more complex in Qualica Decision Suite and SuperDecisions and a longer initiation period was needed to operate with these programs. The commands are sometimes hard to find and the next operating step is mostly not obvious. The other three programs are more intuitive in handling, they operate using step-by-step methods. Especially operating with MakeItRational and SelectPro is easily possible after a very short initiation period.

User error protection (US₄, see Table 3) is well-performed in almost all DSS alternatives, the differences are slight. Best hints, handling and protection from errors [7] are implemented in MakeItRational and SelectPro. Within MakeItRational, we did not manage to produce any errors. Owing to the non-step-by-step operating structure of SuperDecisions, errors may occur within the working process. Easy-mind has to be saved manually by the user after each operating step which produces errors if forgotten.

Regarding the user interface aesthetics (US₅, see Table 3), the operation interfaces of SelectPro, MakeItRational and Qualica Decision Suite are modern, appealing and clearly arranged. In addition, they provide a large set of optical adaptation possibilities according to the needs of the user, for example, the size of the windows, whereby SelectPro is notably more professional compared to the other two software alternatives. Although symbols and view are generally clear, the menues and total interface of easy-mind are a little amateurish, and it is not possible to make any adaptations. Finally, the menues and structure of SuperDecisions are a little too complex and adaptation possibilities are missing, too.

Maturity (RE₁, see Table 4) is good in MakeItRational, Qualica Decision Suite and SuperDecisions. In SelectPro occurred some errors using the export functions, while easy-mind often produced errors in criteria and alternative management as well as browser errors due to easy-mind’s nature of a web-based software product.

Reliability	easy-mind	MakeItRational	Qualica D S	SelectPro	SuperDecisions
RE1: Maturity
C.R. = 0.00443
easy-mind	1	1/6	1/6	1/3	1/6
MakeItRational	6	1	1	3	1
Qualica D S	6	1	1	3	1
SelectPro	3	1/3	1/3	1	1/3
SuperDecisions	6	1	1	3	1
Local priority	0.04393	0.28420	0.28420	0.10348	0.28420
RE2: Fault tolerance
C.R. = 0.00296
easy-mind	1	1/4	1/4	1/3	1/4
MakeItRational	4	1	3	3	2
Qualica D S	4	1/3	1	1	1/2
SelectPro	3	1/3	1	1	1/2
SuperDecisions	4	1	1	2	1
Local priority	0.06137	0.26469	0.26469	0.14457	0.26469
RE3: Recoverability
C.R. = 0.00000
easy-mind	1	4	4	4	4
MakeItRational	1/4	1	1	1	1
Qualica D S	1/4	1	1	1	1
SelectPro	1/4	1	1	1	1
SuperDecisions	1/4	1	1	1	1
Local priority	0.50000	0.12500	0.12500	0.12500	0.12500

Table 4.

Judgements of the alternatives III (RE).

Regarding fault tolerance (RE₂, see Table 4), all programs except easy-mind were robust and almost operating in a stable manner. Hints on errors and error handling were sometimes missing in SelectPro, whereas easy-mind was not able to catch most errors which led to a crash of the software.

The recoverability (RE₃, see Table 4) of data on error is best solved in easy-mind because the user is forced to save each operating step. In all the other software alternatives, data did not get lost if saved before manually by the user.

The interoperability (CO₁, see Table 5), especially the export of data, is very strong in Qualica Decision Suite, which supports the most important file types. MakeItRational and SelectPro provide satisfying import and export possibilities (e.g. jpg, doc and xls), although MakeItRational is a little more advanced supporting pdf, html and chart images. While SuperDecisions is only able to handle MS Excel-importable text files for the super, limit and cluster matrices, easy-mind provides no import or export functions at all.

Compatibility	easy-mind	MakeItRational Qualica D S	Qualica D S	SelectPro	SuperDecisions
CO1: Interoperability
C.R. = 0.05434
easy-mind	1	1/8	1/9	1/6	1/2
MakeItRational	8	1	1/3	3	6
Qualica D S	9	3	1	5	8
SelectPro	6	1/3	1/5	1	4
SuperDecisions	2	1/6	1/8	1/4	1
Local priority	0.03251	0.26810	0.51290	0.13744	0.04906
CO2: Installability
C.R. = 0.00296
easy-mind	1	1	3	3	2
MakeItRational	1	1	3	3	2
Qualica D S	1/3	1/3	1	1	1/2
SelectPro	1/3	1/3	1	1	1/2
SuperDecisions	1/2	1/2	2	2	1
Local priority	0.31328	0.31328	0.09857	0.09857	0.17630

Table 5.

Judgements of the alternatives IV (CO).

All alternatives run on Windows, which was the testing environment, but installability (CO₂, see Table 5) within other systems is not guaranteed for SelectPro and Qualica Decision Suite by the developer, whereas SuperDecisions runs on different versions of Windows, Mac, Ubuntu and Linux. Thereby, easy-mind has great advantage due to its nature as independent web-based product as well as MakeItRational which can run as desktop version or web-based in a web-browser with MS Silverlight.

Group decision-making (AF₁, see Table 6) is implemented in all DSS products except in SuperDecisions, but Qualica Decision Suite provides only mail questionnaires which have to be inserted by the moderator. The left three software alternatives support remote group decision making, whereby the number of users is only limited in easy-mind. SelectPro is overall the most professional in rating, calculating the mean and comparing the single user votes.

Advanced functions	easy-mind	MakeItRational	Qualica D S	SelectPro	SuperDecisions
AF1: Group decision making
C.R. = 0.04604
easy-mind	1	1/2	4	1/3	6
MakeItRational	2	1	5	1/2	7
Qualica D S	1/4	1/5	1	1/6	4
SelectPro	3	2	6	1	9
SuperDecisions	1/6	1/7	1/4	1/9	1
Local priority	0.18244	0.27821	0.07216	0.43475	0.03245
AF2: Transparency
C.R. = 0.01621
easy-mind	1	1/4	2	1/6	1
MakeItRational	4	1	5	1/2	5
Qualica D S	1/2	1/5	1	1/7	1/2
SelectPro	6	2	7	1	6
SuperDecisions	1	1/5	2	1/6	1
Local priority	0.08358	0.30371	0.05191	0.48022	0.08058
AF3: BOCR
C.R. = 0.00937
easy-mind	1	3	3	1	1/5
MakeItRational	1/3	1	1	1/3	1/9
Qualica D S	1/3	1	1	1/3	1/9
SelectPro	1	3	3	1	1/5
SuperDecisions	5	9	9	5	1
Local priority	0.14578	0.05389	0.05389	0.14578	0.60066
AF4: AHP advancements
C.R. = 0.00000
easy-mind	1	1	1	1	1/9
MakeItRational	1	1	1	1	1/9
Qualica D S	1	1	1	1	1/9
SelectPro	1	1	1	1	1/9
SuperDecisions	9	9	9	9	1
Local priority	0.07692	0.07692	0.07692	0.07692	0.69231

Table 6.

Judgements of the alternatives V (AF).

Transparency (AF₂, see Table 6) is strongest in SelectPro, which shows at every time and on every level the current results for each user, alternative, criterion, weight and priority as comparable data as well as graphically. Each alternative and criterion can be deselected at each time with automatically actualized results. For the criteria, this is possible in MakeItRational, too, which provides slightly less transparency to the user. SuperDecisions and easy-mind give at least a good overview about the partial results and values of the criteria and alternatives on each level, whereas Qualica Decision Suite shows its transparency only regarding the criteria.

Only SuperDecisions has implemented a real and good working (pre-structured) Benefits, Opportunities, Costs, Risks (BOCR) modelling (AF₃, see Table 6). SelectPro supports only cost-score-ratios, while easy-mind handles only direct cardinal entries for BOCR. MakeItRational and Qualica Decision Suite have no implemented BOCR support.

None of the software alternatives except SuperDecisions supports any AHP advancements (AF₄, see Table 6). However, SuperDecisions is the only software product which is able to calculate the results by ANP.

Regarding the costs (initial investment), relevant on account of the financial budget restrictions of the Management Science Department of a medium-sized university, SuperDecisions and easy-mind were the preferred DSS solutions. Users who understand themselves as researchers or educators can receive both products for free. As there was no ordinal assessment, the local priorities were derived by direct cardinal data entry of the cardinal information (see Table 7). Apart from different scaling levels, the criterion initial investment is directed negatively. So, the lowest priorities are assigned to the preferred DSS. This reversed ranking will be transformed in the subsequent synthetization process of the entire model.

Costs
Initial investment
C.R. = 0.00000	easy-mind	MakeItRational	Qualica D S	SelectPro	SuperDecisions
Local priority	0.00060	0.04008	0.90333	0.05539	0.00060

Table 7.

Judgements of the alternatives VI (Costs).

Owing to the qualitative expert judgements, all priorities are used now to construct ${{\displaystyle \widehat{\text{S}}}}^{\text{U}}$ (see Table 8). Due to the (inter-)dependencies determined by DEMATEL, several cluster comparisons had to be made.

	FS1	FS2	US1	US2	US3	US4	US5	RE1	RE2	RE3	CO1	CO2	AF1	AF2	AF3	AF4	A1	A2	A3	A4	A5
FS1	0.0000	0.0000	0.0000	1.0000	0.6667	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	1.0000	0.0000	1.0000	0.5000	0.5000	0.5000	0.5000	0.5000
FS2	0.0000	0.0000	0.0000	0.0000	0.3333	0.0000	0.0000	1.0000	1.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.5000	0.5000	0.5000	0.5000	0.5000
US1	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.1000	0.1000	0.1000	0.1000	0.1000
US2	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.2500	0.2500	0.2500	0.2500	0.2500
US3	0.0000	0.0000	0.0000	0.8333	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.2500	0.2500	0.2500	0.2500	0.2500
US4	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.1500	0.1500	0.1500	0.1500	0.1500
US5	0.0000	0.0000	0.0000	0.1667	1.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.2500	0.2500	0.2500	0.2500	0.2500
RE1	0.0000	0.0000	0.0000	0.0000	0.5000	0.0000	0.0000	0.0000	1.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.3333	0.3333	0.3333	0.3333	0.3333
RE2	0.0000	0.0000	0.0000	0.0000	0.5000	0.0000	0.0000	1.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.3333	0.3333	0.3333	0.3333	0.3333
RE3	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.3333	0.3333	0.3333	0.3333	0.3333
CO1	0.0000	0.0000	0.0000	0.0000	1.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.6667	0.6667	0.6667	0.6667	0.6667
CO2	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.3333	0.3333	0.3333	0.3333	0.3333
AF1	0.0000	0.0000	0.0000	0.0000	0.2000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	1.0000	0.0000	0.0000	0.2000	0.2000	0.2000	0.2000	0.2000
AF2	0.0000	0.0000	0.0000	1.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.2000	0.2000	0.2000	0.2000	0.2000
AF3	0.0000	0.0000	0.0000	0.0000	0.8000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.4000	0.4000	0.4000	0.4000	0.4000
AF4	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.2000	0.2000	0.2000	0.2000	0.2000
A1	0.1187	0.0945	0.1135	0.1174	0.1439	0.1350	0.0425	0.0439	0.0614	0.5000	0.0325	0.3133	0.1824	0.0836	0.1458	0.0769	0.0000	0.0000	0.0000	0.0000	0.0000
A2	0.1620	0.4226	0.4190	0.5382	0.3769	0.4143	0.2257	0.2842	0.2647	0.1250	0.2681	0.3133	0.2782	0.3037	0.0539	0.0769	0.0000	0.0000	0.0000	0.0000	0.0000
A3	0.0277	0.0274	0.0280	0.0482	0.0512	0.0743	0.2257	0.2842	0.2647	0.1250	0.5129	0.0986	0.0722	0.0519	0.0539	0.0769	0.0000	0.0000	0.0000	0.0000	0.0000
A4	0.2962	0.2735	0.1726	0.2481	0.3769	0.2414	0.4169	0.1035	0.1446	0.1250	0.1374	0.0986	0.4347	0.4802	0.1458	0.0769	0.0000	0.0000	0.0000	0.0000	0.0000
A5	0.3954	0.1820	0.2668	0.0482	0.0512	0.1350	0.0892	0.2842	0.2647	0.1250	0.0491	0.1763	0.0325	0.0806	0.6007	0.6923	0.0000	0.0000	0.0000	0.0000	0.0000

Table 8.

Unweighted supermatrix Ŝ^U subnet benefits.

Table 9 shows the arising cluster matrix ${\displaystyle \widehat{\text{C}}}$ of the evaluation.

	Functional suitability (FS)	Usability (US)	Reliability (RE)	Compatibility (CO)	Advanced Functions (AF)	Alternatives (A)
Functional suitability (FS)	0.0000	0.1667	0.5190	0.0000	0.3333	0.0000
Usability (US)	0.0000	0.1667	0.0000	0.0000	0.0000	0.0000
Reliability (RE)	0.0000	0.1667	0.1775	0.0000	0.0000	0.0000
Compatibility (CO)	0.0000	0.1667	0.0000	0.0000	0.0000	0.0000
Advanced functions (AF)	0.0000	0.1667	0.0000	0.0000	0.3333	0.0000
Alternatives (A)	0.0000	0.1667	0.3035	1.0000	0.3333	0.0000

Table 9.

Cluster matrix ( ${\displaystyle \widehat{C}}$ ) subnet benefits.

3.4.3. Final results

For deriving the final priorities, as a first step the weighted supermatrix ( ${{\displaystyle \widehat{\text{S}}}}^{\text{W}}$ ) is generated:

${{\displaystyle \widehat{\text{S}}}}^{\text{W}}={{\displaystyle \widehat{\text{S}}}}^{\text{U}}\times {\displaystyle \widehat{\text{C}}}$ E7

Thus, ${\hat{\text{S}}}^{{\text{W}}^{\text{k}}}(\text{k}=1,2,\dots ,\mathrm{\infty })$ can be raised, until a converging column-stochastic matrix, the limit matrix ( ${{\displaystyle \widehat{\text{S}}}}^{\text{L}}$ ) is reached.

The final synthesized (additive probabilistic variant) ranking of the software products is shown in Figure 4. Thereby, the overall control criteria weighting for subnet benefits was set to 0.8 and to 0.2 for subnet costs (preferential value 4 on standard scale).

Figure 4.

Synthesized evaluation results (global priorities of the software products).

Owing to our subjective judgements, MakeItRational was found to be the preferred software alternative for the purposes of an academic Management Science Department, followed by SuperDecisions, easy-mind and SelectPro. The results show that there is a small distance regarding the level of performance. Regarding the other quality criteria, the differences between these programs are not extreme, but noticeable. Therefore, different rankings could result, if members of other academic departments or of other types of organizations with deviating targets, requirements, preferences and size would have evaluated the alternative software solutions. So, there may exist contexts, in which another software product than MakeItRational would fit better to the needs of the users.

MakeItRational, easy-mind and SelectPro, e.g. are convincing due to their very intuitive handling and step-by-step operating methods. The commands are obvious to find and easy to understand, mostly supported by helping functions or assistants. In general, the initiation period to operate with these programs is very short. Among all software products, MakeItRational is the most intuitive and the less complex. Besides, it provides more visualization and export possibilities and has the best error protection.

But especially when BOCR modelling or ANP is needed within the decision process, SuperDecisions is the only alternative within which these functions are implemented. Additionally, it offers more possibilities and functions than MakeItRational that go beyond the pure AHP application. But, this charges at learnability and operability. Furthermore, this product provides no group decision-making support, which is handled best and most detailed by SelectPro, being in this respect a good alternative to MakeItRational. It is the most professional in rating, calculating the mean and comparing the single votes. Besides, it scores by its transparency, showing current results at every time and on every level as comparable data as well as graphically.

4. ANP-based evaluation assisted by parallel and distributed computing

The described procedure comprised the structuring of software quality criteria and an evaluation of alternative AHP-supporting software products in the multi-personnel framework of a Management Science Department of a medium-sized university. This proceeding delivers an object of reference to solve such structuring and evaluation problems in a modified situation by assistance of parallel and/or distributed computing architecture [7]. If the number of experts, whose requirements towards alternative software products diverge, essentially enhances (compared with the state of affairs in the aforementioned department) and/or if the complexity of the network structure relevant for the evaluation increases considerably, such a computing architecture would be advantageous.

Problems to be solved on a strategic decision level with a demand for scientific computing might attain a degree of complexity that distributed computing architectures are to be recommended. The more complex the ANP-network structure, the more the modelled problem delivers connecting factors for such architectures. With its possibilities to intensify the “interaction” among different criteria [7, 40], their interlacing can be represented within and between the ANP-clusters more clearly. Thus, distributed computing would help to cope with increasing complexity of multi-criteria-decision relevant network structures.

The higher the number of experts and the variety of their requirements for software quality, the more advantageous would be a parallel computing [7] which enables faster computational results [40]. Such computing architectures can support learning processes among the members of an expert group which evaluate the quality of alternative software products simultaneously within the framework of a multi-personnel, interactive process.

5. Conclusion

MCDM-DSS is an important tool aid for solving complex strategic decision problems as, e.g. arising in a Management Science Department of a medium-sized university. Such a support has to suffice the heterogeneous teaching and research tasks of different persons in different functions with deviating experiences, requirements and preferences. For these tasks and the inhered strategic decisions, AHP and ANP are suitable decision support methods. Problems on a standard level of complexity should be solved by AHP, whereby an increasing connectivity induces the application of ANP. Both approaches are subject to a growing importance. At this time, AHP is relatively more important than ANP in the literature due to less sophisticated mathematical calculations but also to a longer existence of the method. A vast number of strategic decision problems can be handled with AHP. Therefore, an adequate DSS is necessary for ensuring mathematical correct method application as well as to bring forward the application of this method.

Owing to the great variety of AHP-DSS, the aim of this paper was a transparent evaluation of five heterogeneous products from the point of view of the members of a Management Science Department. In this context, it was not the aim to give a generalized recommendation for one of these products, but to highlight the distinctive differences and special features of the evaluated products. Thereby, criteria have been derived from ISO/IEC norm and used. As the evaluation was considered as a problem with a higher complexity (connectivity), the ANP was used. In order to integrate the specifically inclined states of knowledge of different department members and to ensure a higher degree of inter-subjectivity, five members of the academic staff with different teaching and research experiences and functions estimated (inter-)dependencies between criteria of software quality. Then, a rating of randomly selected software products as for the fulfilment of the quality criteria took place. To improve the ANP modelling regarding the identification of (inter-)dependencies as well as to meet the requirements of the group members, DEMATEL as the second most important ANP auxiliary tool was added to the evaluation framework. With a combination of DEMATEL and ANP (DANP), a solid framework for the multi-personnel evaluation has been established. Against the backdrop of a certain need of AHP-DSS and a certain lack of adequate software evaluations, the application of DANP to supply the need was pointed out.

It has become clear that the development of further ANP-DSS products can be advised, as well as an integration of DEMATEL into the DSS of AHP and ANP. Furthermore, there is a need for case studies in the field of DEMATEL combined with AHP and ANP which can further clarify and highlight the potential of such a combination and facilitate its usage in practice. The more experts with diverging software quality requirements are sharing the structuring and evaluating process, the more advantageous it would be to assist the procedure by parallel computing architectures. And with increasing complexity of the quality criteria’s network, the development of such a structure by an expert group will be more efficient if supported by distributed computing architectures.

Acknowledgments

Dr. Stefanie Schinke and Dipl.-Kfm. Fabian Burrey, M.Sc., are thanked for their participation in the group evaluation procedure.