Utilizing Surrogate Numbers for Probability Elicitation

Abstract

Comparatively few of the vast number of suggested decision-analytical methods have been widely spread in actual practice. The majority of those methods call for exact and accurate numbers as input, which could be one of several reasons for this lack of actual use; people frequently seem to be unfamiliar with, or reluctant to express those, in a sense, “true” values required. Many alternative methods to resolve this complication have been suggested over the years, including procedures for dealing with incomplete information. One way, which has proliferated for a while, is to introduce so-called surrogate numbers in the form of ordinal ranking methods for multi-criteria weights. In this chapter, we show how those can be adapted for use in probability elicitation. Furthermore, when decision-makers possess more information regarding the relative strengths of probabilities, that is, some form of cardinality, the input information to ordinal methods is sometimes too restricted. Therefore, we suggest a testing methodology and analyze the relevance of a set of cardinal ordering methods in addition to the ordinal ones.

1. Introduction

Elicitation of subjective beliefs has been applied in numerous areas, such as game and decision theory [1, 2], agriculture [3], statistics [4], and various disciplines of economics [5, 6, 7], and many methods have been designed for elicitation purposes [8, 9, 10]. It is in these contexts often assumed that at least experts in the various areas are capable of acting rationally and can provide reliable information so long as they are eliciting probabilities sensitively. Those assumptions are contrary to the fact that even expert estimates may differ significantly from the true probabilities and that there are still no universally accepted methods of probability elicitation available. The process of eliciting adequate quantitative information is one of the substantial challenges within decision analysis [11, 12].

In a classical framework (cf., e.g., [13]), numerical probabilities are assigned to the different events in tree representations of decision problems. The assignments are made after the set of parameters, whose values need to be elicited and whom to choose for providing those estimates, have been identified. Domain experts could, for example, be asked to express their beliefs about the likelihood of a particular event in probabilistic form. Such beliefs cannot be measured objectively, and neither should they be judged in such a manner. Besides, the success of an elicitation process depends on how well a representation of the present subjective opinions can be constructed rather than on some set of objectively true values [14, 15, 16].

Methods for eliciting utilities and probabilities have been thoroughly investigated, resulting in a large number of recommendations and handbooks on the subject. Procedures range from using direct elicitation, gamble, and lottery techniques, as well as more elaborate methods to reduce biases, aversions, and a multitude of other causes of errors while producing as reliable estimates as possible (c.f., e.g., [17, 18, 19, 20, 21, 22]). Here, it is generally assumed that procedures for elicitation should give rise to adequate preference orders, but this assumption is nevertheless often violated in empirical studies (c.f., e.g., [23, 24, 25]). A multitude of methods for ordinal rankings or interval approaches have been suggested to provide more realistic models. The goal is to be able to utilize the information the decision-makers can supply without forcing them to express unrealistic, misleading, or meaningless statements.

To elicit probabilities, methods based on ordinal rankings already constructed for obtaining weights for multi-criteria decision analysis (MCDA) (see [26, 27, 28]) can be adopted. The fundamental idea is that ordinal information that stems from importance can be converted to a set of normalized surrogate numbers, the values of which are consistent with the elicited ordinal rankings. The actual conversion can be done using a variety of methods, such as rank sum (RS) and rank reciprocal (RR) [29], and centroid-based methods (ROC) [30]; originally used for handling criteria weights in MCDA. However, even in MCDA, the use of only ordinal information is sometimes perceived as being too vague or imprecise, resulting in a lack of confidence in the alternatives’ final values or a too large class of non-dominated alternatives.

In this chapter, a set of methods that allow for more expressive information as input when eliciting probabilities, while maintaining the relative correctness and simplicity of ordinal ranking procedures, is proposed. In the following section, we compare and discuss a set of significant features, including correctness and relevance, of various extensions to some existing ranking methods. Following a brief recapitulation and adaptation of some ordinal ranking methods in the following section, we continue with cardinal ranking methods and discuss a set of appealing candidates for probability elicitation. Using simulations, we investigate some properties of the treated methods and conclude with pointing out, according to the results, a particularly attractive method for eliciting probabilities.

2. Ordinal ranking methods in MCDA

Different elicitation formalisms by which a decision-maker can express preferences in MCDA decision situations have been proposed. Such formalisms are sometimes based on scoring points, as in point allocation (PA) or direct rating (DR) methods. In PA, the decision-maker is given some number of points, for example, 100, to distribute over a set of criteria or consequences, depending on the type of decision [31]. Hence, for $\mathit{N}$ criteria or consequences, there are $\left(\mathit{N}\mathit{-}\mathbf{1}\right)$ degrees of freedom (DoF). Direct rating methods, on the other hand, put no limit on the number of points to be allocated.¹ The decision-maker allocates as many points as desired and the points are subsequently normalized. Thus, in DR, there are $\mathit{N}$ degrees of freedom for $\mathit{N}$ criteria. Regardless of elicitation method, the assumption is that all elicitations are made relative to a distribution held by the decision-maker.²

Surrogate methods are utilized in [26, 27, 28] and many others for handling such problems. Regardless of method, however, a key property must be to retain as much information as possible in the surrogate numbers, yet accommodating for the various constraints required by certain types of values.

Stillwell et al. [29] compare a set of different methods for eliciting surrogate numbers from ordinal rankings alone, based on the idea of maximizing the power to discriminate between values. Among those are rank sum and rank reciprocal, for which surrogate weights are derived solely from the rank order of the attributes. Take a simplex ${\mathit{S}}_{\mathit{w}}$ generated by ${\mathit{w}}_{\mathbf{1}}\mathit{>}{\mathit{w}}_{\mathbf{2}}\mathit{>}\mathit{\cdots }\mathit{>}{\mathit{w}}_{\mathit{N}}$, $\mathit{\sum }{\mathit{w}}_{\mathit{i}}\mathit{=}\mathbf{1}$ and $\mathbf{0}\mathit{\le }{\mathit{w}}_{\mathit{i}}$.³ Assign an ordinal number to each item ranked, starting with the highest ranked item as number 1. Denote the ranking number $\mathit{i}$ among $\mathit{N}$ items to rank. Then the rank sum (RS) surrogates for all $\mathit{i}\mathit{=}\mathbf{1}\mathit{,}\dots \mathit{,}\mathit{N}$ are defined by

$${\mathit{w}}_{\mathit{i}}^{\mathrm{RS}}=\frac{\mathit{N}+\mathbf{1}-\mathit{i}}{{\sum }_{\mathit{j}=\mathbf{1}}^{\mathit{N}}\left(\mathit{N}+\mathbf{1}-\mathit{j}\right)}.$$E1

The surrogate numbers produced by the rank reciprocal (RR) method are, as the name implies, based on the reciprocal of the rank of each item. Let the number 1 corresponds to the highest ranked item, the number 2 to the second highest ranked item, and so on. Then for the $\mathit{i}$:th ranked item, the RR surrogate, given a total of $\mathit{N}$ items, are obtained by

$${\mathit{w}}_{\mathit{i}}^{\mathrm{RR}}\mathbf{=}\frac{\raisebox{1ex}{$\mathbf{1}$}\!\left/ \!\raisebox{-1ex}{$\mathit{i}$}\right.}{{\sum }_{\mathit{j}\mathbf{=}\mathbf{1}}^{\mathit{N}}\raisebox{1ex}{$\mathbf{1}$}\!\left/ \!\raisebox{-1ex}{$\mathit{j}$}\right.}\mathbf{.}$$E2

A decade later, Barron [30] suggested a method based on vertices of the simplex of the feasible space. The rank order centroid (ROC) weights are the centroid vector components of the simplex ${\mathit{S}}_{\mathit{w}}$. The weights then become the centroid (mass point) of ${\mathit{S}}_{\mathit{w}}$. The ROC weights for the ranking number $\mathit{i}$ among $\mathit{N}$ items to rank are given by

$${\mathit{w}}_{\mathit{i}}^{\mathrm{ROC}}\mathbf{=}\frac{\mathbf{1}}{\mathit{N}}\sum _{\mathit{j}\mathbf{=}\mathit{i}}^{\mathit{N}}\raisebox{1ex}{$\mathbf{1}$}\!\left/ \!\raisebox{-1ex}{$\mathit{j}$}\right.\mathbf{.}$$E3

However, RS, RR, and ROC perform well only for specific assumptions on decision-maker behavior. If we assume that the decision-maker stores its preferences in a way similar to a given point sum, considering normalization, there are ($\mathit{N}\mathit{-}\mathbf{1})$ degrees of freedom (DoF) for $\mathit{N}$ items. On the other hand, if we assume that the decision-maker stores its preferences in a way that puts no limit to the total number of points (or mass) allocated, and the normalization is made afterward, then there are $\mathit{N}$ degrees of freedom for $\mathit{N}$ criteria. It remains an open question as to how a decision-maker perceives the nature of the basis of a particular preference order. Whether the linear dependence in $\left(\mathit{N}\mathit{-}\mathbf{1}\right)$ DoF models is accounted for, or if preference values are allocated with no particular limits in accordance with $\mathit{N}$ DoF models. Surrogate numbers obtained by RR and ROC models agree with a preference structure based on $\left(\mathit{N}\mathit{-}\mathbf{1}\right)$ degrees of freedom, while the RS model conforms to a preference structure based on $\mathit{N}$ degrees of freedom. Due to this apparent difference, a model, SR, in which features from both RS and RR were incorporated, was proposed by [34]. The SR method is an additive combination of the Sum and the Reciprocal functions as in

$${\mathit{w}}_{\mathit{i}}^{\mathrm{SR}}=\frac{\frac{\mathbf{1}}{\mathit{i}}+\frac{\mathit{N}+\mathbf{1}-\mathit{i}}{\mathit{N}}}{{\sum }_{\mathit{j}=\mathbf{1}}^{\mathit{N}}\frac{\raisebox{1ex}{$\mathbf{1}$}\!\left/ \!\raisebox{-1ex}{$\mathit{j}$}\right.\left(\mathit{N}+\mathbf{1}-\mathit{j}\right)}{\mathit{N}}}.$$E4

To exemplify the above, given any probability simplexes such as $\left({\mathit{p}}_{\mathbf{1}},{\mathit{p}}_{\mathbf{2}},{\mathit{p}}_{\mathbf{3}}\right)$ and $\left({\mathit{p}}_{\mathbf{1}}^{\prime },\dots ,{{\mathit{p}}_{\mathbf{6}}}^{\prime }\right)$ that satisfy the previously laid out assumptions, the various methods would assign to them numbers as in Tables 1 and 2.

Method	$p_1$	$p_2$	$p_3$
Rank sum (RS)	0.50	0.33	0.17
Rank reciprocal (RR)	0.55	0.27	0.18
Rank order centroid (ROC)	0.61	0.28	0.11
Sum reciprocal (SR)	0.52	0.30	0.17

Table 1.

Probabilities corresponding to the ordinal ranking, ${\mathrm{p}}_1>{\mathrm{p}}_2>{\mathrm{p}}_3$, rounded to two decimal places.

Method	${p_1}^{’}$	${p_2}^{’}$	${p_3}^{’}$	${p_4}^{’}$	${p_5}^{’}$	${p_6}^{’}$
Rank sum (RS)	0.29	0.24	0.19	0.14	0.10	0.05
Rank reciprocal (ROC)	0.41	0.20	0.14	0.10	0.08	0.07
Rank order centroid (ROC)	0.41	0.24	0.16	0.10	0.06	0.03
Sum reciprocal (SR)	0.34	0.22	0.17	0.13	0.09	0.06

Table 2.

Probabilities corresponding to the ordinal ranking, ${{\mathrm{p}}_1}^{\prime }>{{\mathrm{p}}_2}^{\prime }>{{\mathrm{p}}_3}^{\prime }>{{\mathrm{p}}_4}^{\prime }>{{\mathrm{p}}_5}^{\prime }>{{\mathrm{p}}_6}^{\prime }$, rounded to two decimal places.

In theory, the choice of elicitation method should preferably be based on the manner in which a decision-maker constructs a preference order; whether it is based on a $\left(\mathit{N}\mathit{-}\mathbf{1}\right)$ DoF or $\mathit{N}$ DoF model, or even a mixed model. However, since we cannot know the ratio of $\left(\mathit{N}\mathit{-}\mathbf{1}\right)$ DoF to $\mathit{N}$ DoF employed in the mind of the decision-maker, an elicitation method, to be robust, should work about equally well regardless. The SR method, assuming the principal features of both RS and RR, was found to be the method best accommodating for such a need.

In the following, the methods for weight elicitation within MCDA presented above will be augmented with information denoting the relative difference between adjacent items and modified to meet the requirements of probability elicitation. We adhere to a standard, one-level decision-tree model, in which each of $\mathit{M}$ alternatives has $\mathit{N}$ consequences. Hence, there are $\mathit{M}$ times $\mathit{N}$ consequences in total.

3. Cardinal ranking methods

Providing ordinal rankings puts fewer demands on decision-makers; they are, in a sense, effort saving. Furthermore, there are techniques such as those mentioned earlier for handling ordinal rankings with some success. For use in probability elicitation, the same ordering is asked of the decision-maker, but this time in reference to how probable events are as outcomes of a chosen alternative of action.⁴ Nevertheless, decision-makers might, in many cases, have more knowledge of the decision situation, even if the information still is not precise. For instance, cardinal probability relation information may implicitly exist, entailing that the surrogates may not really reflect what the decision-maker actually means by a particular ranking.

To improve the conformance of an ordinal ranking to the true subjective beliefs of a decision-maker, information such as the relative differences between adjacent items need to be accounted for. Given a ranking of some number of consequences, relative differences in probability comprise such information. Furthermore, methods that allow for such details can also handle probabilities considered equal, something which purely ordinal methods cannot.

The following notations together with the suggested interpretations are used to exemplify the proposed methods. The symbols denote the relative strength of the difference in probability between consequences.⁵

1. ${>}_0$ “equally probable”

2. ${>}_1$ “slightly more probable”

3. ${>}_2$ “more probable”

4. ${>}_3$ “much more probable”

To derive probabilities for $N$ consequences using cardinal methods, start by ranking the consequences in order of probability, giving the most probable consequence rank one, and the least probable consequence rank $N$. Let $s_j$ be the number of difference steps between probabilities $p_j$ and $p_{j+1}$, denoted by $p_j{>}_{s_j}{p}_{j+1}$. Setting $s_j=1$ is equivalent to $p_j>p_{j+1}$ as in the ordinal case. The sum, $Q$, of all difference steps for $N$ probabilities will then be ${\sum }_{i=1}^{N-1}{s}_i$, and for any probability $p_i$, $i\in \left\{2,\dots ,N\right\}$, the position on this scale of difference steps from 1 to $Q$ is defined by $t\left(i\right)=s_{i-1}+t\left(i-1\right)$, and $t\left(1\right)=1$.⁶

Modifying the above ordinal methods for probability elicitation can now be done without difficulty. The key is to normalize the values of the cardinal rank positions such that the higher positions result in the lower probabilities. In the cardinal version of RS (CRS), the probabilities should mirror the differences in cardinal rank positions. Hence, the CRS probabilities are given by

$$p_i^{\mathrm{CRS}}=\frac{Q+1-t\left(i\right)}{{\sum }_{j=1}^N\left(Q+1-t\left(j\right)\right)}$$E5

where $t\left(i\right)\in \left\{1,\dots ,Q\right\}$ are the cardinal rank positions derived from the cardinal differences provided by the decision-maker such that $t\left(i\right)\le t\left(j\right)$ if and only if $i<j$. The cardinal variation of rank reciprocal (CRR) is defined in a similar fashion, and the CRR probabilities are obtained by

$$p_i^{\mathrm{CRR}}=\frac{1/t\left(i\right)}{{\sum }_{j=1}^{N}1/t\left(j\right)}.$$E6

with the usual property that a higher probability is assigned to lower ranking numbers. ROC is generalized in the same way, and the corresponding strength rank order centroid (CRC) probabilities are obtained as

$$p_i^{\mathrm{CRC}}=\frac{{\sum }_{j=t\left(i\right)}^{Q}1/j}{{\sum }_{k=1}^Q\left({\sum }_{j=t\left(j\right)}^{Q}1/j\right)}.$$E7

Finally, generalizing SR is done in the same way and using the above equation, the corresponding cardinal SR (CSR) probabilities are obtained as

$$p_i^{\mathrm{CSR}}=\frac{\frac{1}{t\left(i\right)}+\frac{Q+1-t\left(i\right)}{Q}}{{\sum }_{j=1}^N\left(\frac{1}{t\left(j\right)}+\frac{Q+1-t\left(j\right)}{Q}\right)},$$E8

which is a generalization similar to the others. Thus, using the idea of cardinal steps, ordinal methods are easily transformed to their respective cardinal counterparts. See Table 3 for an example of probabilities correlated with a certain cardinal ranking.

3.1. Application to a decision problem

To show the merits of cardinal methods for elicitation of both probabilities and values, we present a decision on the choice of programming language, adapted from [35]. Due to resource constraints in a current Prolog implementation, the staff considered two options:

1. $A$: Rewriting the whole system in C.

2. $B$: Trying to find an implementation of Prolog that could handle the system.

After thoroughly discussing the pros and cons of the two options, they arrived at the following possible scenarios. For option $A$, the staff concluded that either ($c_1$) a prototype in C would be ready on time or ($c_2$) a prototype in C would be slightly delayed due to external circumstances. For option $B$, ($c_3$) a prototype in Prolog would be ready on time, ($c_4$) a prototype in Prolog would be slightly delayed due to external circumstances, or ($c_5$) only fractions of a prototype in Prolog would be ready on time.

Having considered the probabilities and the values of the consequences, the staff arrived at the following assumptions, where $p_i$ represents the probability of $c_i$, and $v_i$ represents the value of $c_i$. For option $A$, $p_1$ was thought to be at least 0.67, and consequently, $p_2$ would be at most 0.33. For option $B$, $p_5$ would lie somewhere between 0.40 and 0.90, hence the sum of $p_3$ and $p_4$ could range from 0.10 to 0.60.

With regard to value, $c_3$ was the most desirable with $c_4$ being slightly less so. Both $c_3$ and $c_4$ would be clearly better than $c_1$, which in turn was slightly better than $c_2$. By far, $c_5$ would be the worst consequence.

Applying a cardinal ranking to the above probabilities and values, the staff agreed on the following:

$$p_1{>}_3{p}_2$$

$$p_5{>}_3{p}_3{>}_0{p}_4$$

$$v_3{>}_1{v}_4{>}_2{v}_1{>}_1{v}_2{>}_3{v}_5$$

Using a purely ordinal ranking, one would end up with the following, less precise ordering:

$$p_1>p_2$$

$$p_5>p_3>p_4\mathit{\text{or}}{p}_5>p_4>p_3$$

$$v_3>v_4>v_1>v_2>v_5$$

Computing the values of the rankings about using SR and CSR yielded the results presented in Tables 4 and 5. Let ${\mathrm{E}}_o^m$ be the expected value of option $o$ given elicitation method $m$, we then have ${\mathrm{E}}_A^{\mathrm{SR}}=0.16$ and ${\mathrm{E}}_B^{\mathrm{SR}}=0.19$, resulting in option $B$ being the preferred alternative, while ${\mathrm{E}}_A^{\mathrm{CSR}}=0.16$ and ${\mathrm{E}}_B^{\mathrm{CSR}}=0.14$ resulting in the contrary.

Cardinal method	$p_1$	$p_2$	$p_3$	$p_4$	$p_5$	$p_6$
Rank sum (CRS)	0.29	0.23	0.19	0.13	0.13	0.03
Rank reciprocal (CRR)	0.49	0.16	0.12	0.08	0.08	0.05
Rank order centroid (CRC)	0.45	0.21	0.16	0.09	0.09	0.02
Sum reciprocal (CSR)	0.37	0.20	0.17	0.11	0.11	0.04

Table 3.

Probabilities corresponding to the ranking, $p_1{>}_2{p}_2{>}_1{p}_3{>}_2{p}_4{>}_0{p}_5{>}_3{p}_6$, rounded to two decimal places.

Method	$p_1$	$p_2$	$p_3$	$p_4$	$p_5$
SR	0.67	0.33	0.24	0.24	0.52
CSR	0.80	0.20	0.17	0.17	0.66

Table 4.

The probabilities of the consequences of the options of the decision on a computer programming language.

Method	$v_1$	$v_2$	$v_3$	$v_4$	$v_5$
SR	0.18	0.12	0.38	0.25	0.07
CSR	0.17	0.13	0.38	0.26	0.05

Table 5.

The values of the consequences of the options of the decision on computer programming language.

Possible alternatives in the case of ordinal rankings, such as strict uses of either of the rankings $p_5>p_3>p_4$ or $p_5>p_4>p_3$, do not affect the preference order of the options. However, a change from $p_5{>}_3{p}_3{>}_0{p}_4$ to $p_5{>}_2{p}_3{>}_1{p}_4$ when applying the cardinal method would alter the outcome, indicating the decision problem’s sensitivity to uncertainty, something which is not reflected when using a purely ordinal ranking of the consequences.

When treating the decision problem in the professional software program DecideIT [36], the probability values from the original assumptions can be used, resulting in the constraints:

1. $p_1\ge 2/3$ and $p_1+p_2=1$

2. $0.4\le p_5\le 0.9$ and $p_3+p_4+p_5=1$

which in turn can be entered directly into the program. The values are specified as in the cardinal case. Eventually, we end up with the decision tree in Figure 1.

Figure 1.

Modeling the decision problem of choosing a programming language in DecideIT.

Looking at the expected value graph in Figure 2, we note that although about 50% contraction is needed, it is arguable so that either the assumptions need to be revised or option $A$ should be the preferred one. Consequently, a cardinal ranking of the probabilities and the values seem to resonate better with a more detailed examination in which imprecise numbers are taken into account than a purely ordinal one.

Figure 2.

The expected value graph generated by DecideIT.

4. Assessing models for cardinal relations

Given that we have a set of cardinal methods as in the previous section, how can they be validated? For ordinal relations in MCDA, simulation studies similar to [27, 37, 38, 39] and others have become a kind of de facto standard. The simulations herein are based on the fundamental idea that a set of genuine, or “true,” probabilities, in fact, exists in the mind of the decision-maker. No elicitation method is capable of completely mirroring these probability values, but in the simulations, the potency of the probability ranking approaches is judged by comparing the “true” values to those elicited by the methods mentioned earlier.

The modeling caters to the two extremes of decision-makers’ mindsets outlined earlier in the way the decision problem vectors are randomly generated. Following an $N$ DoF model, a vector is generated, where the components are kept within [0%, 100%], and subsequently normalized, that is, a process with $N$ degrees of freedom. Details on this kind of simulation can be found, for example, in [40]. For an $\left(N-1\right)$ DoF model, the components are generated such that they sum to 100% already from the outset; that is, using a process of $\left(N-1\right)$ degrees of freedom. This simulation is based on a homogeneous $N$-variate Dirichlet distribution generator. Details on this kind of simulation can be found, for example, in [41].

The “true” probabilities in the minds of decision-makers might, of course, follow a distribution different from the ones used in this study, and there might eventually be models available to elicit values following those. Nonetheless, the crucial observation is that the validity and reliability of the results of the simulations are highly dependent on how the minds of decision-makers are modeled. Although the difference in the number of degrees of freedom is only one of several parameters related to cognitive behavior, it still offers a meaningful way of distinguishing between cognitive models.

4.1. Biases of simulation studies

The results of the simulations are highly dependent on the type of generator used. A generator corresponding to an $N$ DoF model is referred to as an N-generator. In the same manner, a generator corresponding to an $\left(N-1\right)$ DoF model is referred to as an $\left(N-1\right)$-generator. In applying the $N$-generator, probabilities elicited by the RS method outperforms those elicited by other methods. Not because the RS method is superior per se, but because it produces numbers at regular intervals. Likewise, ROC outperforms the other methods when an $\left(N-1\right)$-generator is used for the reason that the values elicited by the ROC method are similarly skewed toward the lower end of the interval.

In actual fact, it is impossible to determine whether decision-makers, in general (or even some), elicit values in particular accordance with $\left(N-1\right)$ or $N$ DoF representations of their knowledge. As a group, or as individuals, it is possible that they completely adhere to either one or that they follow an arbitrary mix of the two. Due to this uncertainty pertaining the cognitive processes of decision-makers, a rank ordering mechanism, to be robust, must elicit values that conform to both types of representations reasonably well. Therefore, to find the method that yields the most robust and efficient assignments, the evaluations in this study employ both types of generators and combinations of them.

4.2. Comparing the methods

To evaluate the validity of the RS, RR, and ROC methods for multi-criteria weight elicitation, Barron and Barrett [27] simulated a large set of “true” weights, using an $\left(N-1\right)$-generator. Based on those, they then produced a corresponding set of surrogate weights for each of the elicitation methods. As shown earlier, the generation procedure does have significant effects with regard to such a comparison. The set of “true” weights is dependent on how we model the minds of decision-makers. Barron and Barrett [27] presented a computer simulation consisting of four main steps, which for probability elicitation is modified as follows:⁷

4.2.1. Generation procedure

1. For a decision problem with $M$ alternatives, where each alternative can result in one of $N$ consequences, generate $M$ probability vectors, ${\mathbf{p}}_1,\dots ,{\mathbf{p}}_M$, in $N$ dimensions. These vectors contain the so-called true probabilities. Then, for each elicitation method ${\mu }^{\prime }$, produce vectors ${\mathbf{p}}_1^{{\mu }^{\prime }},\dots ,{\mathbf{p}}_M^{{\mu }^{\prime }}$, according to the order of the “true” probabilities.

2. After that, generate an $M\times N$ matrix of random numbers $v_{\mathit{ij}}$ corresponding to consequence $j$, of the $i$th alternative. These are the values of each consequence.

3. Let $p_{\mathit{ij}}^{\mu }$ be the probability obtained by method $\mu$ for consequence $j$ of alternative $i$ (where $\mu$ is either ${\mu }^{\prime }$ or “true”). For each method $\mu$ and each alternative $i$, calculate the expected value $E_i^{\mu }={\sum }_{j=1}^N{p}_{\mathit{ij}}^{\mu }{v}_{\mathit{ij}}$. Note the rank order of $E_i^{\mu }$ for each $\mu$, that is, the preference order of the alternatives for each method.

4. Lastly, determine if method $\mu$ resulted in the same most preferred alternative as “true.” If that is the case, then record a hit.

The abovementioned procedure (a simulation round) is repeated a large number of times, with the ratio of the number of hits to the total number of simulation rounds used as a measure of efficacy. In some MCDA studies, two additional measures of efficacy have been reported, namely the average value loss and the average proportion of the achieved maximum value range. These measures do not, however, add anything in particular in terms of value due to their strong correlation with hit ratio.

Using an $\left(N–1\right)$-generator MCDM simulation model over the simplex $S_x$, ROC outperforms the other two methods. But a study by Roberts and Goodwin [40] came up with a different result where RS performed better than ROC with RR in third place, by employing a different distribution generating function where a fixed number, say 100, is given to the most important criterion and the others are uniformly generated as U[0,100]. This $N$-generator is, of course, different from $\left(N–1\right)$-generators based on a Dirichlet distribution, and thus, their simulation study instead yields the result that RS outperforms ROC with RR in third place. As we see later, using surrogate weights for probability elicitation yields similar results.

4.3. Simulations of the cardinal surrogate numbers

This chapter focuses on the performance of cardinal surrogate weights in probability elicitation. The simulations included 20 different scenarios made up of 3, 6, 9, 12, and 15 alternatives, and for each of those cases, the number of consequences was set to 3, 6, 9, and 12, respectively. Each combination was simulated 10 times, and this was then repeated 10,000 times, totaling 2,000,000 decision problems. The probability vectors for the $N$ DoF model was generated using a standard round-robin random number generator with subsequently normalized numbers, and the probability vectors for the $N-1$ DoF model was generated using an $N$-variate Dirichlet distribution. The value vectors were generated on a uniform interval but left unscaled, analogous to MCDA studies such as [27]. Compared to alternative value distributions, there was no significant difference in the results.

A subset of the results, using a 50% combination of $N$ DoF and $N-1$ DoF models, is presented in the tables mentioned later. As described earlier, the numbers denote the ratio of the number of times the most preferred alternative according to method ${\mu }^{\prime }$ coincides with the most preferred alternative obtained from the “true” probabilities to the total number of simulation runs.⁸

The marginal utility of various levels of cardinal expressibility was evaluated by varying the levels of maximum cardinal differences in the simulations. The numbers, say $j$, in the “Symbols” column denotes the maximum $>$-index, $max\left(i\right)$, of the set of ${>}_i$ symbols, such that $max\left(i\right)=j$, for $j>0$. The results are obtained from the ordinal counterparts when $j=0$. Hence, $j=1$ implies $\left\{{>}_0,{>}_1\right\}$, $j=2$ means $\left\{{>}_0,{>}_1,{>}_2\right\}$, and so on. The actual results of the methods, the hit ratios, are given in percentages.⁹

Albeit there is a clear advantage of introducing the possibility of equivalence (${>}_0$) between probabilities, the results, in general, indicate diminishing returns with an increase of the maximum $>$-index, in particular, when the number of possible consequences for each alternative is high. For example, at nine or more consequences, cardinal probability elicitation with only ${>}_0$ and ${>}_1$ is to prefer for CRC. Also, the cognitive burden on decision-makers tends to increase with the granularity of the scale. Hence, a maximum $>$-index of 2 seems to provide enough expressional power for probability elicitation (Tables 6–12).

Combined DoF	Symbols	ROC/CRC	RS/CRS	RR/CRR	SR/CSR
Three consequences and three alternatives	0	86.9	86.8	86.7	87.0
	1	88.8	87.7	87.9	88.0
	2	89.3	90.1	89.4	90.3
	3	89.5	91.2	90.2	91.3

Table 6.

Comparing the methods using three consequences to each alternative and three alternatives.

Combined DoF	Symbols	ROC/CRC	RS/CRS	RR/CRR	SR/CSR
Three consequences and 15 alternatives	0	70.9	69.2	69.1	69.5
	1	73.9	71.2	70.7	71.0
	2	74.6	76.1	74.4	76.2
	3	74.6	77.7	76.5	78.6

Table 7.

Three consequences to each alternative and 15 alternatives.

Combined DoF	Symbols	ROC/CRC	RS/CRS	RR/CRR	SR/CSR
Six consequences and six alternatives	0	79.4	79.8	78.0	80.8
	1	83.5	80.8	81.0	82.4
	2	84.0	83.9	80.5	85.4
	3	83.8	85.3	79.3	86.4

Table 8.

Six consequences to each alternative and six alternatives.

Combined DoF	Symbols	ROC/CRC	RS/CRS	RR/CRR	SR/CSR
Six consequences and 12 alternatives	0	75.0	73.8	72.3	75.1
	1	78.6	75.6	75.3	77.2
	2	78.3	78.3	74.1	78.4
	3	77.6	79.6	72.9	80.9

Table 9.

Six consequences to each alternative and 12 alternatives.

Combined DoF	Symbols	ROC/CRC	RS/CRS	RR/CRR	SR/CSR
Nine consequences and nine alternatives	0	76.7	76.5	72.2	78.3
	1	81.0	77.2	76.8	79.8
	2	80.1	78.7	73.3	81.3
	3	78.5	79.7	70.2	81.6

Table 10.

Nine consequences to each alternative and nine alternatives.

Combined DoF	Symbols	ROC/CRC	RS/CRS	RR/CRR	SR/CSR
Twelve consequences and six alternatives	0	80.1	80.5	73.2	82.2
	1	83.6	79.9	78.3	82.5
	2	81.9	81.1	73.3	83.5
	3	80.2	81.7	70.0	83.3

Table 11.

Twelve consequences to each alternative and six alternatives.

Combined DoF	Symbols	ROC/CRC	RS/CRS	RR/CRR	SR/CSR
Twelve consequences and 12 alternatives	0	74.8	74.4	67.9	76.3
	1	78.5	73.8	72.3	76.8
	2	77.0	75.7	67.3	78.2
	3	74.8	77.2	62.0	78.7

Table 12.

Twelve consequences to each alternative and 12 alternatives.

The results show that cardinal methods markedly outperform the ordinal methods. Among the cardinal methods, it is CRC and CSR that provide the best outcomes, but the difference between these lies within this investigation’s margin of error.

5. Summary and conclusion

Elicitation methods available today are often either too cognitively demanding and require too much time and effort or unable to use the available information. The aim of this study was to offer decision-makers a set of methods for probability elicitation with a reasonable balance between simplicity and usability on the one hand and correctness and accuracy on the other hand. In particular, we strived to reduce the issues of applicability by loosening the requirements of precise information, while allowing for more details than what ordinal methods can handle. Also, the methods should be relatively robust and applicable to a wide range of decision problems.

By augmenting a set of ordinal elicitation methods, originally developed for weight elicitation within MCDA, with a notion of a difference between adjacent probabilities, we arrived at a collection of cardinal probability elicitation methods. The robustness of the methods was evaluated using a large number of simulated decision problems, where we also accommodated for two different cognitive models, based on the degree of freedom used during the ranking, including a mix of them. The results of the simulations point toward a significant improvement of cardinal methods over purely ordinal ones. In particular, due to the introduction of equality between probabilities. Among the cardinal methods, in particular, CSR and CRC seem to provide the most robust results. CSR generalizes SR from [42] by also taking the cardinal differences of the probabilities into account in a more straightforward way than, for example, [34].

More fine-grained expressions seem to produce diminishing returns when the number of consequences of each alternative becomes high. For alternatives with 12 consequences or less, a cardinal method with the set $\left\{{>}_0,{>}_1,{>}_2\right\}$ seems to supply the decision-maker with adequate options for producing quite reliable probability elicitations. For alternatives with more than 12 consequences, the reduced set $\left\{{>}_0,{>}_1\right\}$ seems to provide a sufficient granularity.

In conclusion, cardinal methods rather than ordinal ones should be preferred for eliciting probabilities when applicable. More specifically, CSR and CRC have been shown to produce surrogates, which outperform those of their competitors. They keep decision-makers from having to provide too much detail, something which has turned out to be difficult for decision-makers in general, while at the same time reducing the risk of neglecting available information.

Acknowledgments

This research was funded by the Swedish Research Council FORMAS, project number 2011-3313-20412-31, as well as by Strategic funds from the Swedish government within Information and Communications Technology (ICT)—The Next Generation.

Conflict of interest

All authors report no conflicts of interest relevant to this chapter.