The summary of upper and lower bounds for different hyperparameters, with p and for N = 50 iterations.
A key question in Bayesian analysis is the effect of the prior on the posterior, and how we can measure this effect. Will the posterior distributions derived with distinct priors become very similar if more and more data are gathered? It has been proved formally that, under certain regularity conditions, the impact of the prior is waning as the sample size increases. From a practical viewpoint it is more important to know what happens at finite sample size n. In this chapter, we shall explain how we tackle this crucial question from an innovative approach. To this end, we shall review some notions from probability theory such as the Wasserstein distance and the popular Stein’s method, and explain how we use these a priori unrelated concepts in order to measure the impact of priors. Examples will illustrate our findings, including conjugate priors and the Jeffreys prior.
- conjugate prior
- Jeffreys prior
- prior distribution
- posterior distribution
- Stein’s method
- Wasserstein distance
A key question in Bayesian analysis is the choice of the prior in a given situation. Numerous proposals and divergent opinions exist on this matter, but our aim is not to delve into a review or discussion, rather we want to provide the reader with a description of a useful new tool allowing him/her to make a decision. More precisely, we explain how to effectively measure the effect of the choice of a given prior on the resulting posterior. How much do two posteriors, derived from two distinct priors, differ? Providing a quantitative answer to this question is important as it also informs us about the ensuing inferential procedures. It has been proved formally in [1, 2] that, under certain regularity conditions, the impact of the prior is waning as the sample size increases. From a practical viewpoint it is however more interesting to know what happens at finite sample size n, and this is precisely the situation we are considering in this chapter.
Recently, [3, 4] have devised a novel tool to answer this question. They measure the Wasserstein distance between the posterior distributions based on two distinct priors at fixed sample size n. The Wasserstein (more precisely, Wasserstein-1) distance is defined as
for X1 and X2 random variables with respective distribution functions P1 and P2, and where stands for the class of Lipschitz-1 functions. It is a popular distance between two distributions, related to optimal transport and therefore also known as earth mover distance in computer science, see  for more information. The resulting distance thus gives us the desired measure of the difference between two posteriors. If one of the two priors is the flat uniform prior (leading to the posterior coinciding with the data likelihood), then this measure quantifies how much the other chosen prior has impacted on the outcome as compared to a data-only posterior. Now, the Wasserstein distance being mostly impossible to calculate exactly, it is necessary to obtain sharp upper and lower bounds, which will partially be achieved by using techniques from the so-called Stein method, a famous tool in probabilistic approximation theory. We opt for the Wasserstein metric instead of, e.g., the Kullback-Leibler divergence because of precisely its nice link with the Stein method, see .
The chapter is organized as follows. In Section 2 we provide the notations and terminology used throughout the paper, provide the reader with the minimal necessary background knowledge on the Stein method, and state the main result regarding the measure of the impact of priors. Then in Section 3 we illustrate how this new measure works in practice, by first working out a completely new example, namely priors for the scale parameter of the inverse gamma distribution, and second giving new insights into an example first treated in both [3, 4], namely priors for the success parameter in the binomial distribution.
2. The measure in its most general form
In this section we provide the reader with the general form of the new measure of the impact of the choice of prior distributions. Before doing so, we however first give a very brief overview on Stein’s method that is of independent interest.
2.1 Stein’s method in a nutshell
Stein’s method is a popular tool in applied and theoretical probability, typically used for Gaussian and Poisson approximation problems. The principal goal of the method is to provide quantitative assessments in distributional comparison statements of the form where Z follows a known and well-understood probability distribution (typically normal or Poisson) and W is the object of interest. Charles Stein  in 1972 laid the foundation of what is now called “Stein’s method” by aiming at normal approximations.
Stein’s method consists of two distinct components, namely
Part A: a framework allowing to convert the problem of bounding the error in the approximation of W by Z into a problem of bounding the expectation of a certain functional of W.
Part B: a collection of techniques to bound the expectation appearing in Part A; the details of these techniques are strongly dependent on the properties of W as well as on the form of the functional.
We refer the interested reader to [7, 8] for detailed recent accounts on this powerful method. The reader will understand in the next sections why Stein’s method has been of use for quantifying the desired measure, even without formal proofs or mathematical details.
2.2 Notation and formulation of the main goal
We start by fixing our notations. We consider independent and identically distributed (discrete or absolutely continuous) observations from a parametric model with parameter of interest . We denote the likelihood of by where are the observed values. Take two different (possibly improper) prior densities and for our parameter θ; the famous Bayes’ theorem then readily yields the respective posterior densities
where are normalizing constants that depend only on the observed values. We denote by and the couples of random variables and cumulative distribution functions associated with the densities and .
These notations allow us to formulate the main goal: measure the Wasserstein distance between and , as this will exactly correspond to the difference between the posteriors resulting from the two priors p1 and p2. Sharp upper and lower bounds have been provided for this Wasserstein distance, first in  for the special case of one prior being flat uniform, then in all generality in . The determination of the upper bound has been achieved by means of the Stein Method: first a relevant Stein operator has been found (Part A), and then a new technique designed in  has been put to use for Part B. The reader is referred to these two papers for details about the calculations; since this chapter is part of a book on Bayesian inference, we prefer to keep out those rather probabilistic manipulations.
2.3 The general result
The key element in the mathematical developments underlying the present problem is that the densities and are nested, meaning that one support is included in the other. Without loss of generality we here suppose that , allowing us to express as with
The following general result has been obtained in , where we refer the reader to for a proof.
Theorem 1.1 Consider the set of Lipschitz-1 functions on and define
where ai is the lower bound of the support of pi. Suppose that both posterior distributions have finite means μ1 and μ2, respectively. Assume that is differentiable on I2 and satisfies (i) , (ii) is integrable for all and (iii) for all . Then
and, if the variance of exists,
where stands for the infinity norm.
This result quantifies in all generality the measure of the difference between two priors p1 and p2, and comprises of course the special case where one prior is flat uniform. Quite nicely, if ρ is a monotone increasing or decreasing function, the bounds do coincide, leading to
hence an exact result. The reader notices the sharpness of these bounds given that they contain the same quantities in both the upper and lower bounds; this fact is further underpinned by the equality Eq. (2). Finally we wish to stress that the functions from Eq. (1) are called Stein kernel in the Stein method literature and that these functions are always positive and vanish at the boundaries of the support.
3. Applications and illustrations
Numerous examples have been treated in [3, 4], such as priors for the location parameter of a normal distribution, the scale parameter of a normal distribution, the success parameter of a binomial or the event-enumerating parameter of the Poisson distribution, to cite but these. In this section we will, on the one hand, investigate a new example, namely the scale parameter of an inverse gamma distribution, and, on the other hand, revisit the binomial case. Besides providing the bounds, we will also for the first time plot numerical values for the bounds and hence shed new intuitive light on this measure of the impact of the choice of the prior.
3.1 Priors for the scale parameter of the inverse gamma (IG) distribution
The inverse gamma (IG) distribution has the probability density function
where α and β are the positive shape and scale parameters, respectively. This distribution corresponds to the reciprocal of a gamma distribution (if then ) and is frequently encountered in domains such as machine learning, survival analysis and reliability theory. Within Bayesian Inference, it is a popular choice as prior for the scale parameter of a normal distribution. In the present setting, we consider as the parameter of interest and α is fixed. The observations sampled from this distribution are written .
The first prior is the popular noninformative Jeffreys prior. It is invariant under reparameterization and is proportional to the square root of the Fisher information quantity associated with the parameter of interest. In the present setting simple calculations show that it is proportional to . The resulting posterior P1 then has a density of the form
which is none other than a gamma distribution with parameters .
Now, the gamma distribution happens to be the conjugate prior for the scale parameter of an IG distribution. We consider thus as second prior a general gamma distribution with density , where the shape and scale parameters η and are strictly positive. The ensuing posterior distribution P2 has then the density
which is a gamma distribution with updated parameters .
Considering Jeffreys prior as p1 and the gamma prior as p2 leads to the ratio
One can easily check that all conditions of Theorem 1.1 are fulfilled, hence we can calculate the bounds. The lower bound is directly obtained as follows:
In order to acquire the upper bound we need to calculate
and, writing the random variable associated with and the related density, we get
From the Stein literature we know that the Stein kernel for the gamma distribution with parameters corresponds to . Employing the triangular inequality we have thus
Now we need to calculate the expectation
The final expression for the upper bound then corresponds to
The Wasserstein distance between the posteriors based on the Jeffreys prior and conjugate gamma prior for the scale parameter β of the IG distribution is thus bounded as
It can be seen that both the lower and upper bound are of the order of . In addition, it is noticeable that for the larger observations, the rate of convergence is getting slower.
In order to show the performance of the methodology which leads to have the lower and upper bounds, we have conducted a simulation study including two parts. First we simulate samples for each sample size from the inverse gamma distribution with parameters in each iteration. For each of these samples we calculate the lower and upper bounds of the Wasserstein distance and calculate the average over all N replications, together with the difference between the bounds. Finally we plot these values for each sample size in Figure 1. We repeat the same process for samples with the same sizes. The hyperparameters from the prior gamma distribution are . We clearly observe how fast these values decrease with the sample size. Of course, augmenting the number of replications does not increase the speed of convergence, however the curves become noticeably smoother.
This methodology not only can help the practitioners to make a decision between existing priors in theory, but also helps them to know from what sample size on the effect of choosing one prior becomes less important, especially in situations when the cost and time matter. This can be particularly useful when the hesitation is between a simple, closed-form prior and a more complicated one. It is advisable to use the simpler one when there is no considerable difference between the effect of the two priors.
3.2 The impact of priors for the success parameter of the binomial model
The probability mass function of a binomial distribution is given by
where is the number of observed successes, the natural number n indicates the number of binary trials and stands for the success parameter. In this setting we suppose n is fixed and the underlying parameter of interest is θ.
A comprehensive comparison of various priors for the binomial distribution including a beta prior, the Haldane prior and Jeffreys prior, has been done in , based on the methodology described above. Therefore, since there is a complete reference for the reader in this case, we use the binomial distribution as a second example to show numerical results.
The theoretical lower and upper bounds between a prior and the flat uniform prior are given by
where x is the observed number of successes. We see that both lower and upper bounds are of the order of . This rate of convergence remains even in the extreme cases and . We invite the reader to see [3, 9] for more details.
In order to illustrate the behavior of the lower and upper bounds and the distances between them, we have conducted a two-part simulation study for the binomial distribution. First, we consider 100 sample sizes (number of trials in the binomial distribution) varying from 10 to 1000 by steps of 10, and generate binomial data exactly once for every sample size (with ). The results of the bounds, obtained for hyperparameters from the beta prior, are reported in Figure 2a and we can see that, even with only one iteration, when the number of trials (the sample size) increases the lower and upper bound become closer, which is a numerical quantification of the fact that the influence of the choice of the prior wanes asymptotically. This becomes also visible from the distance between the two bounds. Sampling only once for each sample size leads to slightly unpleasant variations in the lower bounds (non-monotone behavior), which however nearly disappear in the second considered scenario. Indeed, in Figure 2b we increased the number of iterations to 50 for the same different sample sizes and took averages. A better smoothness is the consequence. This simulation study not only provides the reader with numerical values for the bounds, to which he/she can compare his/her bounds obtained for real data, but also gives a nice visualization of the impact of the choice of the prior at fixed sample size. The main conclusion is that the impact drops fast at small sample sizes, and the bounds start to become very close for medium-to-large sample sizes.
Finally, we investigate the impact of the hyperparameters on the upper and lower bounds. To this end, we varied both α and β in Table 1. The situation with α fixed to two and relatively small β corresponds well with , which explains why the upper and lower bounds, and hence the Wasserstein distance and thus the impact of the prior, are the smallest. Increasing β more augments the distance. On the contrary, fixing yields priors rather centered around large values of p and hence bigger distances. Moreover, the more α is increased, the more the distance augments, as the prior is further away from the data and hence impacts more on the posterior at a fixed sample size. For the sake of illustration, we present three choices of hyperparameters together with the bounds and the related prior density in Figure 3. This will help understanding our conclusions.
|Hyperparameters (α, β)||Average of the lower bounds||Average of the upper bounds|
In this chapter we have presented a recently developed measure for the impact of the choice of the prior distribution in Bayesian statistics. We have presented the general theoretical result, explained how to use it in a particular example and provided some graphics to illustrate it numerically. The practical importance of this study is when practitioners hesitate between two proposed priors in a given situation. For instance, Kavetski et al.  considered a storm depth multiplier model to represent rainfall uncertainty where the errors appear under multiplicative form and are assumed to be normal. They fix the mean, but state that “less is understood about the degree of rainfall uncertainty,” i.e., the multiplier variance, and therefore studied various priors for the variance. Knowledge of the tools presented in this chapter would have simplified the decision process.
In case of missing data, the present methodology can still be used. Either the data get imputed, in which case nothing changes, or the missing data simply are left out from the calculation of upper and lower bounds, whose expression does of course not alter.
Further developments on this new measure might lead to a more concrete quantification of words such as “informative, weakly informative, noninformative” priors, and we hope to have stimulated interest in this promising new line of research within Bayesian Inference.
This research is supported by a BOF Starting Grant of Ghent University.