Open access peer-reviewed chapter - ONLINE FIRST

# On the Impact of the Choice of the Prior in Bayesian Statistics

Submitted: April 24th 2019Reviewed: August 2nd 2019Published: September 21st 2019

DOI: 10.5772/intechopen.88994

## Abstract

A key question in Bayesian analysis is the effect of the prior on the posterior, and how we can measure this effect. Will the posterior distributions derived with distinct priors become very similar if more and more data are gathered? It has been proved formally that, under certain regularity conditions, the impact of the prior is waning as the sample size increases. From a practical viewpoint it is more important to know what happens at finite sample size n. In this chapter, we shall explain how we tackle this crucial question from an innovative approach. To this end, we shall review some notions from probability theory such as the Wasserstein distance and the popular Stein’s method, and explain how we use these a priori unrelated concepts in order to measure the impact of priors. Examples will illustrate our findings, including conjugate priors and the Jeffreys prior.

### Keywords

• conjugate prior
• Jeffreys prior
• prior distribution
• posterior distribution
• Stein’s method
• Wasserstein distance

## 1. Introduction

A key question in Bayesian analysis is the choice of the prior in a given situation. Numerous proposals and divergent opinions exist on this matter, but our aim is not to delve into a review or discussion, rather we want to provide the reader with a description of a useful new tool allowing him/her to make a decision. More precisely, we explain how to effectively measure the effect of the choice of a given prior on the resulting posterior. How much do two posteriors, derived from two distinct priors, differ? Providing a quantitative answer to this question is important as it also informs us about the ensuing inferential procedures. It has been proved formally in [1, 2] that, under certain regularity conditions, the impact of the prior is waning as the sample size increases. From a practical viewpoint it is however more interesting to know what happens at finite sample size n, and this is precisely the situation we are considering in this chapter.

Recently, [3, 4] have devised a novel tool to answer this question. They measure the Wasserstein distance between the posterior distributions based on two distinct priors at fixed sample size n. The Wasserstein (more precisely, Wasserstein-1) distance is defined as

dWP1P2=suphHEhX1EhX2

for X1 and X2 random variables with respective distribution functions P1 and P2, and where Hstands for the class of Lipschitz-1 functions. It is a popular distance between two distributions, related to optimal transport and therefore also known as earth mover distance in computer science, see [5] for more information. The resulting distance thus gives us the desired measure of the difference between two posteriors. If one of the two priors is the flat uniform prior (leading to the posterior coinciding with the data likelihood), then this measure quantifies how much the other chosen prior has impacted on the outcome as compared to a data-only posterior. Now, the Wasserstein distance being mostly impossible to calculate exactly, it is necessary to obtain sharp upper and lower bounds, which will partially be achieved by using techniques from the so-called Stein method, a famous tool in probabilistic approximation theory. We opt for the Wasserstein metric instead of, e.g., the Kullback-Leibler divergence because of precisely its nice link with the Stein method, see [3].

The chapter is organized as follows. In Section 2 we provide the notations and terminology used throughout the paper, provide the reader with the minimal necessary background knowledge on the Stein method, and state the main result regarding the measure of the impact of priors. Then in Section 3 we illustrate how this new measure works in practice, by first working out a completely new example, namely priors for the scale parameter of the inverse gamma distribution, and second giving new insights into an example first treated in both [3, 4], namely priors for the success parameter in the binomial distribution.

## 2. The measure in its most general form

In this section we provide the reader with the general form of the new measure of the impact of the choice of prior distributions. Before doing so, we however first give a very brief overview on Stein’s method that is of independent interest.

### 2.1 Stein’s method in a nutshell

Stein’s method is a popular tool in applied and theoretical probability, typically used for Gaussian and Poisson approximation problems. The principal goal of the method is to provide quantitative assessments in distributional comparison statements of the form WZwhere Z follows a known and well-understood probability distribution (typically normal or Poisson) and W is the object of interest. Charles Stein [6] in 1972 laid the foundation of what is now called “Stein’s method” by aiming at normal approximations.

Stein’s method consists of two distinct components, namely

1. Part A: a framework allowing to convert the problem of bounding the error in the approximation of W by Z into a problem of bounding the expectation of a certain functional of W.

2. Part B: a collection of techniques to bound the expectation appearing in Part A; the details of these techniques are strongly dependent on the properties of W as well as on the form of the functional.

We refer the interested reader to [7, 8] for detailed recent accounts on this powerful method. The reader will understand in the next sections why Stein’s method has been of use for quantifying the desired measure, even without formal proofs or mathematical details.

### 2.2 Notation and formulation of the main goal

We start by fixing our notations. We consider independent and identically distributed (discrete or absolutely continuous) observations X1,,Xnfrom a parametric model with parameter of interest θΘR. We denote the likelihood of X1,,Xnby x; θwhere x=x1xnare the observed values. Take two different (possibly improper) prior densities p1θand p2θfor our parameter θ; the famous Bayes’ theorem then readily yields the respective posterior densities

piθ; x=κixpiθx; θ,i=1,2,

where κ1x,κ2xare normalizing constants that depend only on the observed values. We denote by Θ1P1and Θ2P2the couples of random variables and cumulative distribution functions associated with the densities p1θ; xand p2θ; x.

These notations allow us to formulate the main goal: measure the Wasserstein distance between p1θ; xand p2θ; x, as this will exactly correspond to the difference between the posteriors resulting from the two priors p1 and p2. Sharp upper and lower bounds have been provided for this Wasserstein distance, first in [3] for the special case of one prior being flat uniform, then in all generality in [4]. The determination of the upper bound has been achieved by means of the Stein Method: first a relevant Stein operator has been found (Part A), and then a new technique designed in [3] has been put to use for Part B. The reader is referred to these two papers for details about the calculations; since this chapter is part of a book on Bayesian inference, we prefer to keep out those rather probabilistic manipulations.

### 2.3 The general result

The key element in the mathematical developments underlying the present problem is that the densities p1θ; xand p2θ; xare nested, meaning that one support is included in the other. Without loss of generality we here suppose that I2I1, allowing us to express p2θ; xas κ2xκ1xρθp1θ; xwith

ρθ=p2θp1θ.

The following general result has been obtained in [4], where we refer the reader to for a proof.

Theorem 1.1 Consider Hthe set of Lipschitz-1 functions on Rand define

τiθ; x=1piθ; xaiθμiypiy; xdy,i=1,2,E1

where ai is the lower bound of the support Ii=aibiof pi. Suppose that both posterior distributions have finite means μ1 and μ2, respectively. Assume that θρθis differentiable on I2 and satisfies (i) EΘ1μ1ρΘ1<, (ii) ρθa1θhyEhΘ1p1(y; x)dyis integrable for all hHand (iii) limθa2,b2ρθa1θhyEhΘ1p1y; xdy=0for all hH. Then

μ1μ2=Eτ1Θ1; xρΘ1EρΘ1dWP1P2Eτ1Θ1; xρΘ1EρΘ1

and, if the variance of Θ1exists,

μ1μ2dWP1P2ρVarΘ1EρΘ1

where stands for the infinity norm.

This result quantifies in all generality the measure of the difference between two priors p1 and p2, and comprises of course the special case where one prior is flat uniform. Quite nicely, if ρ is a monotone increasing or decreasing function, the bounds do coincide, leading to

dWP1; P2=Eτ1Θ1; xρΘ1EρΘ1,E2

hence an exact result. The reader notices the sharpness of these bounds given that they contain the same quantities in both the upper and lower bounds; this fact is further underpinned by the equality Eq. (2). Finally we wish to stress that the functions τiθ; x,i=1,2,from Eq. (1) are called Stein kernel in the Stein method literature and that these functions are always positive and vanish at the boundaries of the support.

## 3. Applications and illustrations

Numerous examples have been treated in [3, 4], such as priors for the location parameter of a normal distribution, the scale parameter of a normal distribution, the success parameter of a binomial or the event-enumerating parameter of the Poisson distribution, to cite but these. In this section we will, on the one hand, investigate a new example, namely the scale parameter of an inverse gamma distribution, and, on the other hand, revisit the binomial case. Besides providing the bounds, we will also for the first time plot numerical values for the bounds and hence shed new intuitive light on this measure of the impact of the choice of the prior.

### 3.1 Priors for the scale parameter of the inverse gamma (IG) distribution

The inverse gamma (IG) distribution has the probability density function

xβαΓαxα1expβx, x>0,

where α and β are the positive shape and scale parameters, respectively. This distribution corresponds to the reciprocal of a gamma distribution (if XGammaαβthen 1XIGαβ) and is frequently encountered in domains such as machine learning, survival analysis and reliability theory. Within Bayesian Inference, it is a popular choice as prior for the scale parameter of a normal distribution. In the present setting, we consider θ=βas the parameter of interest and α is fixed. The observations sampled from this distribution are written x1,,xn.

The first prior is the popular noninformative Jeffreys prior. It is invariant under reparameterization and is proportional to the square root of the Fisher information quantity associated with the parameter of interest. In the present setting simple calculations show that it is proportional to 1β. The resulting posterior P1 then has a density of the form

p1βx1ββexpβi=1n1xi=β1expβi=1n1xi

which is none other than a gamma distribution with parameters i=1n1xi.

Now, the gamma distribution happens to be the conjugate prior for the scale parameter of an IG distribution. We consider thus as second prior a general gamma distribution with density βκηΓηβη1expκβ, where the shape and scale parameters η and κare strictly positive. The ensuing posterior distribution P2 has then the density

p2βxβη1expκβ×βexpβi=1n1xi=β+η1expβi=1n1xi+κ

which is a gamma distribution with updated parameters +ηi=1n1xi+κ.

Considering Jeffreys prior as p1 and the gamma prior as p2 leads to the ratio

ρβ=p2βp1βκηΓηβη1expκβ1β=κηΓηβηexpκβ.

One can easily check that all conditions of Theorem 1.1 are fulfilled, hence we can calculate the bounds. The lower bound is directly obtained as follows:

dWP1P2μ1μ2=i=1n1xi+ηi=1n1xi+κE3
=i=1n1xi+nακi=1n1xiηi=1n1xii=1n1xii=1n1xi+κE4
=nακηi=1n1xii=1n1xii=1n1xi+κ.E5

In order to acquire the upper bound we need to calculate

ρβ=κηΓηβη1expκβηκβ

and, writing Θ1the random variable associated with Gammai=1n1xiand fGammai=1n1xiβthe related density, we get

EρΘ1=0κηΓηβηexpκβ×fGammai=1n1xiβE6
=κηΓηi=1n1xiΓ0βηexpκββ1expβi=1n1xiE7
=κηΓηi=1n1xiΓ0β+η1expβi=1n1xi+κE8
=κηΓηi=1n1xiΓΓ+ηi=1n1xi+κ+ηE9
=κηBetaηi=1n1xii=1n1xi+κ+η.E10

From the Stein literature we know that the Stein kernel for the gamma distribution with parameters i=1n1xicorresponds to τβx=βi=1n1xi. Employing the triangular inequality we have thus

EτΘ1; xρΘ1=EΘ1i=1n1xiκηΓηΘ1η1expκΘ1ηκΘ1E11
κηi=1n1xiΓηEΘ1ηexpκΘ1η+κΘ1.E12

Now we need to calculate the expectation

EΘ1ηexpκΘ1η+κΘ1E13
=0βηexpκβη+κβ×fGammai=1n1xiβE14
=i=1n1xiΓ0ηβ+η1expβi=1n1xi+κE15
+i=1n1xiΓ0κβ+ηexpβi=1n1xi+κE16
=i=1n1xiΓηΓ+ηi=1n1xi+κ+η+κΓ+η+1i=1n1xi+κ+η+1.E17

The final expression for the upper bound then corresponds to

dWP1P2κηi=1n1xiΓη×i=1n1xiΓηΓ+ηi=1n1xi+κ+η+κΓ+η+1i=1n1xi+κ+η+1κηBetaη×i=1n1xii=1n1xi+κ+ηE18
=Betaηi=1n1xi+κ+ηΓΓηΓ+ηi=1n1xi×1i=1n1xi+κ+ηη+κ+ηi=1n1xi+κE19
=1i=1n1xiη+κ+ηi=1n1xi+κ.E20

The Wasserstein distance between the posteriors based on the Jeffreys prior and conjugate gamma prior for the scale parameter β of the IG distribution is thus bounded as

nακηi=1n1xii=1n1xii=1n1xi+κdWP1P21i=1n1xiη+κ+ηκ+i=1n1xi.

It can be seen that both the lower and upper bound are of the order of On1. In addition, it is noticeable that for the larger observations, the rate of convergence is getting slower.

In order to show the performance of the methodology which leads to have the lower and upper bounds, we have conducted a simulation study including two parts. First we simulate N=100samples for each sample size n=10,11,,100from the inverse gamma distribution with parameters αβ=0.5,1in each iteration. For each of these samples we calculate the lower and upper bounds of the Wasserstein distance and calculate the average over all N replications, together with the difference between the bounds. Finally we plot these values for each sample size in Figure 1. We repeat the same process for N=1000samples with the same sizes. The hyperparameters from the prior gamma distribution are κη=0.2,2. We clearly observe how fast these values decrease with the sample size. Of course, augmenting the number of replications does not increase the speed of convergence, however the curves become noticeably smoother.

This methodology not only can help the practitioners to make a decision between existing priors in theory, but also helps them to know from what sample size on the effect of choosing one prior becomes less important, especially in situations when the cost and time matter. This can be particularly useful when the hesitation is between a simple, closed-form prior and a more complicated one. It is advisable to use the simpler one when there is no considerable difference between the effect of the two priors.

### 3.2 The impact of priors for the success parameter of the binomial model

The probability mass function of a binomial distribution is given by

xnxθx1θnx

where x01nis the number of observed successes, the natural number n indicates the number of binary trials and θ01stands for the success parameter. In this setting we suppose n is fixed and the underlying parameter of interest is θ.

A comprehensive comparison of various priors for the binomial distribution including a beta prior, the Haldane prior and Jeffreys prior, has been done in [9], based on the methodology described above. Therefore, since there is a complete reference for the reader in this case, we use the binomial distribution as a second example to show numerical results.

The theoretical lower and upper bounds between a Betaαβprior and the flat uniform prior are given by

x+1n+2α+β2n+α+βα1n+α+βdWP1P21n+2α1+x+αn+α+ββ1α1,

where x is the observed number of successes. We see that both lower and upper bounds are of the order of On1. This rate of convergence remains even in the extreme cases x=0and x=n. We invite the reader to see [3, 9] for more details.

In order to illustrate the behavior of the lower and upper bounds and the distances between them, we have conducted a two-part simulation study for the binomial distribution. First, we consider 100 sample sizes (number of trials in the binomial distribution) varying from 10 to 1000 by steps of 10, and generate binomial data exactly once for every sample size (with θ=0.2). The results of the bounds, obtained for hyperparameters αβ=24from the beta prior, are reported in Figure 2a and we can see that, even with only one iteration, when the number of trials (the sample size) increases the lower and upper bound become closer, which is a numerical quantification of the fact that the influence of the choice of the prior wanes asymptotically. This becomes also visible from the distance between the two bounds. Sampling only once for each sample size leads to slightly unpleasant variations in the lower bounds (non-monotone behavior), which however nearly disappear in the second considered scenario. Indeed, in Figure 2b we increased the number of iterations to 50 for the same different sample sizes and took averages. A better smoothness is the consequence. This simulation study not only provides the reader with numerical values for the bounds, to which he/she can compare his/her bounds obtained for real data, but also gives a nice visualization of the impact of the choice of the prior at fixed sample size. The main conclusion is that the impact drops fast at small sample sizes, and the bounds start to become very close for medium-to-large sample sizes.

Finally, we investigate the impact of the hyperparameters on the upper and lower bounds. To this end, we varied both α and β in Table 1. The situation with α fixed to two and relatively small β corresponds well with p=0.2, which explains why the upper and lower bounds, and hence the Wasserstein distance and thus the impact of the prior, are the smallest. Increasing β more augments the distance. On the contrary, fixing β=2yields priors rather centered around large values of p and hence bigger distances. Moreover, the more α is increased, the more the distance augments, as the prior is further away from the data and hence impacts more on the posterior at a fixed sample size. For the sake of illustration, we present three choices of hyperparameters together with the bounds and the related prior density in Figure 3. This will help understanding our conclusions.

Hyperparameters (α, β)Average of the lower boundsAverage of the upper bounds
(0.2, 0.4)0.0025613830.003726728
(0.2, 0.8)0.002960020.003344393
(2, 2)0.0026993250.00490119
(2, 5)0.00081153840.007984289
2100.0045062710.01241273
2150.0082088870.01626326
2300.017501770.02581062
2500.027392050.0359027
2,1000.045922350.05470826
2,2000.070717660.07976386
2,5000.11030480.1196464
210000.13999610.1495087
1020.028133670.03132908
3520.085711150.09033568
5020.11271360.1178113
100,20.18302720.189071
200,20.27837220.2853418
400,20.39333380.401145
700,20.49012090.4985089
100020.54828690.5569829

### Table 1.

The summary of upper and lower bounds for different hyperparameters, with p =0.2and for N = 50 iterations.

## 4. Conclusions

In this chapter we have presented a recently developed measure for the impact of the choice of the prior distribution in Bayesian statistics. We have presented the general theoretical result, explained how to use it in a particular example and provided some graphics to illustrate it numerically. The practical importance of this study is when practitioners hesitate between two proposed priors in a given situation. For instance, Kavetski et al. [10] considered a storm depth multiplier model to represent rainfall uncertainty where the errors appear under multiplicative form and are assumed to be normal. They fix the mean, but state that “less is understood about the degree of rainfall uncertainty,” i.e., the multiplier variance, and therefore studied various priors for the variance. Knowledge of the tools presented in this chapter would have simplified the decision process.

In case of missing data, the present methodology can still be used. Either the data get imputed, in which case nothing changes, or the missing data simply are left out from the calculation of upper and lower bounds, whose expression does of course not alter.

Further developments on this new measure might lead to a more concrete quantification of words such as “informative, weakly informative, noninformative” priors, and we hope to have stimulated interest in this promising new line of research within Bayesian Inference.

## Acknowledgments

This research is supported by a BOF Starting Grant of Ghent University.

chapter PDF

## More

© 2019 The Author(s). Licensee IntechOpen. This chapter is distributed under the terms of the Creative Commons Attribution 3.0 License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

## How to cite and reference

### Cite this chapter Copy to clipboard

Fatemeh Ghaderinezhad and Christophe Ley (September 21st 2019). On the Impact of the Choice of the Prior in Bayesian Statistics [Online First], IntechOpen, DOI: 10.5772/intechopen.88994. Available from: