The summary of upper and lower bounds for different hyperparameters, with p

## Abstract

A key question in Bayesian analysis is the effect of the prior on the posterior, and how we can measure this effect. Will the posterior distributions derived with distinct priors become very similar if more and more data are gathered? It has been proved formally that, under certain regularity conditions, the impact of the prior is waning as the sample size increases. From a practical viewpoint it is more important to know what happens at finite sample size n. In this chapter, we shall explain how we tackle this crucial question from an innovative approach. To this end, we shall review some notions from probability theory such as the Wasserstein distance and the popular Stein’s method, and explain how we use these a priori unrelated concepts in order to measure the impact of priors. Examples will illustrate our findings, including conjugate priors and the Jeffreys prior.

### Keywords

- conjugate prior
- Jeffreys prior
- prior distribution
- posterior distribution
- Stein’s method
- Wasserstein distance

## 1. Introduction

A key question in Bayesian analysis is the choice of the prior in a given situation. Numerous proposals and divergent opinions exist on this matter, but our aim is not to delve into a review or discussion, rather we want to provide the reader with a description of a useful new tool allowing him/her to make a decision. More precisely, we explain how to effectively measure the effect of the choice of a given prior on the resulting posterior. How much do two posteriors, derived from two distinct priors, differ? Providing a quantitative answer to this question is important as it also informs us about the ensuing inferential procedures. It has been proved formally in [1, 2] that, under certain regularity conditions, the impact of the prior is waning as the sample size increases. From a practical viewpoint it is however more interesting to know what happens at finite sample size *n*, and this is precisely the situation we are considering in this chapter.

Recently, [3, 4] have devised a novel tool to answer this question. They measure the Wasserstein distance between the posterior distributions based on two distinct priors at fixed sample size *n*. The Wasserstein (more precisely, Wasserstein-1) distance is defined as

for *X*_{1} and *X*_{2} random variables with respective distribution functions *P*_{1} and *P*_{2}, and where *earth mover distance* in computer science, see [5] for more information. The resulting distance thus gives us the desired measure of the difference between two posteriors. If one of the two priors is the flat uniform prior (leading to the posterior coinciding with the data likelihood), then this measure quantifies how much the other chosen prior has impacted on the outcome as compared to a data-only posterior. Now, the Wasserstein distance being mostly impossible to calculate exactly, it is necessary to obtain sharp upper and lower bounds, which will partially be achieved by using techniques from the so-called Stein method, a famous tool in probabilistic approximation theory. We opt for the Wasserstein metric instead of, e.g., the Kullback-Leibler divergence because of precisely its nice link with the Stein method, see [3].

The chapter is organized as follows. In Section 2 we provide the notations and terminology used throughout the paper, provide the reader with the minimal necessary background knowledge on the Stein method, and state the main result regarding the measure of the impact of priors. Then in Section 3 we illustrate how this new measure works in practice, by first working out a completely new example, namely priors for the scale parameter of the inverse gamma distribution, and second giving new insights into an example first treated in both [3, 4], namely priors for the success parameter in the binomial distribution.

## 2. The measure in its most general form

In this section we provide the reader with the general form of the new measure of the impact of the choice of prior distributions. Before doing so, we however first give a very brief overview on Stein’s method that is of independent interest.

### 2.1 Stein’s method in a nutshell

Stein’s method is a popular tool in applied and theoretical probability, typically used for Gaussian and Poisson approximation problems. The principal goal of the method is to provide quantitative assessments in distributional comparison statements of the form *Z* follows a known and well-understood probability distribution (typically normal or Poisson) and *W* is the object of interest. Charles Stein [6] in 1972 laid the foundation of what is now called “Stein’s method” by aiming at normal approximations.

Stein’s method consists of two distinct components, namely

**Part A**: a framework allowing to convert the problem of bounding the error in the approximation of*W*by*Z*into a problem of bounding the expectation of a certain functional of*W*.**Part B**: a collection of techniques to bound the expectation appearing in Part A; the details of these techniques are strongly dependent on the properties of*W*as well as on the form of the functional.

We refer the interested reader to [7, 8] for detailed recent accounts on this powerful method. The reader will understand in the next sections why Stein’s method has been of use for quantifying the desired measure, even without formal proofs or mathematical details.

### 2.2 Notation and formulation of the main goal

We start by fixing our notations. We consider independent and identically distributed (discrete or absolutely continuous) observations *θ*; the famous Bayes’ theorem then readily yields the respective posterior densities

where

These notations allow us to formulate the main goal: measure the Wasserstein distance between *p*_{1} and *p*_{2}. Sharp upper and lower bounds have been provided for this Wasserstein distance, first in [3] for the special case of one prior being flat uniform, then in all generality in [4]. The determination of the upper bound has been achieved by means of the Stein Method: first a relevant Stein operator has been found (Part A), and then a new technique designed in [3] has been put to use for Part B. The reader is referred to these two papers for details about the calculations; since this chapter is part of a book on Bayesian inference, we prefer to keep out those rather probabilistic manipulations.

### 2.3 The general result

The key element in the mathematical developments underlying the present problem is that the densities *nested*, meaning that one support is included in the other. Without loss of generality we here suppose that

The following general result has been obtained in [4], where we refer the reader to for a proof.

Theorem 1.1 Consider

where *ai* is the lower bound of the support *pi*. Suppose that both posterior distributions have finite means *μ*_{1} and *μ*_{2}, respectively. Assume that *I*_{2} and satisfies (i)

and, if the variance of

where

This result quantifies in all generality the measure of the difference between two priors *p*_{1} and *p*_{2}, and comprises of course the special case where one prior is flat uniform. Quite nicely, if *ρ* is a monotone increasing or decreasing function, the bounds do coincide, leading to

hence an exact result. The reader notices the sharpness of these bounds given that they contain the same quantities in both the upper and lower bounds; this fact is further underpinned by the equality Eq. (2). Finally we wish to stress that the functions

## 3. Applications and illustrations

Numerous examples have been treated in [3, 4], such as priors for the location parameter of a normal distribution, the scale parameter of a normal distribution, the success parameter of a binomial or the event-enumerating parameter of the Poisson distribution, to cite but these. In this section we will, on the one hand, investigate a new example, namely the scale parameter of an inverse gamma distribution, and, on the other hand, revisit the binomial case. Besides providing the bounds, we will also for the first time plot numerical values for the bounds and hence shed new intuitive light on this measure of the impact of the choice of the prior.

### 3.1 Priors for the scale parameter of the inverse gamma (IG) distribution

The inverse gamma (IG) distribution has the probability density function

where *α* and *β* are the positive shape and scale parameters, respectively. This distribution corresponds to the reciprocal of a gamma distribution (if *α* is fixed. The observations sampled from this distribution are written

The first prior is the popular noninformative Jeffreys prior. It is invariant under reparameterization and is proportional to the square root of the Fisher information quantity associated with the parameter of interest. In the present setting simple calculations show that it is proportional to *P*_{1} then has a density of the form

which is none other than a gamma distribution with parameters

Now, the gamma distribution happens to be the conjugate prior for the scale parameter of an IG distribution. We consider thus as second prior a general gamma distribution with density *η* and *P*_{2} has then the density

which is a gamma distribution with updated parameters

Considering Jeffreys prior as *p*_{1} and the gamma prior as *p*_{2} leads to the ratio

One can easily check that all conditions of Theorem 1.1 are fulfilled, hence we can calculate the bounds. The lower bound is directly obtained as follows:

In order to acquire the upper bound we need to calculate

and, writing

From the Stein literature we know that the Stein kernel for the gamma distribution with parameters

Now we need to calculate the expectation

The final expression for the upper bound then corresponds to

The Wasserstein distance between the posteriors based on the Jeffreys prior and conjugate gamma prior for the scale parameter *β* of the IG distribution is thus bounded as

It can be seen that both the lower and upper bound are of the order of

In order to show the performance of the methodology which leads to have the lower and upper bounds, we have conducted a simulation study including two parts. First we simulate *N* replications, together with the difference between the bounds. Finally we plot these values for each sample size in Figure 1. We repeat the same process for

This methodology not only can help the practitioners to make a decision between existing priors in theory, but also helps them to know from what sample size on the effect of choosing one prior becomes less important, especially in situations when the cost and time matter. This can be particularly useful when the hesitation is between a simple, closed-form prior and a more complicated one. It is advisable to use the simpler one when there is no considerable difference between the effect of the two priors.

### 3.2 The impact of priors for the success parameter of the binomial model

The probability mass function of a binomial distribution is given by

where *n* indicates the number of binary trials and *n* is fixed and the underlying parameter of interest is *θ*.

A comprehensive comparison of various priors for the binomial distribution including a beta prior, the Haldane prior and Jeffreys prior, has been done in [9], based on the methodology described above. Therefore, since there is a complete reference for the reader in this case, we use the binomial distribution as a second example to show numerical results.

The theoretical lower and upper bounds between a

where *x* is the observed number of successes. We see that both lower and upper bounds are of the order of

In order to illustrate the behavior of the lower and upper bounds and the distances between them, we have conducted a two-part simulation study for the binomial distribution. First, we consider 100 sample sizes (number of trials in the binomial distribution) varying from 10 to 1000 by steps of 10, and generate binomial data exactly once for every sample size (with

Finally, we investigate the impact of the hyperparameters on the upper and lower bounds. To this end, we varied both *α* and *β* in Table 1. The situation with *α* fixed to two and relatively small *β* corresponds well with *β* more augments the distance. On the contrary, fixing *p* and hence bigger distances. Moreover, the more *α* is increased, the more the distance augments, as the prior is further away from the data and hence impacts more on the posterior at a fixed sample size. For the sake of illustration, we present three choices of hyperparameters together with the bounds and the related prior density in Figure 3. This will help understanding our conclusions.

Hyperparameters (α, β) | Average of the lower bounds | Average of the upper bounds |
---|---|---|

(0.2, 0.4) | ||

(0.2, 0.8) | ||

(2, 2) | ||

(2, 5) | ||

## 4. Conclusions

In this chapter we have presented a recently developed measure for the impact of the choice of the prior distribution in Bayesian statistics. We have presented the general theoretical result, explained how to use it in a particular example and provided some graphics to illustrate it numerically. The practical importance of this study is when practitioners hesitate between two proposed priors in a given situation. For instance, Kavetski et al. [10] considered a storm depth multiplier model to represent rainfall uncertainty where the errors appear under multiplicative form and are assumed to be normal. They fix the mean, but state that “less is understood about the degree of rainfall uncertainty,” i.e., the multiplier variance, and therefore studied various priors for the variance. Knowledge of the tools presented in this chapter would have simplified the decision process.

In case of missing data, the present methodology can still be used. Either the data get imputed, in which case nothing changes, or the missing data simply are left out from the calculation of upper and lower bounds, whose expression does of course not alter.

Further developments on this new measure might lead to a more concrete quantification of words such as “informative, weakly informative, noninformative” priors, and we hope to have stimulated interest in this promising new line of research within Bayesian Inference.

## Acknowledgments

This research is supported by a BOF Starting Grant of Ghent University.