Hypothesis testing in Bayesian statistics

In Bayesian statistics, the data D is in favor of the hypotheses H if p(H|D) > p(D), which means that the posterior is greater than the prior. This is true if p(D|H) > p(D|~H), or p(D|H) / p(D|~H) > 1. The latter is called the likelihood ratio, or the Bayes factor; it measures the strength of the evidence.

We can not use the data to form the hypotheses because this process would be very biased, and thus almost any data would be evidence in favor of our hypotheses. Therefore, we need to choose the hypotheses before we see the data. We can have a suite of hypotheses, but in that case we will average over the suite.

Talking about hypotheses (or the prior), they are very subjective. Take for example an experiment with a coin, where we have 140 heads and 110 tail after 250 turns. The question now is: Do these data give evidence that the coin is biased rather than fair? The conclusion, as we will see, depends a lot on our definition of “bias”.

We start we the most simple case. Given x the probability of head (0-100), we define the hypotheses:

  • H0: the coin is unbiased, the probability of x = 50 is 100%
  • H1: the coin is biased, for x != 50, the probability of x follows a uniform distribution.

Here the language seems a little bit confusing, so I will add some comments: In the simple case, we suppose that a coin is biased if its probability is not equal to 50, and for other cases, we consider it biased, and the probability of its level of bias is uniformly distributed, which means that the odd of having a 90%-bias coin is as high as the one with 60%-bias. Under these hypotheses, we found that the likelihood ratio is about 0.47, which suggest a favor toward the unbiased hypothesis H0.

We can see that this is not the way we usually define a biased coin in reality. Intuitively, we know that it is very difficult, or even impossible, to find an absolute biased coin (>80%), while the 60%-bias ones are much more popular. In other words, x is not uniformly distributed. Therefore, a more logical way to define our hypotheses is:

The likelihood ratio in this case is .83, still in favor of the hypothesis H0, but it is already weaker than in the first place.

If we decide that everything under 25% or over 75% is impossible, our hypotheses become:

And boom, the likelihood jump up to 1.12, which means that our coin is biased (even though the evidence is quite weak).

To summarize, the Bayes factor depends on the definition of the priors, or the hypotheses. In our example, the evidence is weak either way (between 0.5 and 1.5)


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s