In Bayesian statistics, the data D is in favor of the hypotheses H if p(H|D) > p(D), which means that the posterior is greater than the prior. This is true if p(D|H) > p(D|~H), or p(D|H) / p(D|~H) > 1. The latter is called the likelihood ratio, or the Bayes factor; it measures the strength of the evidence.
We can not use the data to form the hypotheses because this process would be very biased, and thus almost any data would be evidence in favor of our hypotheses. Therefore, we need to choose the hypotheses before we see the data. We can have a suite of hypotheses, but in that case we will average over the suite.
Talking about hypotheses (or the prior), they are very subjective. Take for example an experiment with a coin, where we have 140 heads and 110 tail after 250 turns. The question now is: Do these data give evidence that the coin is biased rather than fair? The conclusion, as we will see, depends a lot on our definition of “bias”.
We start we the most simple case. Given x the probability of head (0-100), we define the hypotheses:
- H0: the coin is unbiased, the probability of x = 50 is 100%
- H1: the coin is biased, for x != 50, the probability of x follows a uniform distribution.
Here the language seems a little bit confusing, so I will add some comments: In the simple case, we suppose that a coin is biased if its probability is not equal to 50, and for other cases, we consider it biased, and the probability of its level of bias is uniformly distributed, which means that the odd of having a 90%-bias coin is as high as the one with 60%-bias. Under these hypotheses, we found that the likelihood ratio is about 0.47, which suggest a favor toward the unbiased hypothesis H0.
We can see that this is not the way we usually define a biased coin in reality. Intuitively, we know that it is very difficult, or even impossible, to find an absolute biased coin (>80%), while the 60%-bias ones are much more popular. In other words, x is not uniformly distributed. Therefore, a more logical way to define our hypotheses is:
The likelihood ratio in this case is .83, still in favor of the hypothesis H0, but it is already weaker than in the first place.
If we decide that everything under 25% or over 75% is impossible, our hypotheses become:
And boom, the likelihood jump up to 1.12, which means that our coin is biased (even though the evidence is quite weak).
To summarize, the Bayes factor depends on the definition of the priors, or the hypotheses. In our example, the evidence is weak either way (between 0.5 and 1.5)