Bayesian Probability

Bayes' Theorem: What's It All About?

Bayes’ Theorem might seem complex due to the similarity of the terms used. It contains four probability values which are $P(H)$ , $P(H|E)$ , $P(E|H)$ and $P(E)$ . Let's break it down:

$P(H|E)= \frac{P(E|H)P(H)}{P(E)}$ ...(1)

The key to understanding the theorem is in grasping the relationship between $P(H)$ (pre-probability) and $P(H|E)$ (post-probability).

The Essence of Bayes' Theorem

Bayes’ Theorem is all about updating the confidence or credibility of an event having occurred, based on new information.

Why is Bayes' Theorem Often Misunderstood?

Understanding Bayes' Theorem requires shifting our traditional perspective of 'probability'. If we think of probability as 'confidence in an argument', we align ourselves more with Bayesianism than frequencyism.

Terminology

In addition, let's find out what $E$ and $H$ are in the formulas in Equation (1).

In Equation (1), $H$ stands for Hypothesis and means a hypothesis or “argument that an event has occurred.”

In addition, in Equation (1), $E$ stands for Evidence and means 'new information'.

Therefore, $P(H)$ means the reliability of the claim that an event has occurred, and $P(H|E)P(H|E)$ means the reliability updated after receiving new information.

And $P(H)$ and $P(H|E)$ are called pre-probability and post-probability, respectively, and if you think of the words pre- and post-probability, you can understand what happens, that is, ”evidence, and trust in my claim before and after updating it.”

Figure 1. Bayes' theorem is a summary that explains the relationship between pre-probability and post-probability.

From Deductive to Inductive Inference: A Paradigm Shift

The real game-changer with Bayes' Theorem is that it brought a fundamental paradigm shift to statistics. Unlike traditional statistics which relies on deductive reasoning, Bayesian statistics is based on updating prior probabilities using additional information. This is an inductive reasoning method.

Understanding Bayes' Theorem through Problem Solving

Let's solve a problem to see how Bayes Theorem updates the credibility of an event based on new information.

Example

Let's say disease A has an incidence rate of 0.1%. A test for this disease has a sensitivity of 99% and a specificity of 98%. If a person tests positive for this disease, what's the probability they truly have it?

Solution

Defining Hypothesis as the person having the disease and Evidence as the test result, the initial probability or pre-probability ( $P(H)$ ) is the incidence rate, 0.001. The test's sensitivity and specificity correspond to the True Positive ( $P(E|H)= 0.99$ ) and True Negative ( $P(E^c|H^c)=0.98$ ) rates respectively.

Figure 2. Visualizing the values of the given probability values as the relative area of the rectangle.

Using equation (2):

$P(H|E)= \frac{P(E|H)P(H)}{P(E|H)P(H)+P(E|H^c)P(H^c)}$ ...(2)

The updated probability or post-probability ( $P(H|E)$ ) is then:

$P(H|E)= \frac{0.001×0.99}{0.001×0.99+0.999×0.02}=0.047$ ...(3)

This means that there's a 4.7% probability that the person truly has the disease despite testing positive.

Example 2

What if the same person tests positive for the disease a second time? What's the probability they truly have it now?

Solution

In this case, the first test's post-probability becomes the second's pre-probability. This reflects the iterative nature of Bayesian inference where prior probabilities are continually updated with new evidence.

Figure 3. The posterior probability calculated in Example 1 is given back to the prior probability in Example 2 (red arrow), and the updated posterior probability can be calculated once more. Figure visualizes the values of a given probability values as the relative area of a rectangle.

Reapplying equation (2), the updated post-probability is:

$P(H|E)= \frac{0.047×0.99}{0.047×0.99+0.953×0.02}=0.709$ ...(4)

This means that the confidence that the person truly has the disease increases to approximately 70% after the second positive test.