Bayes' Theorem

Bayes can do magic!

Ever wondered how computers learn about people?

Bayes' Theorem is a way of finding a probability when we know certain other probabilities.

The formula is:

P(A|B) = P(A) P(B|A)P(B)

Let us say P(Fire) means how often there is fire, and P(Smoke) means how often we see smoke, then:

P(Fire|Smoke) means how often there is fire when we can see smoke
P(Smoke|Fire) means how often we can see smoke when there is fire

So the formula kind of tells us "forwards" P(Fire|Smoke) when we know "backwards" P(Smoke|Fire)

Home Bonfire

Example:

• dangerous fires are rare (1%)
• but smoke is fairly common (10%) due to barbecues,
• and 90% of dangerous fires make smoke

We can then discover the probability of dangerous Fire when there is Smoke:

P(Fire|Smoke) =P(Fire) P(Smoke|Fire)P(Smoke)

=1% x 90%10%

=9%

So it is still worth checking out any smoke to be sure.

Just 4 Numbers

Imagine 100 people at a party, and you tally how many wear pink or not, and if a man or not, and get these numbers:

Bayes' Theorem is based off just those 4 numbers!

Let us do some totals:

And calculate some probabilities:

• the probability of being a man is P(Man) = 40100 = 0.4
• the probability of wearing pink is P(Pink) = 25100 = 0.25
• the probability that a man wears pink is P(Pink|Man) = 540 = 0.125
• the probability that a person wearing pink is a man P(Man|Pink) = ...

And then the puppy arrives! Such a cute puppy.

But all your data is ripped up! Only 3 values survive:

• P(Man) = 0.4,
• P(Pink) = 0.25 and
• P(Pink|Man) = 0.125

Can you discover P(Man|Pink) ?

Imagine a pink-wearing guest leaves money behind ... was it a man? We can answer this question using Bayes' Theorem:

P(Man|Pink) = P(Man) P(Pink|Man)P(Pink)

P(Man|Pink) = 0.4 × 0.1250.25 = 0.2

Being General

Why does it work?

Let us replace the numbers with letters:

Now let us look at probabilities. So we take some ratios:

• the overall probability of "A" is P(A) = s+ts+t+u+v
• the probability of "B given A" is P(B|A) = ss+t

And then multiply them together like this:

Now let us do that again but use P(B) and P(A|B):

Both ways get the same result of ss+t+u+v

So we can see that:

P(B) P(A|B) = P(A) P(B|A)

Nice and symmetrical isn't it?

It actually has to be symmetrical as we can swap rows and columns and get the same top-left corner.

And it is also Bayes Formula ... just divide both sides by P(B):

Remembering

First think "AB AB AB" then remember to group it like: "AB = A BA / B"

P(A|B) = P(A) P(B|A)P(B)

Cat Allergy?

One of the famous uses for Bayes Theorem is False Positives and False Negatives.

For those we have two possible cases for "A", such as Pass/Fail (or Yes/No etc)

We want to know the chance of having the allergy when test says "Yes", written P(Allergy|Yes)

Let's get our formula:

P(Allergy|Yes) = P(Allergy) P(Yes|Allergy)P(Yes)

• P(Allergy) is Probability of Allergy = 1%
• P(Yes|Allergy) is Probability of test saying "Yes" for people with allergy = 80%
• P(Yes) is Probability of test saying "Yes" (to anyone) = ??%

Oh no! We don't know what the general chance of the test saying "Yes" is ...

... but we can calculate it by adding up those with, and those without the allergy:

• 1% have the allergy, and the test says "Yes" to 80% of them
• 99% do not have the allergy and the test says "Yes" to 10% of them

Let's add that up:

P(Yes) = 1% × 80% + 99% × 10% = 10.7%

Which means that about 10.7% of the population will get a "Yes" result.

So now we can complete our formula:

P(Allergy|Yes) = 1% × 80%10.7%  = 7.48%

P(Allergy|Yes) = about 7%

This is the same result we got on False Positives and False Negatives.

In fact we can write a special version of the Bayes' formula just for things like this:

P(A|B) = P(A)P(B|A) P(A)P(B|A) + P(not A)P(B|not A)

"A" With Three (or more) Cases

We just saw "A" with two cases (A and not A), which we took care of in the bottom line.

When "A" has 3 or more cases we include them all in the bottom line:

P(A1|B) = P(A1)P(B|A1) P(A1)P(B|A1) + P(A2)P(B|A2) + P(A3)P(B|A3) + ...etc