The power of Bayes

Hello RTU followers!

I was thinking over topics I had never written on, and one of the things that immediately sprung to mind was statistics. This may strike some as odd, since it lies at the heart of modern physics however there is a good and simple reason for this: myself and Mekhi don’t enjoy it as much. Given the broad reach of modern science, I think it is permissible to still say you love science whilst not enjoying an area of it, but in the interests of personal betterment we shall banish the elephant from the room.

In fairness todays topic is quite interesting – we will talk about Bayes Theorem, the work of Rev. Thomas Bayes, who died in 1761 of one of those generic illnesses people seemed to die of in times of poorer sanitation. Bayes Theorem initially seems like a probability tool – however it is much more than that, it gives a way of thinking which I believe to be incredibly healthy – this is what gives it a broader appeal making it worthy of a slot on the blog. Those of you who regularly watch the Big Bang Theory my have even spotted Bayes Theorem making a cameo – which hopefully gives some evidence to suggest this is a big popular area in modern science, rather than some niche bit of stats I decided to torture the readers with.


Some simple, slightly dry ideas

In order to appreciate the appeal of Bayes, you need to first understand roughly how it works. Bayes Theorem is presented in a 15,000 word essay, so there are a lot of different themes to it, but the theme is events and evidence.

A fundamental point to consider is that a test and an event are discrete things. In the example we are going to build we will have an event, which is going to be me falling severely ill with a rare disease. Because I don’t want this to happen, we have come up with a test to check for the illness – which is to take my blood pressure and see if it is highly elevated above a certain threshold. The test isn’t exactly perfect – there are many things which can highly elevate my blood pressure, from a stressful day at work to several coffees so the test could incorrectly diagnose me as severely ill (a false positive). It is also possible, although less likely, that there is another factor lowering my blood pressure on the very same day that gives me a negative reading when I am severely ill, say I have just come back from a relaxing holiday to Rome – this would be a false negative.

I’m sure by now you are screaming GIVE ME SOME NUMBERS, because truthfully you can’t explain as efficiently with words. If I am actually ill, there is a 90% chance that the test will read positive – the other 10% of the time, this is the false negative we spoke about, where my blood pressure has been lowered for another reason. On the other side if I were perfectly well then the test will read negative 95% of the time. There is still a 5% chance that the test reads positive even though I am fine. Clearly we can draw from this is that the test probabilities are not real – there are so many factors which can influence the tests that we must make a distinction between the real probabilities and the test probabilities. The real probabilities do exist, and are features of the universe. The test probabilities will be closer to the real probabilities if we can refine our blunt human instruments. In this example, we are rolling with a 1% chance that I am severely ill, and a 99% chance that I am not. All of these probabilities are pretend.

Joe Ill (1%) Joe Well (99%)
Test Positive 90% 5%
Test Negative 10% 95%

From this table we see that I have a 1% chance of actually being ill, but that does not tell me if I am actually ill or not – I know the probability of having any disease but it still does not tell me if I have it… Only a test can tell me that and tests are flawed.  So the question really is – how good is this test, given the presence of these false negatives? If my test is positive – how worried should I be? The test does not seem that bad, given if I am ill it gets it right 90% of the time. We do some analysis on the table – without getting too mathematical, to combine the probability of two events you multiply them. So if the chance of flipping a coin to heads is 50%, the probability of two heads in a row is 25% – which I can write as 0.5 x 0.5 = 0.25. Here I combine the probability for each event considering the probabilities I am actually ill using the real probabilities.

Joe Ill (1%) Joe Well (99%)
Test Positive The True Positive

0.01 x 0.9 = 0.009

The False Positive

0.99 x 0.05 = 0.0495

Test Negative The False Negative

0.01 x 0.1 = 0.001

The True Negative

0.99 x 0.95 = 0.9405

Next I am going to drop Bayes theorem, which will frighten you, then I will explain it and you will see it is quite cute.


Pr(A|X) means probability of A given X; so here we could say the probability of me being ill (A) given a positive test result, X.

Then we have Pr(X|A)Pr(A) which is quite simple – it’s the probability of X given A, multiplied by the probability of A. So if this is the probability of a positive result, given me being ill multiplied by the probability of me being ill. You can see this result in the top left hand corner, which I have made bold for your convenience. Then what we have  it is just “everything” else that could happen given the positive test result – the probability that I am ill given a positive test result multiplied by the probability of being ill, plus the probability of not being ill given a positive test result multiplied by the probability of not being ill. Those words may seem like a total mess – that’s why we use symbols in mathematics, once you know how to read them life is better.

What is the actual answer? If you crunch the numbers you get 15.4% – what this is saying is that the probability of actually being ill given a positive test result is 15.4%. If that seems obvious to you, then you are a better statistician than me. If it seems a little at odds with what I said earlier, then such is Bayes and why it is so important to run the numbers. The reason that this is so is due to the fact the event itself, me being ill, is so rare. Don’t get lured in by the silly human tests – the real probability was 1%. So even though the test gives the correct result given be being ill 90% of the time, it is so rare that I actually am ill that the false positives occur a lot more. Because there is so much room for false positive results, even after a positive result from the test I am probably fine. Pretty dumb test it seems.

Well done, you stuck with the statistics. I didn’t like writing it and you didn’t enjoy reading it, but let us reward ourselves with something which is quite neat.

Some deeper, juicy ideas

The above is a method that you can compute the probability of something given an event, which is useful. It teaches us interesting things about the crude nature of our tests, and highlights how false positives or negatives can render tests almost useless – in the sense that you can’t even rely on the result. What is really interesting is how Bayes theorem can be used for so many tasks – it can (and does) sort spam emails, help robots to learn and even help you to decide things in your own mind. It is believed that your brain employs many different Baysean algorithms without you even realizing (so you kind of know a lot of this already!).

The power in Bayes Theorem comes from using the best available evidence we have, and then using it to update our belief system – this is highly scientific. In the above example, if we had a positive result we now know that I have a 15% chance of actually being ill. This is far different to the situation we started in, when we were doing a test with a 1% chance of me being ill. If we were to run the test again we could perform the same calculations but instead of the 1% we have 15% – now, assuming the same positive result in the test as the one we had before we can now be 76% sure that I am ill. So by running the test twice, a test which is highly flawed, it becomes much more useful. We would only need to do this test once, or maybe two more times with a positive result before we were comfortable enough to start treatment. This is a much better way of diagnosing people, although it comes with an economic cost. What we have done is powerful – we are only doing the same test over and over, but rather than looking at them in isolation we continually update our belief system to become more confident in our assertions.

You might be forgiven for thinking by this point – hang on this just feels like common sense. I hope you do, because Bayes theorem is just that, a codification of common sense but we can use it to find the truth. Bayes theorem tells us that if people are willing to play the game, we can find the truth about the whole universe – it will always win out. Take the big question, God. Richard for example, who often engages in lively discussion here at RTU firmly believes in God, where as I firmly do not. Neither of us actually have any probabilities God exists, except for the ones we have decided. These are our prior credences. If we both played the game and collected evidence, we could keep updating our belief system in the method of Bayes. We take our prior credence, that God does or does not exist and the probability of it, and then take some evidence/event (say for example a scientific discovery that seems to contradict a holy scripture) and update our beliefs. We would need to understand (in our own minds) how likely is it that this indicator is actually a false positive or negative – just like we did in our example above, so say for example the scientific discovery were actually wrong – since this itself is just based on tests. We update our belief system, and we both have a slightly different view of the world – I maybe am firmer in my belief, or less firm depending on the example and we may have moved our beliefs by different amounts, however if we are being honest we will both be closer in our views.

It all depends on prior credences about certain events and pieces of evidence, coupled with a truly honest refreshing of our belief system. There is a deep philosophical message here, that if you are not thorough about seeking and considering all explanations for your evidence, the truth will be hidden from you and your evidence will only serve to confirm your prejudice. Search for evidence widely, consider all options thoroughly and update your prior credences with untapped honesty and be rewarded with the truth. This is what it’s all about – this is how we decide if God is real, determine if there is a multiverse, if string theory is real or conduct medical testing as we saw earlier! This is the definitive way to think if you want to arrive at the truth.

An additional beauty in this theorem, is the change in direction from Popper’s falsifiability criteria – it means we can consider big theories we cannot falsify and make progress in determining if they are true without dying some big scientific death. Non-falsifiable? No issue – we can just run the Bayes method and get closer and closer to the truth.

I must remind you that with great power, comes great responsibility. If this logic is abused you will arrive at the wrong conclusions. Many humans for example have emotions – they want things to be true and they are unwilling to update their credences in the way that they should. Desperately wanting there to be a heaven does not lend itself well to openly considering all options, which will mean that Bayes theorem will not work and we will confirm our prejudices as discussed. People don’t rigorously consider all alternatives which means that their tests don’t have a complete view of the world. Perhaps saddest of all, since it is hardest to get around, sometimes peoples views are just so polarised and the evidence available is so sparse that a human lifetime is far too short to give enough plays of the game to reveal the truth. This is why it is of vital importance that you learn all the lessons past on from our ancestors, and contribute our own tests to the pot to update the credences of those who call us their ancestors.

The search for the truth is bigger than me, you or any of us. So do the right thing and be a good Baysian, update your credences and pass it on for the good of humanity. A fascinatingly simple embodiment of common sense, that pokes around in some very deep pockets of thought.

14 responses to “The power of Bayes

  1. I don’t know how relevant this might be but this item from the Slashdot site of April 16 may be on topic.

    First, “A new study finds that nearly 9 in 10 people who go for a second opinion after seeing a doctor are likely to leave with a refined or new diagnosis from what they were first told,” according to an article shared by Slashdot reader schwit1:
    Researchers at the Mayo Clinic examined 286 patient records of individuals who had decided to consult a second opinion, hoping to determine whether being referred to a second specialist impacted one’s likelihood of receiving an accurate diagnosis. The study, conducted using records of patients referred to the Mayo Clinic’s General Internal Medicine Division over a two-year period, ultimately found that when consulting a second opinion, the physician only confirmed the original diagnosis 12 percent of the time. Among those with updated diagnoses, 66% received a refined or redefined diagnosis, while 21% were diagnosed with something completely different than what their first physician concluded.

    But in a related story, Slashdot reader sciencehabit writes that four machine-learning algorithms all performed better than currently-used algorithm of the American College of Cardiology, according to newly-published research, which concludes that “machine-learning significantly improves accuracy of cardiovascular risk prediction, increasing the number of patients identified who could benefit from preventive treatment, while avoiding unnecessary treatment of others.”

    “I can’t stress enough how important it is,” one Stanford vascular surgeon told Science magazine, “and how much I really hope that doctors start to embrace the use of artificial intelligence to assist us in care of patients.”

    As you well know, like any hound dog, mathematics leaves me Bayesing at the Moon. So I generally keep circling in the maze towards the Minotaur.

    Liked by 1 person

    • Very relevant indeed and completely true – in fact the Bayesian method for diagnosing illness is far better than many other tests – but of course involve running things several times. I would not be surprised if the future involves all illness being diagnosed through computer algorithms. This is one area however where progress will undeniably be slower, since I think a lot of people take great comfort in a medical professional they can converse with. I am not sure how many will feel about a robot doctor


  2. To simplify this post I think that Bayes’ theory IS a probability tool which you can use it with whatever you want. I would have “liked” the topic; (even if I find it a little “head blowing”), if it wasn’t for putting God in the scale of probability which is inconceivable for me.
    It’s a question of trust. Either you trust something or someone in which you don’t have proof, i.e trusting the unknown, or you don’t; which means you don’t trust what you can’t prove. So you have faith or you don’t, God can’t fit into the Bayes theory.

    Liked by 1 person

    • I would love to agree, but I am afraid I cannot! Your comment about not trusting what you can’t prove is Poppers logic, which is the beauty of Bayes – it moves away from this. If we decide to clip our wings only to that which a human can directly prove then there are many areas of popular science which we must refrain from indulging in. What Bayes does is give the most healthy mindset one can have – it takes anything we cannot truly know about, and it allows us to continually refresh the likelihood of it being true given events as they unfold, in a systematic way which can be used to train both man and machine. The God example is actually an old one – and a textbook favorite on the subject. Anyone who says they are 100% certain God exists or God does not exist is both arrogant and wrong – what we mean is we are beyond reasonable doubt – i.e. we have a very high probability. That does not mean however we should close our mind to any new evidence – same with string theory, multiverses etc etc. We must seek evidence and update our belief system. To me to blindly reject anything we cannot prove, is just as bad as blindly accepting something we cannot prove. You should only really base certainty on that which you can prove.


      • As someone who always found God entertaining and Pogo Possum even more so my universe would be poorer without either one and I find them both as unlikely. What does leave me somewhat startled is that a bright person could believe that God could be outwitted. What kind of a God could that be?


      • Well I think that’s good news! I know some atheists who won’t believe in anything they can’t see. While you tell me that some kind of proof would be enough. That means that some day you might find something which will change your mind!

        I’ve made some research on the holy scriptures looking for scientific proof and I found plenty in the Quran here are some:
        One of the most imposing discoveries of modern science, the expansion of the universe “We constructed the universe with power, and We are expanding it.”

        At the time of Muhammad, botany was not advanced enough in any country for scientists to know that plants have both male and female parts. Nevertheless we may read in one of the chapters “He who made the earth a habitat for you; and traced in it routes for you; and sent down water from the sky, with which We produce pairs of diverse plants.”

        We know as well that fruit come from plants that have sexual characterestics even when they come from unfertilized flowers, like bananas. “ And it is He who spread the earth, and placed in it mountains and rivers. And He placed in it two kinds of every fruit. He causes the night to overlap the day. In that are signs for people who reflect..”

        The evolution of the embryo inside the maternal uterus is only briefly described, but the description is accurate, because the simple words referring to it correspond exactly to fundamental stages in its growth. “I fashioned the clinging entity into a chewed lump of flesh and I fashioned the chewed flesh into bones and I clothed the bones with intact flesh.” or another traduction “Then We developed the seed into a clot. Then We developed the clot into a lump. Then We developed the lump into bones. Then We clothed the bones with flesh. Then We produced it into another creature. Most Blessed is God, the Best of Creators.”
        The term ‘chewed flesh’ corresponds exactly to the appearance of the embryo at a certain stage in its development. It is known that the bones develop inside this mass and that they are then covered with muscle. This is the meaning of the term ‘intact flesh’

        This is a brief demonstration to show that if we look for something we find it.
        I know that you’re always busy with work but I’m sure you can manage to find some time to read as well, so I suggest you to read a book called The Bible, the Quran and Science by Maurice Bucaille.

        In his book, his objective study of the texts, Maurice Bucaille clears’ away many preconceived ideas about the Old Testament, the Gospels and the Qur’an. He tries, in this collection of Writings, to separate what belongs to Revelation from what is the product of error or human interpretation. His study sheds new light on the Holy Scriptures. At the end of a gripping account, he places the Believer before a point of cardinal importance: the continuity of a Revelation emanating from the same God, with modes of expression that differ in the course of time. It leads us to meditate upon those factors which, in our day, should spiritually unite rather than divide-Jews, Christians and Muslims. As a surgeon, Maurice Bucaille has often been in a situation where he was able to examine not only people’s bodies, but their souls. This is how he was struck by the existence of Muslim piety and by aspects of Islam which remain unknown to the vast majority of non-Muslims. In his search for explanations which are otherwise difficult to obtain, he learnt Arabic and studied the Qur’an. In it, he was surprised to find statements on natural phenomena whose meaning can only be understood through modern scientific knowledge. He then turned to the question of the authenticity of the writings that constitute the Holy Scriptures of the monotheistic religions. Finally, in the case of the Bible, he proceeded to a confrontation between these writings and scientific data. The results of his research into the Judeo-Christian Revelation and the Qur’an are set out in this book.”

        You may download it for free here:


      • That Muslims, or anybody else could observe nature correctly has very little to do with presuming fundamental conceptions of underlying causes. My door is always open to a visit by a real god and a cup of coffee and a home baked slice of chocolate cake is ready as well. But these persistent aliens from flying saucers who have become aware of Arthur Clarke’s comment that advanced science is easily confused with miracles so frequently try to pull the wool over us Earthling’s eyes that I have become rather suspicious.


  3. Hey there Joseph! You’re blogs are brilliant! I’ve seen that you liked one of my blogs about Pauli’s exclusion principle and combined wave functions(My only blog xD) .I have started reading some of the blogs you’ve written , and I was amazed at the way you explain stuff. I’ll be writing daily blogs since I’ve just finished 10th grade. I have a lot to learn from you and hope that you continue writing. Thanks for the lovely post!


    • Thank you very much! I am glad you found the blog and that you have also found a passion for writing. Keep it up and I look forward to following!


  4. Pingback: THE POWER OF BAYES | vyagers·

  5. Pingback: Who should educate? | Rationalising The Universe·

  6. Pingback: Who should share information?·

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s