Accuracy and prediction… the limits of our intuition
Imagine that an infectious disease is spreading around the world, and that a test is available, with for example a proven accuracy of 90%. Ask yourself the following question:
What is the likelihood that I am infected, if the test is positive?
Usually, people answer that the probability is 90%, that is, equal to the accuracy of the test. But this answer is wrong and betrays our difficulty in reasoning correctly with probabilities.
In reality, the probability in question could be any number between 0% and 100%!
Now I’ll explain. Before doing so, however, a clarification is necessary. A test has two types of accuracy. One that allows it to detect infected people, which is called ‘sensitivity’, and one that allows it to detect non-infected people, which is called ‘specificity’. But to simplify the discussion, we can here consider that the 90% accuracy of the test in question means that both its sensitivity and its specificity are 90%.
So, how is it possible that with a 90% accurate test, the probability that a person is infected, when the test response is positive, can be any percentage, between 0% and 100%?
It’s simple. It is because we usually forget that this probability depends on how many infected persons are present in the population.
To understand this, we must first remember that the probability in question is a conditional probability’. In fact, we are looking for the probability that the person is infected (in), conditional on the fact that the test result is positive (+). Let us denote this probability P(in|+), as is customary in probability theory.
Now we must remember that the conditional probability of the event “in”, knowing that the event “+” is realized, is given by the probability that “in” and “+” are simultaneously realized, divided by the probability that “in” is realized. With obvious notation, we therefore have the following formula:
P (in|+) = P (in & +) / P(+).
In the literature, the probability P(in|+) has a name, it is called “positive predictive value”, often indicated with the acronym PPV. To obtain a more explicit formula, the idea is to further decompose the joint probability P(in & +), noting that we can also write:
P (+|in) = P(in & +) / P(in).
P (in & +) = P (+| n) * P(in),
and inserting this expression into the one above, we get:
P (in|+) = P (+|in) * P (in) / P(+).
At this point, we can further break down P(+), writing:
P (+) = P (+|n) * (in) + P (+|nin) * P(nin),
where “nin” stands for “not infected”.
This formula, which I just used, it is called the ‘law of total probabilities’, but apart from this high-sounding name, it is something very intuitive. It simply expresses the fact that the sum of the probabilities of incompatible partial events is equal to the total probability of the event in question.
Of course, the probability of not being infected, P(nin), is simply 1 minus the probability of being infected, so:
P(nin) = 1 - P(in).
Furthermore, the probability of being positive when infected, P(+|in), is nothing more than the ‘sensitivity’, which for convenience we will denote ‘Se’. Instead, the probability P(+|nin), of being positive when not infected, is equal to 1 minus the probability of being negative when not infected, which is what we have termed ‘specificity’, and will denote ‘Sp’. Therefore:
P(+|nin) = 1 - Sp.
In other words, we have made the following steps explicit:
= P(in & +) /P(+)
= P(+|in) * P(in) / P(+)
= P(+|in) * P(in) / [P(+|in) * P(in) + P(+|nin) * P(nin)]
= Se * P(in) / [Se * P(in) + (1 - Sp) * (1 - P(in))].
So, this is the formula. Let us suppose, to simplify, that Se = Sp, i.e., that the precision in detecting the infected is the same as detecting the non-infected, which we will denote S. And to further simplify the notation, we simply write P for the probability P(in), which obviously, in the absence of other information, is simply given by the ratio between the number of infected people and the total number of people. We can then write:
P(in|+) = S * P / [S * P + (1 - S) * (1 - P)].
We can easily verify that if P tends to zero, the numerator of the above formula also tends to zero, therefore P(in|+) tends to 0. This means that if there is an insignificant number of infected people in the population, this probability will also be insignificant.
Moral: even a very accurate test will be of no predictive power if the number of infected people in the population is too low.
If, instead, P tends to 1, the numerator tends towards S, and the denominator also tends towards S, therefore P(in|+) tends to 1. This means that if the number of infected people in the population is maximum, the probability of being infected.
Moral: even a very inaccurate test can have great predictive power if the number of infected people in the population is extremely high.
Ok, but since we have derived a formula, we can now use it to ask ourselves: what must be the percentage of infected in the population, in order for a 90% accurate test to be able to give the correct positive answer at least in 90% of the times? Well, it is easy to deduce from the formula we have derived that at least 50% of the population must be infected!
On the other hand, if, for example, only 10% of the population were infected, then the predictivity of the test would drop to 50%! In this situation, to have a 90% predictivity, one needs the test to increase its precision up to 98,8%.
In short, when dealing with probabilities, and with scientific reasoning in general, our intuition is of little use.
In fact, it is necessary to activate a slow reasoning process and carefully check all its logical steps.
And in this era where everyone is losing one’s mind, perhaps for the first time some will begin to answer that big question that every mathematics teacher has been asked countless times: “What the hell is math for?”