Prologue: A few murder-mysteries
Case 1: The Wayne Williams Murder Trial
Between 1979 and 1981 there were a string of 28 African Americans, mostly children, were killed in Atlanta, USA. The events have since become known as the Atlanta Child Murders.
Wayne Williams was convicted for two of the murders, and was charged with life in prison. There were no convictions for any of the other murders. Williams currently remains in prison, and maintains his innocence.
What got Williams convicted was carpet fibers. Carpet fibers were found in a victim's hair matching those at Williams' home. Those particular carpet fibers turned out to be rare: only an estimated 82 out of 638,992 Atlanta homes had this kind of carpet fiber.
The prosecution jumped on this evidence. They pointed out that the chance of finding those carpet fibers Williams didn't commit the crime is about 81/638,992, or 1 in 8,000.
However, little further evidence was available against Williams. No other common fibers were found, although he was seen to be driving by one of the murder sites around the time that it happened, and some spotty witness testimony suggested he might have known one two of the victims when they were alive.
Mystery question: Suppose you were on the jury. Would you have convicted Williams?
Case 2: Regina v. Adams
In 1991, a woman in her twenties was walking home late at night near London when someone approached her, asked for the time, and then overpowered and raped her. Semen found on her body afterward was analyzed, and a DNA profile was produced.
In 1994, Denis John Adams was arrested for a different offense, but a routine DNA check showed that his profile matched that for rape case.
So, in 1995, Adams was tried and convicted of the 1991 rape. What is interesting is that the case was decided almost entirely on the basis of DNA evidence. Adams' DNA profile turned out to be a positive match for the DNA found on the victim. According to the forensic team, there was a 1 in 27 million chance that a random citizen of the United Kingdom would have this profile.
A small problem was that none of the other evidence implicated Adams. The victim said her attacker was 20-25, whereas Adams was in his mid-thirties. She also was not able to identify in a criminal lineup, and even said he did not look like her attacker. Adams also had an alibi, that he was with his girlfriend that night, which she confirmed.
However, the DNA evidence was considered strong enough. It was possible to discount the other evidence on the basis of witness unreliability, which is a well-known phenomenon.
Mystery Question: Suppose you were on this jury. Would you have convicted Adams?
Two easier mystery questions
Case 3: The pregnancy test
Jamie is preparing for a night out drinking at the clubs with friends, but on a chance decides to take a pregnancy test.
It's an extremely accuracy test, with less than one in a million false positives and false negatives. (A false positive is a positive reading on the test when it should have read negative; a false negative is a negative reading when the test should have read positive.)
After several minutes of nervous waiting, the test reading comes back: positive. Horrified, the test drops from Jamie's hand and clatters upon the floor. It's decision time.
Mystery Question: How soon should Jamie stop going out drinking with friends and start thinking about the pregnancy?
Our Project
Both of these cases commit a fallacy known as the base rate fallacy.
You will soon understand how to recognize this fallacy, so you can avoid it. You may also become convinced that at least one of the two cases above was a case of wrongful conviction!
Case 4: The hemophiliac mutation
Hemophilia or hemophiliac disease is a condition in which one lacks the protein required for blood to clot. As a consequence, hemophiliacs can experience life-threatening bleeding from even a small cut, scrape or bruise.
A particular mutation of a gene called FVIII is responsible for hemophilia, and is straightforward to identify. So, a genetic test provides a very accurate way to determine whether or not a person has the hemophilia mutation. The presence of the mutation indicates a very high probability of hemophilia.
Now, suppose a woman named Queen V takes this test and determines that she has the mutation. People with hemophilia must take special steps to prevent injury, such as taking special medicine, using only padded furniture that is safe to bump up against, and not engaging in activities that could result in a scrape or cut.
Mystery question: How soon should the Queen start preparing to live with hemophilila?
The case of the missing information
It is tempting to jump to conclusions in both the pregnancy case and the hemophilia case, since both involve extremely accurate tests. But both lack important information that is relevant to determining how we should react.
Can you think of what information is missing?
Suppose that in the pregnancy case, it turned out that our character Jamie was a man by birth. Then would you find a positive result on a highly-accurate pregnancy test convincing?
Of course you wouldn't. The test is accurate with respect to a base rate in a population that does not include men. No matter how accurate the test happens to be on the general population, when we restrcit attention to people who are born as men, the proability that they are pregnant is zero.
Similar base-rate neglect occurs in the hemophilia case, as it turns out that hemophilia only presents itself in men except under very unusual circumstances. The fact that Queen V has the mutation does not imply she is likely to get the disease, since she is not a member of the appropriate base population.
This was in fact the case with Queen Victoria, who famously carried the hemophilia gene but did not have hemophilia, although she passed it on to many of her male descendents, including Prince Leopold, who died of complications of bruising when he fell down some stairs.
The Base Rate Fallacy
All of the examples we have expressed fall prey to the base rate fallacy, sometimes called the prosecutor's fallacy. This fallacy occurs when one ignores the underlying prior probability with which a hypothesis occurs. (That probability is often called the "base rate.")
So, for example, our original the pregnancy judgement ignored the prior probability of the hypothesis that Jamie is pregnant, that is, prior to taking the test. That probability turned out to be zero (since Jamie was male), which resulted in a dramatic change in the outcome of our prediction. A similar base rate was ignored in the case of hemophilia.
But a similar prior probability was ignored in the Williams murder trial, namely, what is the prior probability without any of the fiber evidence that Williams committed the murder?
And in the case of the Adams rape trial: what is the prior probability without any genetic evidence that Adams committed the crime?
It may be the case these prior probabilities are not zero. But if either one is low, then it can dramatically change our prediction about the outcome. Indeed, if it is low enough, then our judgement may change to support an "innocent" verdict instead of a guilty one.
Probabilities and Evidence
To see how this works, we will need to say some minimal things about probabilities.
Let's introduce some variables. Refer to a hypothesis like "Williams is the murderer" using the letter "H" (for "Hypothesis"). Refer to evidence like "Carpet fibers matching those in Williams' home were found on the victim" with the letter "E."
\(H\): stands for "Hypothesis"
\(E\): stands for "Evidence"
In order to talk about how likely these things are, we need to say what the "Probability of the hypothesis \(H\)," and the "Probability of the hypothesis \(H\) given some evidence \(E\)." We'll write that:
\(\Pr(H)\): means "The probability of \(H\)"
\(\Pr(H|E)\): means "The probablity of \(H\) given \(E\)."
We will take all probabilities to be expressed by some value of 0 and 1. So, for us, 1 means that there is a 100% chance, and 0 means that there is no chance at all.
Why would we want to talk in this way? Because then we can use the theory of probability to talk about our testing examples. And why would we want to do that? Because probability is extremely reliable. It is our best mathematical tool for calculating what to do in situations of uncertainty. It's worth betting on probability, since it is this mathematics that made the gambling industry successful.
The aspect of probability theory that is used for describing conditional situations, such as the probability of a hypothesis some evidence, is known as Bayes' Theorem. This theorem is just a mathematical fact about probabilities. It is usually stated as the equality,
\[\Pr(H|E) = \frac{\Pr(E|H)\cdot\Pr(H)}{\Pr(E)}.\]
However, for us it is more useful for us to express it in the following mathematically equivalent way.
\[\Pr(H|E) = \frac{1}{1+\frac{\Pr(E|\neg H)}{\Pr(E|H)}\cdot\frac{\Pr(\neg H)}{\Pr(H)}}\]
where I have written \(\neg H\) to represent "Not \(H\)."
The most important thing to recognize about this is that, to know the probability of a hypothesis given some evidence, we need to know four things.
- \(\Pr(E|\neg H)\): The probability of the evidence given that the hypothesis is false.
- \(\Pr(E| H)\): The probability of the evidence given that the hypothesis is true.
- \(\Pr(\neg H)\): The prior probability that the hypothesis is false, before the evidence \(E\) is even considered.
- \(\Pr(H)\): The prior probability that the hypothesis is true, before the evidence \(E\) is considered.
Most importantly, notice that this is a law of probability theory that says we cannot ignore the base rate. The probability of a hypothesis given some evidence \(\Pr(H|E)\) is only meaningful once we know the prior probability or "base rate" \(\Pr(H)\).
It's easier to consider these things using concrete examples. So, let's go back to our murder-mysteries.
Probabilities in our Murder Mysteries
We can our hypotheses and our evidence now using the language of probability theory. Let's take the following hypothesis and evidence:
\(H = \) "Wayne Williams committed one of the Atlanta murders"
\(E = \) "Fibers matching Williams' carpet were found in the victim's hair."
Then, the million-dollar question in the court case against Williams can be formulated, what is the value of \(\Pr(H|E)\)?
The claims made by the prosecution in the Williams case correspond to the term on the bottom-left of the fraction:
\(\Pr(E|\neg H) = 1/8000\), i.e., the probability of finding those fibers on the victim (the evidence \(E\)) if Williams didn't commit the murder (the negation of the hypothesis, \(\neg H\) is only 1 in 8000.
\(\Pr(E|H) = 1\), i.e., the probability of finding those fibers on the victim (the evidence \(E\) given that Williams did commit the murder is high. In fact it could be smaller than one, like say 0.5 (a 50% chance). This will not change the result of our calculation as long as \(\Pr(E|H)\) takes a value that is large compared to 1/8000.
The important thing to notice is that you cannot know the probability of a hypothesis given some evidence until you have specified the prior probability
When the prosecution says it doesn't matter, they are saying that the prior probability of \(H\), the hypothesis that Williams committed the crime, is 0.5, i.e. he would have killed the kids 50-50.
This kind of claim is hardly fair to Williams. Moreover, it is simply false if \(\Pr(H)\) is sufficiently low; then the probabilities would indicate that he did not commit the crime.
The same thing can be said of the Williams case. As an exercise, you can practice working out how.