The Prosecutor's Fallacy

Later this month – or it could be next month – a group of three judicial “wise men” in the Netherlands should finally settle the fate of a very unlucky woman named Lucia de Berk. A 45-year-old nurse, de Berk is currently in a Dutch prison, serving a life sentence for murder and attempted murder. The “wise men” – an advisory judicial committee known formally as the Posthumus II Commission – are reconsidering the legitimacy of her conviction four years ago.

Lucia is in prison, it seems, mostly because of human susceptibility to mathematical error – and our collective weakness for rushing to conclusions as a single-minded herd.

When a court first convicted her, the evidence seemed compelling. Following a tip-off from hospital administrators, investigators looked into a series of “suspicious” deaths or near deaths in hospital wards where de Berk had worked from 1999 to 2001, and they found that Lucia had been physically present when many of them took place. A statistical expert calculated that the odds were only 1 in 342 million that it could have been mere coincidence.

Open and shut case, right? Maybe not. A number of Dutch scientists now argue convincingly that the figure cited was incorrect and, worse, irrelevant to the proceedings, which were in addition plagued by numerous other problems.

For one, it seems that the investigators weren’t as careful as they might have been in collecting their data. When they went back, sifting through hospital records looking for suspicious cases, they classified at least some events as suspicious only after they realized that Lucia had been present. So the numbers that emerged were naturally stacked against her.

Mathematician Richard Gill of the University of Leiden, in the Netherlands, and others who have redone the statistical analysis to sort out this problem and others suggest that a more accurate number is something like 1 in 50, and that it could be as low as 1 in 5.

More seriously still – and here’s where the human mind really begins to struggle – the court, and pretty much everyone else involved in the case, appears to have committed a serious but subtle error of logic known as the prosecutor’s fallacy.

The big number reported to the court was an estimate (possibly greatly inflated) of the chance that so many suspicious events could have occured with Lucia present if she was in fact innocent. Mathematically speaking, however, this just isn’t at all the same as the chance that Lucia is innocent, given the evidence, which is what the court really wants to know.

To see why, suppose that police pick up a suspect and match his or her DNA to evidence collected at a crime scene. Suppose that the likelihood of a match, purely by chance, is only 1 in 10,000. Is this also the chance that they are innocent? It’s easy to make this leap, but you shouldn’t.

Here’s why. Suppose the city in which the person lives has 500,000 adult inhabitants. Given the 1 in 10,000 likelihood of a random DNA match, you’d expect that about 50 people in the city would have DNA that also matches the sample. So the suspect is only 1 of 50 people who could have been at the crime scene. Based on the DNA evidence only, the person is almost certainly innocent, not certainly guilty.

This kind of error is so subtle that the untrained human mind doesn’t deal with it very well, and worse yet, usually cannot even recognize its own inability to do so. Unfortunately, this leads to serious consequences, as the case of Lucia de Berk illustrates. Worse yet, our strong illusion of certainty in such matters can also lead to the systematic suppression of doubt, another shortcoming of the de Berk case.

Indeed, de Berk’s defense team presented other numbers that should have created serious doubt in the mind of the court, but apparently didn’t. When de Berk worked on the hospital wards in question, from 1999 to 2001, six suspicious deaths occurred. In the same wards, in a similar period of time before de Berk started working there, there were actually seven suspicious deaths.

If de Berk were a serial killer, it certainly would be bizarre that her presence would lead to a decrease in the overall number of deaths.

Of course, the de Berk case is hardly an isolated example of statistical error in the courtroom. In a famous case in the United Kingdom a few years ago, Sally Clark was found guilty of killing her two infants, largely on the basis of testimony given by Roy Meadows, a physician who told the court that the chance that the two both could have died from Sudden Infant Death Syndrome (SIDS) was only 1 in 73 million.

Meadows arrived at this number by squaring the estimated probability for one such death, which is an elementary mistake. Because SIDS may well have genetic links, the chance that a mother who already had one child die from SIDS would have a second one may be considerably higher.

Here, too, the prosecutor’s fallacy seems to have loomed large, as the likelihood of two SIDS deaths, whatever the number, is not the chance that the mother is guilty, though the court may have interpreted it as such.

Even our powerful intuitive belief that “common sense” is a reliable guide can be extremely dangerous. In Sally Clark’s first appeal, statistician Philip Dawid of University College London was called as an expert witness, but judges and lawyers ultimately decided not to take his advice, as the statistical matters in question were not, they decided, “rocket science.” The conviction was upheld on this appeal (although it was subsequently overturned).

Legal experts in the United States and the United Kingdom are taking some tentative steps to rectify this problem – by organizing further education in statistics for judges and lawyers, and by arranging for the use of special scientific panels in court. Still, it will remain difficult to counteract the timeless process of social amplification that can turn the opinions of a few, based on whatever reasoning, into the near certainty of the crowd.

In the wake of the impressive 1-in-342 million number, the Dutch press piled on de Berk, demonizing her as a cold, remorseless killer. They noted, as if it were somehow relevant, that she had suspiciously spent a number of years outside of the Netherlands, and had even worked for a time as a prostitute. Other “evidence” at the trial was an entry from de Berk’s diary, on the same date as one of the deaths, which said that she had “given in to her compulsion.” Elsewhere she wrote that she had “a very great secret,” which she insisted was reading Tarot cards, but the prosecution alleged, and many people believed, referred to her murdering patients.

What ensued was something akin to the Salem witch hunt. Throughout the trial, Lucia maintained her innocence. But the prosecution called an expert witness who testified that serial killers often refuse to confess. So her protestations became yet more evidence against her.

But now that the evidence has been called into question, social opinion, expressed most clearly in the press, has swung the other way. As Gill, the Leiden mathematician, said to me in an e-mail message, the media suddenly have begun pushing the view that maybe there’s been a miscarriage of justice.

“Suddenly we’re seeing real photos of Lucia de Berk as a normal person,” said Gill, “rather than as a kind of caricature of a modern witch. It’s a fascinating glimpse of group psychology, and a huge change seeded by a little bit of information at the right moment.”

In ordinary usage, “common sense” is taken to be something of value. Albert Einstein had a less charitable view. “Common sense,” he wrote, “is nothing more than a deposit of prejudices laid down by the mind before you reach age 18.”

Our ability as people to understand our habitual failings, both individually and socially, is a great part of what sets us apart from the rest of nature. We excel precisely insofar as we manage to use that ability. Sadly, in the legal setting at least, we still have lots of room for improvement.
_________________

In the case of Lucia de Berk, several Dutch scientists deserve enormous credit for their determined exploration of the way Lucia’s case was handled, and especially for exposing the flawed nature of the statistical arguments. Richard Gill has an extensive summary of the details of the case on the web. Ton Derksen, a Dutch philosopher of science, has written a book critical of the case. Both have submitted presentations to a Dutch committee of legal “wise men” which is now considering whether the case should be reopened.