Knowing the limits of our answers

I thought this was a very interesting Wired article about the limitations of the PCL-R.  The author begins with the anecdote of Alan Turing's attempt to answer the question of whether computers could think.  Realizing that the question framed that way was impossible to answer (what does it mean to think?), he reframed the question to be whether computers could pass as humans.  He set up an experiment where test subjects were asked to determine whether responses from an unidentified source came from another human or a computer.

Turing constructed the test in transparently trivial terms. If a computer could fool someone for five minutes 70 percent of the time, it was as good as intelligent. This is powerful not because of its implications for intelligence, but because of its insight into asking tough questions. When we don’t understand the underlying causes of a phenomenon, what scientists call its mechanism, we must resort to studying its effects. But it is crucial that we be aware of the limitations of this approach and remain humble in our inquiry.

He then goes on to compare the difficult misalignment between what we can test and what we hope to learn in terms of the PCL-R and other diagnostic tools for psychopathy:


In the next year, the American Psychological Association will put the finishing touches on the latest version of the Diagnostic Statistical Manual of Mental Disorders, its compendium of psychiatric illnesses. Psychiatry is up against a problem similar to the one Turing faced. The illnesses are complex and their causes hard to discern. Without a clear mechanism, psychiatrists must rely on their patients’ subjective symptoms. It’s a process that’s always fraught, but it works when psychiatrists are realistic about the constraints of their tests.

Things become more troubling when the stakes are high and the diagnoses are tough to change. This is the case in prisons throughout the Western world, where inmates are subjected to the revised Psychopathy Checklist, or PCL-R.

Like the Turing Test, the PCL-R is about effects and symptoms, not causes.

The problems with the PCL-R:


The PCL-R, unlike the Turing Test, is inflexible by design. The Turing Test merely relies on the ability of the machine to be convincing in the present. It doesn’t take into account the machine’s past track record. It leaves open the possibility for change and improvement. The PCL-R is not so forgiving. If a person with a history of psychopathic behaviour were to get better, testers would likely interpret this as deception. After all, deception is a key feature of psychopathy. The PCL-R tries to have it both ways. It relies on observing a set of behaviors, but it resists assigning significance to a change in those behaviors.

Leaving open the possibility of change isn’t about setting serial killers free. But for crimes on the margin, the batteries and assaults and armed robberies, we have to decide whether to deny people who score high on the PCL-R the same opportunities we would give those who score low.

The take away:

The checklist demands that we confront our values. For the possibility of a little more security, are we willing to risk denying a person a second chance? We have to understand the tradeoff and the uncertainty of the reward.

With the Turing Test, it’s pretty straightforward. Five minutes and 70 percent can only tell us so much. How much can the PCL-R tell us?

Alan Turing taught us that when the question is hard, we must know the limits of our answers. At stake here is redemption, the possibility that the wretched can make good. It is an aspiration worth more than a guess. It deserves our humility.


I like this issue about knowing the limits of our answers.  I have recently dipped more into the empirical side of my profession and it has been fascinating and eye-opening to see some of the common mistakes people make in terms of believing that they are "proving" things or that some things are capable of being "known."  There really is a great deal of hubris, and particularly when these pieces of "knowledge" leave the academic area of origin and are used by other people who are unaware of the inherent uncertainty (courts, parole boards, trolls, etc.).  I can understand why people would want to believe things are knowable, particularly when it comes to something as scary as psychopaths, but they just aren't -- at least not currently.