Contributors

Friday, April 24, 2015

The Truly Dismal Science

There are a lot of cop shows on TV. There have been four versions of Law & Order (vanilla, SVU, LA, Criminal Intent and UK). There've been four versions of CSI (vanilla, New York, Miami and Cyber). There've been three versions of NCIS (vanilla, Los Angeles and New Orleans), which completely baffles me: why is the Navy so riddled with crime, and anyway, doesn't the Posse Comitatus Act prevent NCIS officers from messing around in civilian cases? (Turns out, it does.)

According to an analysis on Details.com, 42% of all the characters on American television are cops (including FBI agents, forensic analysts, medical examiners, secret agents). And that's not even counting vigilantes like the Arrow and the Winchester brothers who fight superhuman and supernatural threats.

Now, I'm not going to bemoan the fact that so much of our entertainment is focused on crime, or that average American children see 16,000 murders by the time they're 18, or that seeing this endless parade of death and destruction gives us a warped view of the world, making us think things are far more dangerous than they really are.

For example, child abductions by strangers are much rarer than you'd think based on TV shows. Most kids are taken by relatives:
Only a tiny minority of kidnapped children are taken by strangers. Between 1990 and 1995 the National Center for Missing and Exploited Children handled only 515 stranger abductions, 3.1 percent of its caseload. A 2000 report by the Office of Juvenile Justice and Delinquency Programs reported that more than 3/4 of kidnappings were committed by family members or acquaintances of the child. The study also found that children abducted by strangers were harmed less frequently than those taken by acquaintances.
But perhaps the most misleading aspect of TV cop shows is the accuracy of forensic analysis. Crime scene investigators (CSIs) are wrong a heck of a lot more than the TV shows make you think.

Admittedly, I've always been skeptical of how quickly and easily the guys on CSI identify which implement was was used to hack someone up based on tool marks. Or whether it was this or that suspect based on teeth marks. Or matching hair samples or bullets in a microscope, just by eyeballing it.

FBI forensic hair examiners gave flawed testimony in 95% of the cases examined.
It turns out that my skepticism was warranted. As reported by the Washington Post, an FBI investigation of 268 cases determined that the forensic hair testimony in 257 of those cases was flawed, or an error rate of more than 95%. Worse, 32 of those were death penalty cases. In one of them they matched human hair to a dog.

For something supposedly straightforward like ballistics, the error rate is variously claimed to be 1.2% or 10% or not really known. Given that bullets that hit people are pretty smashed up, the error rate will intrinsically be quite high. Ditto for fingerprints: various studies have found false negative results as high as 10%, though fortunately the false positive rate is typically much less than 1%. DNA testing still boils down to an analyst sitting down and comparing a couple of blotches on a printout.

The cop shows like to emphasize that crime scene analysis is Science!. But that's a mischaracterization of what's really going on. CSIs are technicians who use scientific tools like microscopes, DNA analysis and mass spectrometers to gather information about crime scene evidence, which they then interpret.

They need technical training to make sure that they properly prepare samples for analysis, avoid cross-contamination and understand the limitations of their measurements.

But that doesn't make the real task they perform "science," any more than using a pH tester makes the guy who installed your water softener a scientist, or your accountant an electronic engineer because he uses a calculator.

Crime scene analysis is an interpretive art, not a science.
Analyzing the data from testing of crime scene evidence is an interpretive art, not science. Scientists run statistical analyses on large samples of data. They conduct experiments over and over again, or verify the results of other scientists' experiments. They formulate new theories based on mathematical models of matter and the universe, and then predict results of experiments before they do them. They control as many variables at once and then vary them one at a time to make sure they're really measuring what they they think they're measuring.

The guys at Mythbusters are more like scientists than crime scene analysts.

And that's not CSIs' fault. They have to decide who committed a crime based on one or two tiny bits of evidence. Their samples are frequently contaminated by victims and paramedics. They can't reproduce results by getting the criminal to commit the crime multiple times. Their job is intrinsically difficult, made more so by the demands of the justice system and the expectations of victims.

The scientific method can be utilized to determine the accuracy of the methods CSIs employ -- experiments and statistical analyses can be run on the performance of these experts. But that's only marginally useful: molecules and atoms and photons follow the laws of chemistry and physics, and always behave the same way under the same conditions.

Human beings don't: they get distracted, bored, tired. Their vision gets blurry from lack of sleep and dirty contacts. They go through divorces. They're influenced by the detectives in the case, or the brutality of the crime. They have large backlogs of cases and are in a hurry to catch up. From the results of the FBI's analysis of hair forensics, it's clear that these factors have as much influence on crime scene analysis as any underlying science.

If crime scene analysis is a science, it -- rather than economics -- is truly the dismal science.

No comments: