Advertisement

The question sent to the lab seemed relatively simple. But the answer was incredibly complex.

A ski mask left behind after a bank robbery was the piece of evidence. The fabric showed a mixture of touch DNA including four people, but due to its complexity, it initially appeared as a mixture of only two people. The labs were given two of the four likely contributors, along with a fifth person.

But that fifth person was not in the mixture, and had never touched the ski mask.

It’s not a real case. It’s a hypothetical, a test; part of an inter-laboratory study involving more than 100 labs conducted by the National Institute of Standards and Technology.

Seventy percent of the labs got the wrong answer, by including that fifth, and innocent, person within the mixture.

The results showed that, as DNA sensitivity has grown by leaps and bounds, interpretation of those tiny clues has not kept pace. But those results were reached five years ago—and the study has not yet been published in a peer-reviewed journal. NIST says they have pushed out the results to other experts at major forensic conferences in the ensuing years—and now plan to submit a full paper to a major journal by the end of this month.

Critics say that justice has been deferred by the years-long delay, potentially landing untold numbers of people in prison based off faulty DNA mixture interpretation methods. Some experts said they were unaware of the potentially fortune-reversing findings—and others said the results were turned aside by judges because there was no official publication of the findings.

Just how many criminal cases were impacted by these mixture interpretation methods during the last five years—and even earlier than that—remains to be seen.

“This is about five years of people being convicted by bad interpretations,” said Greg Hampikian, a Boise State DNA expert. “This is a huge story. It’s the problem with DNA—the one everyone trusts.”

CASE FIVE

The study is known as MIX13. It was the latest of NIST studies about DNA interpretation, including one in 2005 called MIX05.

MIX13, conducted in 2013, remains the biggest to date. It involved five cases, with two-person, three-person and four-person mixtures that grew increasingly complex—and challenging.

Case One involved a two-person mixture from a sexual assault; Case Two involved a mock handgun with a three-person mixture; Case Three was a three-person mixture from a hypothetical sexual assault involving two minor contributors who were brothers; Case Four was a theoretical sexual assault showing two contributors to the genetic mixture—but at a challenging level that would see if labs tried to distinguish a major from a minor contributor.

Then came Case Five.

The four-person mixture involving equal amounts of genetic material was made to be difficult. Seventy-six laboratories out of 108 got it wrong by including the fifth person in their interpretation. Most were using the method of combined probability of inclusion, otherwise known as CPI, an FBI-approved method of separating out mixtures.

Twenty-two labs deemed the results “inconclusive” in including the three suspects. Three additional labs found it inconclusive for the fifth, and innocent, person—while still including the other two correctly.

Only seven laboratories got the hard problem totally “right”—by correctly excluding the fifth and innocent person from the four-person mixture. But even the reasons they cited were different. Four of the laboratories cited a missing allele at a key location. Two more, using data from the Identifiler Plus (a ThermoFisher PCR amplification kit), showed that the fifth person could not fit.

A PowerPoint slide showing results from the MIX13 study conducted by NIST researchers in 2013. (Credit: Courtesy of NIST)

One used probabilistic genotyping—in particular a software program called TrueAllele that uses big-data statistical interpretations to understand likelihoods of someone being in the alphabet soup of genes.

John Butler, the special assistant to the Director for Forensic Science in NIST’s special programs office, told Forensic Magazine in an interview that the results of MIX13 were an inter-laboratory study. The tests, including “Case Five,” were a way to not only gauge how labs were doing with their mixtures—it was also to provide a “teaching moment.”

“The mixture itself was designed to not show too many alleles,” Butler called. “People would be tricked into thinking there are only two or three people there, instead of the four people that were really there.

“The way that it was designed was on purpose, to kind of help people realize that CPI can falsely include people—that was its purpose,” he added. “And it demonstrated that really nicely.”

One cannot draw real-world conclusions from Case Five, Butler added in the interview.

“We asked specific questions of labs, and part of it was a teaching moment,” he added. “It wasn’t to say, here’s your error rate—because that’s not what the purpose of that was. This was a teaching moment to realize you can falsely include somebody with CPI.”

But some in the field have taken the results literally, and seriously.

They said the results show a huge problem exists: that casework has been inaccurate for years—with real-world consequences.

Potentially thousands of innocent people could be behind bars, based on faulty mixture interpretation, according to critics like Hampikian.

“Out of the 76 labs that got this wrong, don’t you think some of them over the past five years—or even before—were doing the same things with actual casework?” said Hampikian. “Is there any reason to believe they were not doing the same thing with casework?"

A FIVE-YEAR DELAY

When Forensic Magazine asked about publication, NIST said it was going ahead with official publication of MIX13.

Butler said the paper has been drafted for peer review publication, and the 41-page paper is currently under review at NIST. It will be submitted by the end of this month, he said—likely to the journal Forensic Science International: Genetics. It’s a publication where many other studies of DNA mixture problems have been published.

“Hopefully by the summertime it will be fully available to everybody,” said Butler.

NIST and Butler point to available PowerPoint slides that were presented at forensic conferences starting in 2014 by both Butler and the co-author of the study, Mike Coble, then a NIST scientist.

They included a presentation by Coble at the American Bar Association’s forensic conference in June 2014 in New York City, as well as a meeting of the American Society of Crime Laboratory Directors/Laboratory Accreditation Board in January 2015, by both Butler and Coble.

An annotated chart included in a PowerPoint about Case 5 of NIST's MIX13 study. (Credit: Courtesy of NIST)

But what has driven the criticism of Hampikian and some other forensic experts is what they call the inaccessible nature of the study results.

Without peer review these last five years, MIX13 has not been admitted into court rooms, despite attempts by defense teams, said some experts.

PowerPoint slides are far from peer-reviewed results, according to Hampikian. So while the results were presented to forensic laboratory and DNA experts, the results were essentially “hidden” from the public, he contends.

“It’s like discovering the airbags are bad, and only telling Toyota,” said Hampikian.

When asked about why there were years of delay, Butler pointed to Coble.

“It was Mike Coble’s project—in terms of why it didn’t get written up faster, you can talk to him about that, if you want,” he said.

Coble, the designer of the 2005 and 2013 studies with Butler, has since left NIST and moved on to a job with the University of North Texas Health Science Center, one of the most well-known DNA labs in the country. He responded to a pair of emails from Forensic Magazine.

“I am quite busy and excited with my new position,” said Coble. “I am glad to see the publication of the study is moving forward.

“I think it’s hard to extrapolate that based on the 2013 and 2005 studies that people are being erroneously included in mixtures,” Coble added. “That would require a much different investigation than a simple interlab study.”

TWO CRIMINAL CASES

The controversial “Case Five” of MIX13 has a real-life analog in Missouri: the tale of Marlo Hodges, currently prisoner number 164381 at the Southeast Correctional Center in Charleston.

Hodges was convicted of a first-degree robbery and burglary committed in 2011. The evidence presented at trial (four years later) involved two witnesses: one who said Hodges was not the attacker, and the victim, according to court documents. The victim fingered Hodges; that victim, a convicted felon who had pending drug counts at the time of his robbery potentially leading to life in prison, instead eventually received probation from prosecutors, according to appeals documents. Hodges himself had an alibi for the time of the crime.

Marlo Hodges (Photo: Courtesy of the Southeast Correction Center)

One crucial piece of evidence against Hodges was a DNA mixture. Like in the hypothetical Case Five from the MIX13 study, it was taken from a ski mask at the crime scene. (Hand cuffs and a glove that were part of the evidence in the case were not analyzed for DNA.)

The DNA expert, an analyst named Courtney Workman from the Missouri Highway Patrol Crime Laboratory, used CPI to assess the ski mask mixture.

Workman identified a three-person mixture, and found that Hodges “could not be excluded as a contributor of the mixture profile from the mask.” The statistics she calculated showed that 1 in 262 black individuals could be expected to be included as a part of the mixture, according to court documents.

The defense attorneys never challenged those DNA interpretations, according to appellate filings. Now, Hodge’s appeals attorney, Kent Gipson, is seeking to enlist all the DNA mixture findings that could prove how the CPI findings may be wrong. Central to this, he said, could be the MIX13 findings.

“If this had been published at the time it was made, I dare say that it could have been utilized to discredit the prosecution’s case,” Gipson told Forensic Magazine. “I guess that sort of gives a ‘truth is stranger than fiction’ example of how them dragging their feet on this could have really led to a wrongful conviction. Hopefully we can get this rectified.”

But whether it can be included is a whole other matter, under the prevailing Daubert testimony—admission rules that hold sway in the United States.

Take Kerry Robinson, prisoner number 000775363 in the Georgia Department of Corrections system, currently housed at the Coffee Correctional Facility in Nicholls.

Robinson is serving 20 years in prison after his 2002 rape conviction. Robinson was found guilty of taking part in a gang rape of a woman allegedly involving three men. One other man who was convicted in the crime had been positively identified by the woman. That man, in turn, identified Robinson as one of the other attackers. The defense had argued that Robinson, who was not identified by the victim, was identified by the other man in the belief Robinson had reported him to police for various crimes.  

Kerry Robinson (Photo: Courtesy of the Georgia Department of Corrections)

The first man matched the semen mixture at all 12 DNA loci that produced results. Robinson was excluded from 10 of the 12 loci, at the same time. But the Georgia Bureau of Investigation (GBI) concluded that Robinson “could not be excluded.”

Hampikian and two other forensic DNA experts looking at the case instead concluded that Robinson should be excluded from the evidence interpretation. Then Hampikian and Itiel Dror, a cognitive neuroscientist from the University College London known for his look at forensic work, used the DNA evidence from the Robinson conviction as a case study. They posed the DNA evidence available in the Robinson case to a group of 17 experts at an accredited laboratory. Similar to MIX13 in content if not in scope, it asked the experts to determine whether Robinson was in the mixture.

Sixteen of the experts disagreed with the prosecution’s case, with 13 excluding Robinson, and three finding the results inconclusive. Only one concurred with the GBI findings. Hampikian and Dror published their results in the Elsevier journal Science and Justice in 2011.

Hampikian, who became involved in the Robinson appeals, tried to introduce the MIX13 results. It would, he figured, bolster his findings that the CPI results showed significant problems with the prosecution’s case.

But his attempts to convince a judge to admit the PowerPoint slides failed. Because MIX13 wasn’t in a prestigious journal, the court refused to admit them, Hampikian said.

“Judges under Daubert recognize peer review, and error analysis—my study in one lab doesn’t compare to NIST’s study of 108 labs across the country,” said Hampikian, adding that Butler’s DNA textbooks published in 2015 and 2016 have not included the MIX13 results, either. “There is no written record of MIX13 in the peer-reviewed literature.”

IMPLICATIONS

John Wixted, a psychologist from the University of California San Diego who has done a series of studies on eyewitness testimony, has focused some of his latest research on statistical uncertainty involving both witness testimony and DNA.

Discovering MIX13’s findings recently was startling, said Wixted, in an interview.

“Are people wrong to think that these are sort of bombshell results?” said Wixted. “They certainly seem to be, on the surface. And if they are—what explains the fact that so many years have gone by without it being published?”

Butler and Coble both said the results don’t establish an error rate, and don’t necessarily mean innocent people are in jail.

“In terms of drawing/extrapolating from (Case Five), and saying there’s lot of innocent people that are in jail, I don’t know if you can really do that, scientifically,” said Butler. “You can’t with the data we have.”

But forensic experts, including both Butler and Hampikian, acknowledge that complex DNA mixtures have been an increasing part of the forensic load, due to better sensitivity of the technology.

In a survey conducted a decade ago in more than a dozen labs done as a part of the Scientific Working Group on DNA Analysis Methods, complex DNA evidence was part of 11 percent of all DNA samples in their work. The consensus is that has grown over the ensuing years.

“There’s more mixtures that are being done now, because there’s a higher sensitivity,” added Butler. “In terms of what the levels are, I don’t think anybody really knows.”

So more DNA mixtures than ever before are being analyzed among thousands of cases each year in labs across the country—and a majority of those same experts are still using CPI, according to experts.

Probabilistic genotyping like TrueAllele and the system STRmix present a whole new landscape for interpretation, according to Butler.

About 40 labs are using probabilistic genotyping currently, according to Butler. A number of others are in the process of validating their software, whether it’s the more common STRmix, or TrueAllele. Those programs will improve the way DNA mixtures are handled, he added.

“We’re in the process of changing,” Butler said. “In terms of how well people will do … my guess is you’d still see variation, but we don’t know what that will be. That’s a test down the road.”

As for the last five years of DNA mixture results, Butler said no national review was necessary. Local agencies like San Diego—where hundreds of cases are being scrutinized—need to take the lead, Butler said.

“There are efforts to go back and review old cases. And certainly that’s happened in Austin, Texas and Washington, D.C., and other places where there have been issues raised,” said Butler. “But it shouldn’t be done on a wholesale level—it’s up to the individual labs to do that.”

Hampikian, who has been asking for the MIX13 peer review for several years, said it wasn’t enough.

“Shouldn’t this have been an urgent matter, when it was discovered that the vast majority of laboratories are trying to answer questions that they will answer incorrectly?” said Hampikian.

“If it was contaminated peanut butter, or faulty airplanes, or airbags that failed, wouldn’t NIST have felt compelled to do something more than just a couple PowerPoint shows?” Hampikian added. “This is not just a case of salmonella—this is 20 years in prison. Some of these people died in prison.”

Advertisement
Advertisement