Every discipline in this series failed in the open, where anyone with the file could eventually check the work. The software now interpreting complex DNA mixtures does not permit that. When one court ordered a release, analysts found within a year what years of studies had missed.
Probabilistic genotyping software interprets DNA mixtures too complex for conventional analysis and reports a likelihood ratio to the jury. In 2016 a federal judge ordered release of the source code for one such program used by the New York City medical examiner. Analysts found a function that overestimated the likelihood of guilt. No study conducted without the code had detected it, and the tool was subsequently retired.
Is this the same as saying DNA evidence is unreliable?
No, and the distinction is important. Single source DNA comparison is the most robust identification method in forensic science. Probabilistic genotyping addresses a much harder problem, mixtures of several contributors at low quantities, and the strength of the underlying discipline does not transfer automatically to the software built on top of it.
What is a likelihood ratio?
A number expressing how much more probable the observed data would be under one hypothesis than another, typically that a named person contributed to the mixture versus that an unrelated person did. It is a comparison of two explanations, not a probability of guilt, and the difference is frequently lost between the report and the jury box.
Do validation studies not already establish that the software works?
They establish that it produces expected outputs on known inputs. That is necessary and not sufficient. A program can produce reasonable results across a test set while containing a function that behaves incorrectly in circumstances the test set does not include.
Why would a court refuse disclosure?
Developers assert that source code is a trade secret with commercial value that disclosure would destroy. Courts weigh that against the defendant’s interest, and outcomes have varied. Protective orders are a common middle path, though they can restrict what a reviewing expert may afterward say publicly.
What I Notice First About This One
Everything in this series so far has been retrospective, and I want to be direct about why this installment is not.
Hair comparison, arson doctrine, bite mark analysis, bullet lead. Every one of those is settled. The research is done, the conclusions are public, and the remaining question is what is owed to the people convicted before the answer arrived. That is a historical question with present consequences, which is a comfortable thing to write about.
This installment concerns a method being used right now, in Michigan courtrooms among others, whose validity I am not asserting is compromised. I want that stated plainly before anything else, because the pattern-matching temptation here is severe and I do not think the evidence supports it.
What I am asserting is narrower and, I think, harder to argue with. Every failure documented in the previous five installments was eventually discoverable because the underlying method was open to examination. Someone could read the transcripts, replicate the reasoning, run the studies. The examination was late and it was performed by volunteers, but it was possible.
Here the examination is not possible without permission, and the one time permission was granted, it produced exactly what the prior installments would predict.
The Problem the Software Solves
Traditional DNA analysis works best on a clean sample from one person. Much of what actually comes off a crime scene is not that. It is a mixture of material from two or more contributors, often in small quantities and partially degraded, where the peaks in the data may reflect a real allele, an artifact of the amplification process, or contribution from someone who touched an object days earlier.
Older interpretive approaches handled this badly. Probabilistic genotyping was developed to handle it properly, using statistical modeling to evaluate how well competing explanations account for the observed data and reporting the comparison as a likelihood ratio. The fully continuous systems that dominate the field also use peak height data rather than discarding it.
The statistical foundations here are real. Markov chain Monte Carlo methods long predate their forensic application and are well established across the sciences. This is not a discipline like bite mark comparison with premises that were never established.
The validity question has moved one layer down. For hair and bite marks the question was whether the method could work. Here the method can work. The question is whether a particular implementation does, and that question cannot be answered from the outside, because the implementation is the code and the code is proprietary.
What Happened When Someone Read the Code
The Forensic Statistical Tool was developed and used by the Office of Chief Medical Examiner in New York City, and was employed in thousands of criminal prosecutions.
In 2016, in the course of a Daubert hearing, a federal judge in the Southern District of New York ordered that the defense be given access to the source code. Analysts examined it and identified a function, not disclosed in the program’s documentation, that tended to overestimate the likelihood of guilt. The judge later lifted the protective order. ProPublica obtained the code and published it, allowing general scrutiny. The medical examiner’s office stopped using the tool and moved to a commercial competitor.
The detail that matters most is not the error. It is when the error was found. Prior studies of the tool had been conducted without access to the source code, and none of them detected it. The error was found within about a year of the code becoming available, by people reading it.
Errors have also been identified in the commercial programs. Reviewers examining a competing system under protective order reported coding problems affecting reliability, and the developer has acknowledged software defects affecting a set of cases in Australia. That developer publishes a defence access policy, which is a meaningful step and also an illustration of the underlying condition: access is a policy granted by a vendor rather than a right held by a defendant.
The Field Kit includes the Clutch Justice FOIA templates with jurisdiction guidance and exemption anticipation language, including the structure used to request agency validation documentation.
Open the Field KitThe Lag, Now Enforced by Law
Return to the frame this series has been building. The validation lag is the interval between a method’s admission into evidence and its scientific validation. In Parts II through VI that interval was maintained by ordinary institutional neglect. Nobody was assigned to check, so nobody checked.
The situation here is different in kind. People want to check. Defense experts have identified the examination they would perform and the courts have been asked to permit it. The interval is maintained by an affirmative legal claim asserted by the developer and adjudicated by a judge.
That has one consequence worth stating carefully. A lag maintained by neglect ends when someone finally looks. A lag maintained by law ends only when the law changes, and in the meantime it is renewed case by case, by judges applying a balancing test with no settled national answer.
In every earlier installment, the barrier to finding the error was that nobody had the job. Here the barrier is that somebody has the right. That is a more durable arrangement, and it was built deliberately, by people who were not thinking about criminal defendants at all.
Probabilistic genotyping systems enter forensic laboratories to address complex and low-template mixtures. Both public agency tools and commercial products are deployed, each validated internally and through published studies.
In litigation in Washington state, a defense expert obtains review of a commercial system’s source code under a protective order and identifies problems affecting reliability. The order restricts what he may afterward disclose.
In the course of a Daubert hearing, a judge in the Southern District of New York orders that the defense receive access to the source code of the Forensic Statistical Tool used by the city medical examiner.
Review identifies a function that tended to overestimate the likelihood of guilt, undetected by any prior study performed without the code. The protective order is lifted, ProPublica publishes the code publicly, and the tool is retired in favor of a commercial system.
A New York appellate decision addresses whether a probabilistic genotyping analysis is testimonial, concluding that the developer’s availability for cross-examination addressed the confrontation concern raised.
A New Jersey appellate court holds that a defendant is entitled to source code of novel probabilistic genotyping software upon a showing of particularized need, reasoning in part from what the earlier source code release had revealed.
What Michigan Has Not Answered
I have written this section three times trying to make it a finding, and it is not one. It is a question, and the fact that it is still a question is the point.
The Michigan State Police forensic science division operates the state’s accredited laboratory system and handles a very large annual caseload. Complex DNA mixture interpretation is routine forensic work. It is therefore reasonable to expect that Michigan uses probabilistic genotyping software, and reasonable to want to know three things: which program, what internal validation was performed and documented, and whether any policy governs defense access to the implementation.
All three are matters of record. None of them, as far as I can determine, has been reported publicly. I am not asserting that Michigan is hiding anything, and I have no reason to think so. I am observing that a question this basic about evidence used to convict people in this state has no published answer, and that the reason is that nobody has filed the request.
Public disclosure of forensic software in use and its validation record. The identity of the program, its version history, and the agency’s internal validation documentation should be published as a matter of course. None of that is the source code, none of it is a trade secret belonging to the state, and its absence means the public cannot evaluate what it is being asked to trust.
Procurement conditioned on defense access. The most workable lever is not litigation but purchasing. A state that buys forensic software can require, as a term of the contract, that the vendor provide code access to a qualified defense expert under protective order in any case where the output is offered against a defendant. This converts access from a right a defendant must win into a condition the state has already secured.
Why This Matters Beyond DNA
I want to end by saying what I think this installment is actually about, because it is not really about DNA.
Every discipline in this series was admitted before it was validated, and the gap was closed eventually because the method was inspectable. Hair comparison could be studied because anyone could look through a microscope. Arson doctrine could be tested because anyone could set a fire in a laboratory and watch what happened. Bite mark premises could be examined because teeth and skin are available to researchers.
The forensic tools now entering courtrooms are not inspectable in that way. They are software, they are commercial, and the thing that would have to be examined is protected by a body of law developed to protect competitive advantage in ordinary markets, applied to a setting where the other party is a person facing prison.
The pattern of the previous six installments is that errors surface late, through effort, by people with no obligation to make it. Take away the ability to look at all and there is no reason to think the errors stop occurring. There is only reason to think we stop finding them.
That is what changes here. Not the rate of error. The rate of discovery.
Six installments have identified the same missing instrument from six directions: a record connecting convictions to the disciplines that produced them, held by a body with authority to act on it. Part VIII closes the series with the document that already specifies it. Michigan’s own Task Force on Forensic Science designed this instrument, described how to keep it independent, and recommended its creation. It was not built.
Continue Your Investigation
If this reporting raised more questions, use the Clutch Justice ecosystem to keep going.
