The NRC formed the Committee on DNA Technology in Forensic Science, which issued its first report in 1992. In that report they advised against using cold hit results as evidence, and insisted that only the frequencies related to loci not used in the original identification should be presented at trial, that is, that the evidence used to identify the suspect should not be used as evidence against the suspect.
This recommendation has been criticized by many because it underestimates the value of cold-hit matches. The problem was, given a certain amount of evidence the expert, prior to suspect identification, had to make a somewhat subjective decision of how to divide the evidence into two items: one to be used only in the suspect identification, and one to be used only in the trial itself as evidence against the suspect. This overly limited the utility of the evidence and introduced an unnecessary element of subjectivity. It also opened the gate for multiple testing with various evidence division points, and multiple testing leads to its own statistical problems. But let’s put this issue aside.
This recommendation has been criticized by many because it underestimates the value of cold-hit matches. The problem was, given a certain amount of evidence the expert, prior to suspect identification, had to make a somewhat subjective decision of how to divide the evidence into two items: one to be used only in the suspect identification, and one to be used only in the trial itself as evidence against the suspect. This overly limited the utility of the evidence and introduced an unnecessary element of subjectivity. It also opened the gate for multiple testing with various evidence division points, and multiple testing leads to its own statistical problems. But let’s put this issue aside.
NRC II withdrew the earlier recommendation. However, the contrast between low RMP and the frequency of DNA matches in actual database searchers was indeed stark. For instance, the Arizona Department of Public Safety searched for matching profiles in a database comprising 65,000 individuals. The search found 122 pairs of people whose DNA partially matched at 9 out of 13 loci; 20 pairs people who matched at 10 loci; and one pair of people who matches at 12 loci. So it is not that unlikely to find two people in a database who share the same genetic profiles (examples of fairly high counts of DNA matches in database searches was actually used by Barlow in the Diana Sylvester case). In light of this contrast, NRC II recommended the use of DMP rather than RMP. NRC II recommended also that in cold-hit cases the likelihood ratio $R$ associated with the DNA match should be divided by $d+1$. Their first recommendation was about a correction of the random match probability, and this second recommendation is about the likelihood ratio.
One argument by NRC employed an analogy involving coin tosses. If you toss several different coins at once and all show heads on the first attempt, this seems strong evidence that the coins are biased. If, however, you repeat this experiment sufficiently many times, it is almost certain that at some point all coins will land heads. This outcome should not count as evidence that the coins are biased. According to NRC II, repeating the coin toss experiment multiple times is analogous to trying to find a match by searching through a database of profiles. As the size of the database increases, so does the number of attempts at finding a match, and it is more likely that someone in the database who had nothing to do with the crime would match.
Another argument provided by NRC II compared a database trawl to multiple hypothesis testing, and multiple hypothesis testing should be avoided if possible in light of classical statistical methods.
Third, NRC II was concerned with the fact that in cold-hit cases the identification of a particular defendant occurs after testing several individuals. This concern has to do with the data-dependency of one’s hypothesis: seemingly, the hypothesis `at least one person in a given database matches the DNA profile in question’ changes its content with the choice of the database.
We will start with the coin analogy. It is in fact unclear how the analogy with coin tossing translates to cold-hit cases. Searching a larger database no doubt increases the probability of finding a match at some point, but is the increase as fast as the Arizona Department of Public Safety examples and the coin analogy suggest? Quite crucially, following (Donnelly & Friedman, 1999) we need to pay attention to what hypotheses are tested, what probabilistic methods the context recommends, and what exactly the evidence we obtained is. For instance, one hypothesis of interest is what we will call a {general match hypothesis}:
| (General match hypothesis) | At least one of the profiles in the database of size $n$ matches the crime sample. |
The general match hypothesis is what NRC II seems to have been concerned with. If for each data point RMP$=\gamma$ were held constant, and if random matches with different data points $\mathsf{match_1, match_2, \dots, match_d}$ excluded each other, the probability of there being at least one random match would be the same as the probability of their disjunction and could be calculated by the additivity axiom:
\begin{align}
\mathsf{P}(\mathsf{at\,\,\, least\,\,\, one\,\,\, match}) & = \mathsf{P}(\mathsf{match_1} \vee \mathsf{match_2} \vee \cdots \vee \mathsf{match_d})
& = \sum_{i}^d \mathsf{P}(\mathsf{match_i}) = \gamma \times d
\end{align}
This calculation would result in the outcome recommended by NRC II, if the value of the evidence were to be a function of the probability of (General match hypothesis).
The first question is, whether a directly additive calculation should be applied to database matches. Notice that in applications DMP does not really behave like probability. Take a simple example. Suppose a given profile frequency is $.1$ and you search for this particular profile in a database of size 10. Does the probability of a match equal $.1 \times 10=1$? The answer is clearly negative. Multiplication by database size would make sense if we thought of it as addition of individual match probabilities, provided matches exclude each other and so are not independent. Here is a coin analogy. Suppose I toss a die, and my database contains $n=$ three different numbers: $1, 2$ and $3$. Then, for each element of the database, the probability $p$ of each particular match is $\frac{1}{6}$, and the probability of at least one match is $\frac{1}{6}+\frac{1}{6}+\frac{1}{6}=\frac{1}{6}\times 3 = n\times p =\frac{1}{2}$. We could use addition in such a situation because each match excludes the other matches, a condition that is not satisfied in the database scenario.
Another reason why DMP is problematic can be seen by taking a limiting case. Suppose everyone in the world is recorded in the database. In this case, a unique cold-hit match would be extremely strong evidence of guilt, since everybody except for one matching individual would be excluded as a suspect. But if RMP were to be multiplied by the size of the database, the probative value of the match as measured by DMP should be extremely low. This is highly counter-intuitive.
Even without a world database, the NRC II proposal remains problematic, since it sets up a way for the defendant to arbitrarily weaken the weight of cold-hit DNA matches. It is enough to make more tests against more profiles in more databases. Even if all the additional profiles are excluded (intuitively, pointing even more clearly to the defendant as the perpetrator), the NRC II recommendation would require to devalue the cold-hit match even further. This, again, is highly counter-intuitive.
Donnelly, P., & Friedman, R. D. (1999). DNA Database Searches and the Legal Consumption of Scientific Evidence. Michigan Law Review, 97(4), 931. https://doi.org/10.2307/1290377
Written on th, by Rafal Urbaniak