In the first study designed to specifically explore reliability and biasability in digital forensics, renowned cognitive bias expert Itiel Dror and co-author Nina Sunde have illustrated that examiners’ observations are biased by contextual information, and consistency between the experts is low.
“This knowledge is an essential foundation for transparency and designing effective measures that can minimize error, or detect them before they cascade further into the investigation process,” the authors write in their paper, published in FSI: Digital Investigation.
For their study, the researchers recruited 53 digital forensic examiners from eight countries (Canada was included, but the U.S. was not).
All participants were tasked with analyzing an evidence file that contained programs and files typically present on a work computer, such as those for handling documents, spreadsheets or presentations, e-mail, chat, internet browsing, etc. And while all participants were given the scenario of confidential information leakage, they each received different contextual information that indicated either strong guilt, weak guilt or innocence. The control group only received the scenario.
Dror and Sunde selected 11 different “traces” within the evidence file that they could use to compare the observations of the digital forensic examiners. Some of the traces were easier to find, such as e-mail content, documents and chats, while others required more in-depth examination; however, none were complex.
According to the study, none of the participants observed all 11 traces. Sixty-six percent of the participants identified 5 to 8 traces, 26% found 1 to 4 traces and only 8% found 8 to 10 traces. When comparing the groups, higher numbers were observed in the guilt groups, followed by the control group.
“The innocence group observed the least number of traces, indicating that they were biased to find less evidence,” the authors write. “The guilt groups had the highest number of traces, indicating that they were biased to find more evidence. The control group was between the Innocence and guilt groups. However, there was very little difference between the strong guilt group and the control group in the proportion of observations, which indicates that the strong guilt context (the suspect had confessed) did not bias the participants to observe more traces than an examination without such context.”
The weak guilt group observed significantly more traces, suggesting that the ambiguous weak guilt context—wage conflict where the suspect had taken side with the workers—biased the group to find more traces.
Additionally, the researchers used a statistical measure to gauge the consistency of observations within each participant group that analyzed the evidence file based on the same contextual information. The team recorded an overall low/inadequate reliability of less than 0.667—for this specific co-efficient, values 0.80 or higher are considered strong, while those lower than 0.667 are considered inadequate. The highest reliability score was seen at the observation level for examiners receiving strong guilt context (0.51) and innocence context (0.44).
“Although high reliability between [digital forensic] examiners is anticipated, it is important to be aware that consistency does not imply accuracy or validity,” the authors write. “Consistency may arise from a variety of reasons. This entails that quality measures should not only be directed toward the tools and technology, but also the human. It is not possible to calibrate a human the same way as a technical instrument, but measures such as blind proficiency testing through the use of fake test cases may provide knowledge about human performance.”
Still, the authors conclude there is a “serious and urgent need for quality assurance” in digital forensic examinations. To minimize bias, they suggest ensuring that all digital forensic examinations occur based on task-relevant contextual information only—exposure to “other” contextual information, such as whether a suspect has confessed or has prior arrests, should be minimized at all costs.