ISO 17025 requires laboratories to document how uncertainty was calculated. Find out ways to become more comfortable with your uncertainty calculations.

In this article, we will address some questions about calculating uncertainty by using trigger pull measurements as an example. You don’t have to be a firearms examiner to understand the article. In fact, we chose this example over other measurements common to forensic labs, like quantitative drug analysis, because trigger pull is a fairly straight forward measurement. For our experiment, we used the setup displayed in Figure 1.

The readout from the measuring device, a Lyman Electronic Trigger Pull, is depicted in the drawing in Figure 2. A careful reading of the manual tells us that the digital display reports pounds and ounces. So, the display is showing 5 lbs 11.5 oz. This measurement raises several questions. For example:

Question 1: If you took just one measurement and got the result in Figure 2, is it okay to report the trigger pull as 5 lb 11.5000 oz?

Most of you probably said “no” because you recognized that the trigger pull device does not give us any information about those trailing zeros after the 11.5.We don’t know what those numbers actually are.

Question 2: Is the right answer more like: Trigger Pull = 5 lbs 11.5 oz ± 0.05 oz?

If you answered yes, you are getting warmer, and this topic was covered in a previous Forensic Magazine article which can be found at www.forensicmag.com/articles.asp?pid=193 (Please note the hard copy of this article has a math error that has been corrected in the online version). However, as we will see in a moment, the real calculation of uncertainty is a little more complicated and interesting. Finally, let’s pose one last question.

Question 3: What if we took several measurements (as indicated in Figure 3); should we use the high and low value to get an answer like: Trigger pull range = 5 lb 1.5 oz – 6 lb 6.0 oz ?

This approach makes some intuitive sense, but it is not quite right. If you have been exposed to formal calculations of confidence intervals and uncertainty values, then you may have already made friends with the equation in Figure 4. This equation is based on something called a Gaussian Distribution or more commonly a Bell Curve. Although named after Johann Carl Friedrich Gauss, the idea was first published in 1718 by Abraham de Moivre. About 80 years later, Johann Carl Friedrich Gauss formalized the mathematical relationships. It is key to a lot of different measurements in science and relies on two rules of probability, the Law of Large Numbers and the Central Limit Theorem. These two rules tell us that if we take multiple measurements of anything, the average of those measurements will tend towards the actual average value of the thing you are trying to measure—in our case, trigger pull. And if you were to graph how your measurements are distributed, it would look a lot like a Bell Curve. In fact, there are many different probability distributions. Other examples include Cauchy, Laplace, Maxwell-Boltzmann, Binomial, and Poisson. Depending on what you are really measuring, you may need to use one of these other distributions to make assumptions about probability. We are using a variation of the Normal Distribution called a Student’s t Distribution.

We run into the Bell Curve all the time in our everyday lives. For example, we might start paying attention to how tall people are. In this case, some measurement of height would be on the x axis. The y axis is ameasure of frequency or how popular a result is. The Bell Curve in this example tells us that over the course of a day we encounter a few short people, lots of medium size people, and a few tall people. Turning our attention to the data in Figure 3, we see a similar pattern. Fifteen of the values fall in the 5 lb 10.0 oz – 6 lb 5.0 oz range while only five fall outside of this range. Figure 5 shows what happens when we apply the equation in Figure 4 to the data in Figure 3.

This result tells us that there is a 99.8%chance that the actual trigger pull average value is between 5 lb 11 oz – 6 lb 3 oz. (You might be wondering why we also rounded the numbers in the final answer. Since ± 4.352 oz tells us our uncertainty is at least 4.3 oz, it does not make sense to include numbers that suggest more precision.) To review how this number was calculated, let’s take a look at each of the variables and parameters in the uncertainty equation.

μ is the actual average value of the thing we are trying to measure.

x is the average of all of the measurements we made = 95.025 oz

t Once we decide how sure we want to be about the answer (i.e., 95%, 99%, or 99.8%), a t-table gives us the number needed to plug into the equation. NIST (www.itl.nist.gov/ div898/handbook/ eda/section3/eda3672.htm) or any statistics text book is a good source of a t-table. In our case = 3.579 (for 99.8% confidence). Be sure to use the two-tailed value.

s is the standard deviation of all your measurements. The instrument you are using may have a published value for the standard deviation based on numerous experiments and measurements. In this case,mathematicians will sometimes use the Greek letter σ instead of the letter s. In our case = 5.3001

n is the number of measurements you took minus one—this last part is in deference to an idea in statistics called Degrees of Freedom. In our case 20 – 1 = 19 Degrees of Freedom.

One interesting consequence of this equation is that the more measurements you take the smaller the uncertainty becomes. This result is true even if you are using a really imprecise measuring technique. Imagine trying to determine the location of a recently removed dart board by using only the darts that completely missed the board. If we have only two measurements, the wall might look like Figure 6, and the dart board could have been anywhere.On the other hand, if we had hundreds of measurements, the wall might look like Figure 7.

One important note is that uncertainty, the Law of Large Numbers, and the Central Limit Theorem all have to do with random error. We can limit the effects of random error in several different ways. If we have a really good instrument, perhaps a precision scale to measure trigger pull, our s will be smaller. If we take a lot of measurements (like in Figure 7), then our n gets bigger.And if we are willing to accept a lower level of confidence (i.e., 95%instead of 99.8%), then our t gets smaller. The effect of any of these changes makes our reported uncertainty smaller.

There is another type of error that is not random.What if our firearms examiner changed the way the measurement was collected and instead of pulling straight back she angled the instrument 45° down as indicated in Figure 8?

In this case we would get a new set of measurements as indicated in Figure 9 and our final answer would look something like Figure 10.

Please note that the uncertainty measurement has not changed significantly, but the average value of the measurements has shifted. Uncertainty or random error can never be fully avoided. However, systemic error is caused by some mistake or flaw in the experiment. And these errors can and should be corrected.

Complying with Requirements
Probably the most important question for forensic laboratory directors, and examiners is, “how does all of this affect my operation and our accreditation?” ISO17025, Section 5.4.6.2 states:

Testing laboratories shall have and shall apply procedures for estimating uncertainty of measurement. In certain cases the nature of the test method may preclude rigorous, metrologically and statistically valid, calculation of uncertainty of measurement. In these cases the laboratory shall at least attempt to identify all the components of uncertainty and make a reasonable estimation, and shall ensure that the form of reporting of the result does not give a wrong impression of the uncertainty. Reasonable estimation shall be based on knowledge of the performance of the method and on the measurement scope and shall make use of, for example, previous experience and validation data.

John Neuner, International Program Manager for the American Society of Crime Laboratory Directors/Laboratory Accreditation Board (ASCLD/LAB) reminds us that their ISO17025 Accreditation Program does not require laboratories to include the uncertainty measures in the report. Instead, laboratories are required to have appropriate documentation available in the laboratory that explains how uncertainty was calculated. He also reminds us that uncertainty of measurement can be calculated for the measurement method, and the uncertainty of the method can then be applied to each case worked. The ASCLD/LAB International Program also has a white paper on Estimating the Uncertainty that spells out six key steps to complying with this requirement.More information can be found at www.ascld-lab.org.

Randall Robbins,Manager of Accreditations, Forensic Quality Services – International (FQS-I) agrees that the exam documentation needs to show how the uncertainty calculations weremade.He also would invite us to review the FQS-I Website for an Uncertainty of Measurement Presentation at: www.forquality.org/FQS-I%20Presentations/UM%20 podcast%20slides_files/frame.htm.

The Scientific Working Group for Firearms and Toolmarks (SWGGUN) published a white paper,Transition from ASCLD/LAB Legacy to ISO/IEC17025, in October 2008. Section 2.2.4 gives a variation of the type of uncertainty analysis described in this article:

The statistical spread results in a series of measurements may yield, through its standard deviation, a measure of the uncertainty. This could be achieved by having all examiners within their unit take test measurements, collecting the aggregate data, and then having standard deviations calculated (this procedure will incorporate instrumentation errors). A plus or minus figure could be assigned to a measurement based on one or two standard deviations from the mean.

Presenting Findings
Perhaps the most important issue is to decide the best way to present your results in a report. Professional ethics demand that we take great effort to present clear, unambiguous findings that cannot be easily misrepresented by either the prosecution or the defense. In this case you may want a report that says:

The average trigger pull for the 9 mm Beretta pistol (Tag 001) was determined to be: 5 lb 15 oz ± 4 oz (with 99.8%confidence). Data used for the ± uncertainty calculation are available upon request from the laboratory.

Another ethical issue is deciding which confidence level to use. Many researchers consider 95%to be the conventional standard. Scientific Working Groups are in a good position to publish guidance about what confidence level to use for a specific experimental technique.We used 99.8% in this article to lend support to the idea that forensic case work should meet a high standard.

One final caution—be careful about the math involved in answering specific questions. In a typical court case involving a shooting, the prosecution may have a suspect who can only fire a weapon that has a trigger pull of less than 5 lbs 10 oz (or 90 oz). She wants to know the chances of getting that result. Based on our 99.8% confidence level that the average value is 5 lb 15 oz (or 95 oz) ± 4 oz, can we conclude that there is less than a 0.2% (or 2/1000) chance of measuring the trigger pull to be 5 lb 10 oz? The answer is a resounding “NO!” Figure 11 illustrates the problem. This new question is actually, “What are the chances of getting a measurement in the red, as opposed the blue, shaded area?”

To rigorously calculate a mathematical answer, we need to make a few assumptions, but we can eyeball the graph and reach a fairly accurate conclusion. We will get measurements less than 5 lb 10 oz about 16% of the time (a lot different than the 0.2% we may have been tempted to report).

In an upcoming article, we will tackle the idea of error propagation by examining a forensic measurement that involves several sources of uncertainty.

Dana Sevigny is Firearm Examiner with the U.S. Army Criminal Investigation Laboratory. She has a BS in Chemistry from Emory University in Atlanta, and an MS in Forensic Science from Virginia Commonwealth University. She completed the Bureau of Alcohol, Tobacco, Firearms, and Explosives National Firearm Examiner Academy in 2003. She has worked in crime laboratories in Vermont and Georgia as a Firearms Examiner, and she is a member of the Association of Firearms& Toolmarks Examiners. She can be reached at dana.sevigny@us.army.mil.

Jeff Salyards is the Program Manager for Science&Technology at the U.S. Army Criminal Investigation Laboratory. He holds a PhD in Chemistry from Montana State University, a MFS from George Washington University, and has completed a Fellowship in Forensic Medicine from the Armed Forces Institute of Pathology. A former director of the Defense Computer Forensic Laboratory, he has 21 years of combined experience in investigations, forensic consulting, and teaching. He has served on the Board of Directors for the American Society of Crime Laboratory Directors/Laboratory Accreditation Board, the Department of Justice National Steering Committee for Regional Computer Forensic Laboratories, and the Council of Federal Forensic Laboratory Directors.He can be reached at jeff.salyards@us.army.mil.

The opinions contained herein are the private views of the author and are not to be construed as official or as reflecting the views of the Department of the Army or the Department of Defense.