Sigma Metrics, Total Error Budgets & QC

Make sure your test system performance and quality control procedures are aligned with your quality goals

By Curtis A. Parvin, PhD, John Yundt-Pacheco and Max Williams

Good laboratory practice requires that laboratories design their quality control (QC) procedures to assure that reported patient results meet the quality required for their intended use.¹ In previous articles in this series, we argued that a laboratory's quality goals should be focused on the risk of reporting unreliable patient results containing measurement error that exceeds the allowable error specified for the analyte.^2-5 In this article, we discuss two metrics-sigma values and total error budgets-that can help provide general guidance on how well a laboratory's test system performance and QC procedures align with its specified quality goals.

Sigma values

Laboratory quality specifications are often defined in terms of allowable total error limits (TEa). If the difference between the true concentration of an analyte and the reported concentration in a patient's specimen exceeds TEa the result is considered unreliable. The sigma value expresses the number of analytical standard deviations of the test system process that fit within the specified allowable total error limits. That is,

Sigma = (TEa - Bias) / SD

Bias is the systematic difference between the expected results obtained by the laboratory's test method and the results that would be obtained from an accepted reference. The reference may be another test method, a standard or a consensus reference like a proficiency program or an inter-laboratory peer-comparison program. SD is the total analytical standard deviation of the test method. Equivalently, the quantities can be given as percents:

Sigma = (%TEa - %Bias) / %CV

where %CV is the analytical coefficient of variation of the test method. The Figure gives a graphical example of a test method with 1% bias, 2.5% coefficient of variation and a specified TEa of 10%.

In this case, the sigma value is (10 - 1) / 2.5 = 3.6. That is, 3.6 analytical SDs fit within the 10% quality specification.

Bias can have a significant impact on analytical quality and should usually be removed from the laboratory test system when it is identified. However, eliminating bias below a certain threshold can be difficult and attempts to do so are more likely to increase the overall imprecision of the test method. In general, the value for bias used in sigma computations should be the minimum threshold at which bias is actionable (an attempt to remove it will be made).

Sigma values and QC strategy design

Sigma values are useful for guiding QC strategy design. For a high sigma process it is relatively easy for the laboratory to design a QC procedure to detect any out-of-control condition that could pose a significant risk of producing unreliable results. A relatively large out-of-control condition would have to occur before there would be much chance of producing results that contained errors that exceed the TEa specification and it is easy to design QC procedures that can detect large out-of-control conditions.

Fig. 1 

On the other hand, for a low sigma process a relatively small out-of-control condition may pose an unacceptably high risk of producing unreliable patient results. It can be challenging to design QC procedures that are good at detecting small out-of-control conditions.

Simple guidelines for choosing appropriate QC rules based on sigma values have been proposed.⁶ An example of one such guideline is shown in Table 1.

Table 1 

For lower sigma values more QC samples and more powerful QC rules are recommended. Note, a 13S QC rule rejects if any of the QC results differ from their target concentration by more than three SDs. Multirules are combinations of individual QC rules that tend to be more powerful than simple rules such as the 13S QC rule. In general:

• For large sigma value processes (≥6 sigma), simple QC rules with low false rejection rates are adequate.
• For intermediate sigma value processes (sigma values between 3.5 and 6) quality goals can be met, but more elaborate QC strategies may be required.
• For low sigma values (<3.5 sigma) it will be difficult to meet the laboratory's quality goals without finding ways to further reduce the test systems analytical bias, or its analytical imprecision.

Allowable total error, total error and total error budgets

TEa limits specify the measurement error requirements that must be met for the test system to provide patient results that satisfy their purpose. On the other hand, total error (TE) is commonly defined as a summary measure that provides a "reference range" of measurement errors for a test system based on the test system's analytical imprecision and bias. Whereas TEa is a quantity specified by the laboratory that reflects the quality requirement for an analyte, TE is a summary measure reflecting the test system's analytical performance capability. TE is generally defined as:

TE = Bias + Zp*SD

or alternatively as:

%TE = %Bias + Zp*%CV

If Zp is set to 2.33 then about 99% of measurement errors should fall within the TE limits. If Zp is set to 1.645 then about 95% of measurement errors should fall within the TE limits.

The total error budget (TEB) is a quantity that relates the laboratory's test system process capability (TE) to the laboratory's quality requirement (TEa):

TEB = 100*TE / TEa

TEB reflects the percentage of the TEa in patient results that is "consumed" by the laboratory's inherent test system imprecision and bias. How large can TEB be before a lab should be concerned that its test system capability is not well aligned with its quality goals?

We have already shown that sigma values can be used as a guide to the amount of QC effort required to assure that the lab's quality requirements are met. The TEB is a quantity that relates process capability to quality requirements. Table 2 computes both TEB and sigma values assuming TEa = 10% for different amounts of test system bias and imprecision.

Table 2: TEB and Sigma with TE a = 10% as Bias and Imprecision Decrease

In the first row of Table 2 the bias is 2.5% and the CV is 4.5%. In this case, TE is 9.9% (using the 95% limit definition), TEB is 99%, and the sigma value is 1.67. Thus, a process where nearly 100% of the TEa is consumed by the test system's bias and imprecision is associated with a very poor sigma value.

In the fourth row of Table 2 the bias is 0.5% and the CV is 1.7% (TE = 3.3%, TEB = 33% and sigma = 5.59). A process where only a third of the TEa specification is consumed by the test system's bias and imprecision is associated with a high sigma value. Row 3 of the table suggests that for the test system to be capable of meeting the laboratory's desired quality goals (sigma values ≥3.5), the TEB should not exceed 50%. If more than 50% of your allowable error specification is "consumed" by your test system bias and imprecision, you are going to have a difficult time assuring that your reported patient results are meeting your quality goals. A lab should strive for a TEB of 33% or less.

Dr. Parvin is manager of Advanced Statistical Research; John Yundt-Pacheco is Scientific Fellow; and Max Williams is Division Global Marketing Manager, Bio-Rad

References

1. International Organization for Standardization (2007) Medical laboratories - particular requirements for quality and competence. ISO 15189. International Organization for Standardization (ISO), Geneva.
2. Parvin CA, Yundt-Pacheco J, Williams M. The focus of laboratory quality control: Why QC strategies should be designed around the patient, not the instrument. ADVANCE for Administrators of the Laboratory 2011;20(3):48-9.
3. Parvin CA, Yundt-Pacheco J, Williams M. Designing a quality control strategy: In the modern laboratory three questions must be answered. ADVANCE for Administrators of the Laboratory 2011;20(5):53-4.
4. Parvin CA, Yundt-Pacheco J, Williams M. The frequency of quality control testing. QC testing by time or number of patient specimens and the implications for patient risk are explored. ADVANCE for Administrators of the Laboratory 2011;20(7):66-9.
5. Parvin CA, Yundt-Pacheco J, Williams M. Recovering from an out-of-control condition: The laboratory must assess the impact and have a corrective action strategy. ADVANCE for Administrators of the Laboratory 2011;20(11):42-4.
6. Westgard JO. Six sigma quality design & control, 2nd ed. Madison WI: Westgard QC Inc., 2006.

Copyright 2015