![](/uploads/1/2/7/6/127682432/606585372.jpg)
The contents of articles or advertisements in The Clinical Biochemist – Reviews are not to be construed as official statements, evaluations or endorsements by the AACB, its official bodies or its agents. Statements of opinion in AACB publications are those of the contributors. Print Post Approved - PP25. © 2005 The Australasian Association of Clinical Biochemists Inc. No literary matter in The Clinical Biochemist – Reviews is to be reproduced, stored in a retrieval system or transmitted in any form by electronic or mechanical means, photocopying or recording, without permission. Requests to do so should be addressed to the Editor. ISSN 0159 – 8090.
IntroductionExternal quality assurance (EQA) has been described as the ‘fifth pillar’ of laboratory standardisation, added to the four pillars of reference materials, reference methods, reference laboratories and traceable reference intervals and decision points. One vital component of an EQA program is to provide criteria to allow participating laboratories to assess their performance. These criteria, also described as quality standards, may be based on statistical comparison with peers, expert opinion, clinical need or other criteria. In Australia, the RCPA Quality Assurance Programs (RCPAQAP) quality standards are known as the Allowable Limits of Performance (ALP) and are used to assist with interpretation of all numerical QAP results in the program. The stated goal of these limits is that they are clinically based to provide warning of clinically important changes in assay performance. As these limits were derived some years ago it is appropriate to review them. This document describes the assessment and implementation of a hierarchical approach to setting quality standards for EQA programs using the approach used by the RCPAQAP Chemical Pathology as an example.
Quality Standards for EQA: The Need for DefinitionsQuality standards vary considerably between different EQA programs for the same analytes. This may be due to different meanings behind the standards, assessing different data or making different interpretations of available information. However when a quality standard is applied it is important that the meaning of the standard is known.
For example, because failing the regulatory standards in use in the USA or Germany has important consequences for the laboratory’s right to practice, they are usually able to be achieved by nearly all laboratories. By contrast targets set as goals for improvement may not be reached by all laboratories at the time of implementation. For example when the 10% coefficient of variation (CV) requirement for troponin assays was first proposed it was not generally able to be reached at the important decision point although recent assays are able to meet this criterion. Thus when EQA quality standards are being used by laboratories, it is important to clearly understand the intention of the standard in order to use it appropriately. Similarly, when comparing quality standards from different EQA programs it is important to be aware of the rationale behind them. Thus in the process of setting quality standards for an EQA program the initial step is to define the purpose and meaning of the standards as this will greatly influence the decision-making process for the EQA as well as guide the response of client laboratories to the standards.Quality standards for EQA programs may be used on different types of data from the same laboratory.
For example standards may be applied to single results shortly after the time of analysis, or to accumulated data at the end of a distribution cycle. Standards for the former are necessarily total error standards because single results will always include errors caused by both bias and imprecision. When multiple results for an analyte are available it is possible to separately assess bias and precision and apply specific standards. For the purposes of this discussion the main focus will be on quality standards applied to individual results as total error standards.
The development of separate bias and imprecision standards for accumulated data is also an important process but is not considered here in detail. Quality Standards and Between Laboratory VariationThe traditional service provided by EQA programs allows individual laboratories to assess the performance of an assay and respond accordingly. More commonly now laboratories have multiple analysers or are parts of laboratory networks in which a patient’s result may be generated from any one of many analysers in the system. It is also becoming more common that patients may have pathology testing performed in many different laboratories, and that results may be collated, either informally in a collection of printouts or formally in a combined database. In these settings a different paradigm becomes apparent.
Biological Variation 1 - Prague - 16 May 2006 After this lecture, participants should:. be able to list the types of biological variation,. know how to generate and/or find data on random biological variation, and. appreciate how to use the data in laboratory medicine. Biological Variation: From Principles to Practice. Article Info & Metrics PDF. Them in available literature, and use them in everyday laboratory practice.
The EQA results provide information about the analytical performance between as well as within laboratories, and quality standards must be designed with this in mind.For diagnostic purposes, the question is whether different laboratories can use the same decision points or reference intervals, and for monitoring purposes the issue is whether a patient can be reliably monitored using results from different laboratories. This relates to a key issue with regard to common pathology databases, namely whether we can combine the results from different laboratories in the same database and whether the combined data is suitable for diagnosis and/or monitoring. This provides an additional purpose for EQA quality standards to provide information on the use of data in combined pathology databases. Quality Standards and TargetsWhen assessing individual results in an EQA program, the quality standard is the allowable difference from a target. Targets may be set in a number of ways including: statistical values such as the median of all returned data or of the relevant method groups; reference measurements performed on the samples; comparison with reference materials; or weighed in values for substances such as drugs. Assessment of results using quality standards expressed as a difference from a target automatically makes the assessment one of total error, i.e. Both bias and imprecision are included in the process.
This has the effect of making the QAP target the reference standard for the interpretation and thus the choice of target is vitally important. For that reason target setting will be discussed in further detail.The different approaches and their strengths and weaknesses are as follows. Use of Reference MeasurementsWhere reference measurement procedures are available, they offer the most reliable target-setting, especially when native human serum samples or pools are used as the EQA material. In practice some modification of materials is required to achieve the desired range of results, number of analytes and stability during transportation or storage. These modifications may lead to lack of commutability of the material between different methods, often referred to as a matrix effect. Indeed with the exception of native patient samples, commutability should not be assumed without experimental confirmatory evidence.
Similarly, between-method differences should not be assumed to be due to matrix effects without confirmation. Use of Overall MediansMedians may be used with or without statistical removal of outliers. The RCPAQAP uses all data in its calculations of a median. In common with reference measurements, the use of a single target for results for all methods has the advantage of highlighting the possibility of differences between methods which can then be investigated for matrix effects as needed. A weakness is that the overall medians can be influenced by the method composition of the participating laboratories and favours any numerically dominant method. Indeed with two similarly sized method groups which give different results, the median may shift dramatically from challenge to challenge depending on the exact numbers in the groups, and on some occasions may be representative of no group in particular.
Use of Method-specific MediansUse of medians based on method groups has the advantage of removing the matrix effect but limits the interpretation to the question “Is my method working similarly to other users of the same method?” without providing any information on between-method differences. Method-specific medians are also problematical with open-systems such as are used in some chemistry analysers where there may be a choice of possible comparators. Is the appropriate peer the analyser, an analyser group (i.e. From the same manufacturer), the analytical principle, the reagent manufacturer, the calibrator or some other variable? For closed systems such as automated immunoassay platforms or haematology analysers, the use of medians, either overall or more commonly method-specific, also has the potential to mask changes over time if the entire group is changing as one. This has been seen with a biased calibrator affecting an entire manufacturer group where the change was not apparent because the target (median value) was necessarily changing to the same extent.
Weighed-in ValuesWhile attractive for programs with exogenous analytes such as drugs, there are a number of pitfalls in making simple additions of this type. If a drug is present in the circulation both in native form and as metabolites, this will not be reflected in the EQA material, potentially leading to matrix effects. Addition of drugs to samples in vitro may not produce the same protein-binding seen in vivo, and the presence of multiple drugs in an EQA sample may produce effects different from those seen in patients. Moreover technical problems in obtaining full dissolution of the drug in the sample without degradation can make the weighed-in values erroneous. Thus this method of target assignment needs to be used with caution. Quality SpecificationsThe Stockholm Consensus ConferenceIn 1999 a conference was held in Stockholm under the auspices of the WHO, IFCC and IUPAC.
The outcome was a consensus statement titled: Strategies to Set Global Quality Specifications in Laboratory Medicine. The strategies consist of a series of approaches arranged in a hierarchy, with the aim where practicable to apply models higher in the hierarchy in preference to those at lower levels. The key statement is summarised in with the different levels in the hierarchy considered in more detail below. Implementing the hierarchy involves assessing the available information about a test and applying the highest available criterion. For example, if there is reliable data on biological variation (level 2), this should be used in preference to expert opinion (level 3) or state of the art (level 5). Evaluation of the effect of analytical performance on clinical outcomes in specific clinical settings:The requirement of this criterion is the existence of clinical trial data demonstrating that a certain level of analytical performance is needed for a particular clinical outcome.
A major problem with this approach is the lack of suitable studies to set the criteria. Ideally a study would compare outcomes when assays with different analytical characteristics are used. More commonly studies are used where a difference in result between individuals in two groups leads to a different outcome, thus demonstrating the need to be able to separate results from the two groups analytically.
The most commonly quoted example of this type of data is the Diabetes Control and Complications Trial (DCCT) related to intensity of glucose regulation in type 1 diabetes mellitus. The average HbA 1c in the two treatment groups were approximately 7.0% (53 mmol/mol) and 8.0% (64 mmol/mol), thus leading to the requirement for assays with coefficients of variation of less than about 3% for assays measuring in% (∼4% for assays measuring in mmol/mol), although this calculation does not include any allowance for within-subject biological variation. While this type of study interpretation shows a significant difference between the groups in the trial this difference does not necessarily equate to separating individuals from the two treatment groups. Evaluation of the effect of analytical performance on clinical decisions in general:.data based on components of biological variation.data based on analysis of clinicians’ opinionsBiological variation is an exciting development in the area of assay quality standards as it provides the possibility of an independent, clinically important basis for setting quality specifications.
Perhaps one of the most useful ways of looking at biological variation criteria is in providing limits where further improvement is no longer necessary. For example, if an assay meets the optimal standard for monitoring with a CV. Published professional recommendations:.from national and international expert bodies.from expert local groups or individualsPublished professional recommendations on analytical quality provide an important source of information for setting EQA quality standards.
They should reflect a synthesis of available information regarding clinical use and analytical standards and result from an interaction between clinical experts and laboratory experts. There may be limitations if the consensus group does not include laboratory experts as it is important to ensure a full understanding of any recommendations. For example, are they related to within or between laboratory performance, to imprecision, bias or total error, and are they minimal or optimal standards? A well known example of expert guidelines are the lipid standards from the Centers for Disease Control and Prevention (CDC) in the USA. Performance goals set by:.regulatory bodies.organisers of EQA schemesExamples of regulatory standards are CLIA and RiliBAK as described above. EQA organisers may choose to reflect relevant regulatory standards in their own quality standards as a way of assisting laboratories to meet their regulatory requirements.
![Practice Practice](/uploads/1/2/7/6/127682432/511081266.png)
Certainly setting EQA limits wider than regulatory limits is unlikely to be of benefit to clients. At the least EQA organisers need to be aware of any regulatory standards when setting EQA quality standards. Goals based on the current state of the art:.as demonstrated by data from EQA or Proficiency Testing schemes.as found in current publications on methodology.EQA programs are well placed to apply this type of criterion as they generate data which is directly relevant for the methods and materials used in their programs.
Interpretation of the data can be done in a number of ways. The most objective is a strict statistical approach, with or without an outlier exclusion process. For this a numerical value needs to be applied, e.g.
Should 80, 95 or 99% of laboratories be used to define the standard. Alternatively the standard could be set such that only laboratories clearly outside the distribution of the majority of laboratories are flagged. The state of the art information can also provide information on the number of laboratories which would be flagged outside any proposed quality standards. A high flagging rate may be considered acceptable for aspirational standards, if improvement is both possible end desirable, but inadvisable for regulatory quality standards. Further Comments on the Stockholm HierarchyWhile the Stockholm consensus statement appears to provide a clear rationale for the setting of quality standards for EQA providers, a closer inspection reveals a number of obvious limitations.Firstly there is interlinking between the different levels of the hierarchy.
For example, application of level IIb, standards based on clinician opinion, is necessarily limited by level V, state of the art, as clinicians have not had access to results better than those currently available in routine laboratories. Similarly levels III and IV, a range of expert opinions, will be made by individuals aware of the limitations of currently available techniques.Secondly there is the need to be aware of the rationale for the quality standards as described above. A regulatory standard is likely to be set lower than state of the art in order to avoid major problems with the provision of laboratory services. By contrast, if the purpose of the quality standard is to provide an optimal goal with the aim of improving laboratory performance, any limitation of state of the art is immediately apparent and can be addressed.Once a level of the hierarchy is selected for application there remain further decisions to make. Firstly, there is the identification and assessment of the relevant data. For example if a biological variation criterion is to be applied, it is necessary to ensure that the data are the best available and most relevant.
While much accumulated information is available in the database prepared by Carmen Ricos and colleagues, each entry is a synthesis of one or more studies and a more detailed evaluation of the original data may be required.Following the selection of a level within the hierarchy, there are still options to be considered for implementation. This is clearly demonstrated for biological variation where analytical criteria have been set for standards described as minimal, desirable and optimal. Similar decisions are required for application of a state of the art criterion as described above.The hierarchy also contains an apparent internal contradiction. If the scientific organisers of an EQA program select a quality standard based on, for example biological variation (level II), then as this is selected by an “expert panel”, this decision can also be described as level IV.For the implementation of clinically-based quality standards the use of the assay result should also be considered. In general the two main uses for numerical laboratory data are diagnosis and monitoring.
These different applications lead to different quality requirements for the assay results. For the purpose of diagnosis, the assay must have a sufficiently low total error (bias and precision) that only a small number of patients may be wrongly diagnosed when the results are compared with a decision point. By contrast for monitoring the requirement is for a low imprecision within the time-frame of the monitoring process to avoid errors of interpretation, either falsely classifying a patient as changing condition, or missing a significant change. Quality standards for these criteria have been developed based on within- and between-person biological variation. As the quality standards for monitoring purposes are generally tighter than for diagnosis, if a test has a total error within the required standards for monitoring, then it would usually be considered suitable for diagnosis.
A Process for Setting EQA Quality StandardsApplying the hierarchy to EQA quality standards is an activity which requires clarification of a number of factors as follows:.Assemble personnel with knowledge of laboratory and clinical issues related to the tests involved.Clearly define the purpose for the quality standard(s).Agree on the principles to consider and how to apply the different levels of the hierarchy.Gather information relevant to the different levels of the hierarchy. This would include clinical studies, biological variation data, expert or consensus opinions and current analytical performances.Clearly document the information used to set the quality standards.Record the decision and the rationale to allow later review if needed.This process should allow the setting of documented quality standards where the data, rationale and participants are recorded providing a solid base for the limits used to assess QAP performance. Different Criteria for Different AnalytesIt is an attractive proposition to apply the same quality standard criteria to all analytes in a program.
However previous work has shown that this is not possible. Where current analytical performance is at a high standard, for example at a level where patients can be monitored using results from different laboratories, it would seem appropriate to set quality standards that identify failures to meet this use even if this standard (monitoring) cannot be met for all analytes.
If a lower standard, e.g. For diagnosis, was applied, then a laboratory may not be alerted to a failure that affects monitoring and patient care if the assay is being used for that purpose. It can also be argued that if an analyte reaches a monitoring standard, further work to improve the assay performance is unnecessary and efforts in laboratory quality should be spent in other areas. By contrast, an assay which meets a diagnostic or state of the art criterion may benefit from analytical improvements. Thus if different criteria for quality standards are applied to different analytes, it is necessary to understand which criterion is being applied so as to be aware of the need for possible improvements.
RCPA QAP Chemical PathologyOver the last three years, the majority of chemical pathology programs in RCPAQAP have undergone a revision of their allowable limits of performance. Examples of the revised limits are shown in taken from the General Serum Chemistry program. These revisions have been done by applying the concepts described above. The supporting information in the RCPAQAP circulars provides both the basis for the limits (i.e. Either precision or total error) and the standard (optimal, desirable or minimal).
This supporting information allows users to identify whether data from assays meeting the quality standard can be used to monitor patients within or between laboratories, or share reference intervals. The detailed information recorded by the working parties performing the revision allows for reassessment in the light of additional information as it becomes available. Other EQA providers have also applied biological variability to determine acceptability limits although this has tended to be limited to a subset of analytes or a specific project. AnalyteUnit+/−ToThenBasis of LimitsALTU/L54012%Optimal ImprecisionAlbuming/L2336%Desirable Total ErrorAlk PhosU/L1512512%Desirable Total ErrorAmylaseU/L1010010%Desirable ImprecisionASTU/L54012%Desirable ImprecisionBicarbonatemmol/L22010%Minimal Total ErrorBilirubinμmol/L32512%Optimal ImprecisionConj.
![Pdf Pdf](/uploads/1/2/7/6/127682432/596757818.jpg)
Laboratory analytes for individuals are subject to several sources of variation, including biological variation, preanalytical variation (specimen collection), analytical variation (bias and imprecision), and postanalytical variation (reporting of results). Biological variation consists of within-person (WP) and between-person (BP) variation. These components of biological variation are used to set analytical quality specifications for bias and imprecision, evaluate serial changes in individual analytes, and assess the clinical utility of population-based reference intervals.Desirable quality specifications for imprecision (I), bias (B), and total error have been related to the WP CV (CV w) and the BP CV (CV g) of laboratory analytes. Imprecision should be ideally less than one half of the CV W, and bias should be 1.4) suggests that the reference interval is appropriate. The index of individuality ranged from 0.13 for serum alkaline phosphatase to 0.83 for serum bicarbonate (Table 1 ).In this study, BP and WP estimates of CV were obtained for some selective environmental and nutritional analytes. NHANES III provides a better estimate of CV g than do other short-term local studies because the NHANES III sample was nationally representative.
The CV w estimate was limited by the nonrandom, self-selected design and reflected a mean of 17 days between two measurements. The estimate of CV w could be improved by use of a stratified, multistage probability design over different time periods. Differences for CV w and CV g among subpopulations (gender, age, race, and ethnicity) can be further investigated by use of NHANES III data.
![](/uploads/1/2/7/6/127682432/606585372.jpg)