5/3/2013
How to Perform Molecular
Proficiency Testing and
External Quality Assessment
Lisa Kalman, PhD
Laboratory Research and Evaluation Branch
Division of Laboratory Science and Standards
Office of Surveillance, Epidemiology, and Laboratory Services
Centers for Disease Control and Prevention
Atlanta, Georgia, USA
The findings and conclusions in this report are those
of the author and do not necessarily represent the
official position of the Centers for Disease Control and
Prevention/the Agency for Toxic Substances and
Disease Registry.
2
1
5/3/2013
Polling Question
Do you represent a proficiency testing/external quality
assessment (PT/EQA) provider or a medical (clinical)
laboratory?
A. PT/EQA provider
B. Medical (clinical) laboratory
3
Polling Question
What is your level of experience with proficiency testing?
A. No experience
B. Some experience
C. Extensive experience
4
2
5/3/2013
Polling Question
For users (medical laboratories):
What are the biggest challenges/problems with the PT/EQA
that you use?
A. It does not cover all tests that my laboratory offers.
B. I do not agree with grading of laboratory performance.
C. PT/EQA samples do not cover the range of analytes in a given
test.
D. It is insufficiently comprehensive. Challenges should include
an evaluation of sample preparation, test result
interpretation, and other stages of the testing process.
E. Other
5
Polling Question
For PT/EQA providers:
What are the biggest challenges/problems with the provision
of PT/EQA?
A. We cannot offer PT/EQA for all available laboratory tests.
B. Grading is difficult.
C. We have problems obtaining appropriate materials to
produce PT/EQA samples.
D. We would like to provide PT/EQA for a variety of steps in
clinical laboratory testing, but cannot for many reasons.
E. Other
6
3
5/3/2013
Why Is PT/EQA Important?
Required by some accreditation bodies and
regulations
Allows medical laboratories to compare analytical
performance to other laboratories using similar or
different methods
Helps laboratories to identify:
• Analytical and interpretive errors
• Internal problems with quality control, calibration, and
assay design (systematic errors)
7
Issues Related to Provision of
PT/EQA for Molecular Tests
No PT/EQA available for most molecular genetic tests
due to:
•
•
•
•
Large number of tests available
Small number of laboratories that offer each test
Evolving technologies
Test complexity
Lack of suitable PT/EQA samples to represent the full
range of measurands detected
Difficulties of designing PT/EQA to assess all phases of
the testing process
8
4
5/3/2013
Approaches to Address
Lack of PT/EQA
Laboratories and PT/EQA programs facilitate
sample exchange.
Perform method-based PT/EQA.
Develop new PT/EQA materials.
9
Objectives of CLSI Document MM14
Describe practices that provide effective
PT/EQA or sample exchange to medical
laboratories for molecular diagnostic tests.
Describe method-based PT/EQA that may
evaluate a wide variety of tests for molecular
markers that are performed using the same
test method.
10
5
5/3/2013
Purpose of CLSI Document MM14
Provide guidance for PT/EQA programs as well as laboratories that provide
PT/EQA through sample exchange
Serve as a benchmark for medical laboratories for evaluation of PT/EQA
programs
Supply a description of basic principles and practices for PT/EQA, including:
•
•
•
•
•
•
Design of PT/EQA schemes
Information about material sourcing
Sample production
Sample transport
Documentation
Result reporting
Describe practices that can increase the types of laboratory processes that
may be evaluated, including:
•
•
Laboratories acting as PT providers (sample exchange)
Method-based PT/EQA
11
Scope of CLSI Document MM14
MM14 covers:
Design of PT programs
Guidance for both large formal PT/EQA programs and
medical laboratories that facilitate sample exchange
Principles of nucleic acid (DNA and RNA) PT/EQA in areas
of human genetics, infectious disease, molecular oncology,
and pharmacogenetics
May also be applicable to programs outside of nucleic acid
testing
12
6
5/3/2013
What Is New in the Second Edition?
Updated information about regulatory and guidance
documents
Additional sections describing method-based PT and
alternative assessment strategies
Emphasis on the roles and responsibilities of the medical
laboratory as a PT provider through informal sample
exchange programs
Appendixes with examples of forms, reports, and other
documents used by PT participants and programs
13
CLSI Document MM14
A brief peek inside the pages!
14
7
5/3/2013
Design of PT/EQA
PT vs EQA
PT – Determination of laboratory testing performance
by means of interlaboratory comparisons
• Evaluates the accuracy of measurement procedures as
compared to an external measure
• Primarily focuses on examination (analytical) phase of
testing, but does thoroughly evaluate result interpretation
15
Design of PT/EQA (cont’d)
PT vs EQA
EQA – Evaluation of the laboratory’s performance on
examination of samples of external origin for the
purposes of determining adequacy of the laboratory’s
preexamination, examination, and postexamination
activities (ISO/IEC 17043*)
• Evaluates the accuracy of testing, but, in addition, focuses on
educational value of interlaboratory comparison
• Assesses methods of analysis, interpretation, and reporting
* ISO/IEC 17043, Conformity assessment – General requirements for proficiency testing.
16
8
5/3/2013
Design of PT/EQA (cont’d)
Regulatory Considerations
PT/EQA programs should be designed to meet appropriate
regulatory and accreditation requirements.
These requirements depend on the type of test and
measurement as well as national and local regulations.
(MM14 Appendix A includes globally influential documents that address
PT/EQA.)
17
Design of PT/EQA (cont’d)
Sample Composition
Need to consider:
•
•
•
•
Homogeneity
Stability
Choice of sample source materials
Sample safety
18
9
5/3/2013
Design of PT/EQA (cont’d)
Four Common Sources of PT/EQA Sample
Materials
1. Residual patient samples (eg, blood, urine)
2. Derivatives of patient samples (eg, extracted DNA)
3. Cultured cells (eg, tissue culture cells, pathogens)
4. Engineered constructs (eg, amplicons, clones)
All sources have advantages and disadvantages (described in
MM14 Table 1).
19
Design of PT/EQA (cont’d)
Number and Variety of Test Samples
Number of PT samples per challenge depends on the test.
•
•
Quantitative test – need samples at a variety of
concentrations
Qualitative test – should represent the variants or analytes
that the test is designed to detect, may alternate variants or
analytes over several challenges
Number of samples may also depend on regulatory
requirements.
20
10
5/3/2013
Design of PT/EQA (cont’d)
Interpretation of Participant Results
Acceptance criteria may be based on:
•
•
•
Total allowable error of well-characterized reference
material
Participant laboratory consensus
Referee laboratory consensus
21
Design of PT/EQA (cont’d)
Interpretation of Participant Results (cont’d)
Methods for grading should be objective and unambiguous
and should be clearly defined before the survey.
Interpretation should reference relevant literature,
consensus documents from expert panels or professional
societies, and/or other guidance documents.
22
11
5/3/2013
Process Checklist
PT/EQA providers should develop and maintain a quality management system
(QMS).
CLSI document GP26* describes a “path of workflow” that encompasses all
steps of the development of a PT program.
PT/EQA providers should develop a process checklist with timeframes for:
• Participant sign-up
• Communication of instructions
• Shipment of materials
• Timeframe for performance testing
See MM14 Appendix B
• Submission of results
for a process checklist
example.
• Result scoring and analysis
• Feedback to laboratories
• Reporting summary results to participants
* CLSI document GP26, Quality Management System: A Model for Laboratory
Services.
23
Material Sourcing/Collection
Need to consider:
•
•
•
•
Laws and guidelines governing human specimen
acquisition
Legal, ethical, and societal concerns
Donor confidentiality and privacy
Material safety
24
12
5/3/2013
Material Sourcing/Collection
(cont’d)
Size, scope, and medical/technical needs of the
PT/EQA survey will dictate the most appropriate
source of the materials.
• PT/EQA programs for high-prevalence conditions such as
infectious diseases need large quantities of homogeneous
samples. Commercial biological product suppliers may be
able to provide materials.
• Programs for rare disorders, or with difficult to obtain
samples (such as tumor biopsy specimens), may collaborate
with a clinical entity to obtain appropriate materials.
25
Material Sourcing/Collection
(cont’d)
Actual patient samples, such as whole blood, would be ideal PT
samples, because they would:
• Allow all phases of testing (including sample preparation) to
be evaluated.
• Contain the biological matrix encountered with actual
patient samples.
Patient samples are not practical, because they:
• Are difficult to obtain, especially in large amounts
• May be unstable
• May be infectious
• May be heterogeneous
MM14 Table 1 describes the advantages and
disadvantages of various sample types.
26
13
5/3/2013
Material Sourcing/Collection
(cont’d)
Other Source Materials (rather than patient samples)
Derivatives of patient samples (eg, extracted DNA)
•
•
Stable, allow dilution of target, not infectious
Do not evaluate extraction, lack biological matrix
Cultured cells (eg, tissue culture cells, pathogens)
•
•
Easy to source, allow challenges with rare mutations, contain total gene
sequence
Do not evaluate extraction, lack biological matrix
Engineered constructs (eg, amplicons, clones)
•
•
Well-defined sequence, easy to produce, stable
Low complexity, may not have all gene sequences, may not work in
assays
27
Sample Production/Manufacture
Sample Preservation
The PT/EQA provider must ensure that the appropriate storage and
handling conditions/methods are followed throughout the entire
supply chain preceding receipt by participant.
Sample preservation issues:
• Preservatives may affect sample integrity and may not be
compatible with methods used by PT participants
• Shipping temperatures
• Microbial or DNA/RNA contamination of PT samples
• Nuclease activity
28
14
5/3/2013
Sample Production/Manufacture
(cont’d)
Sample Characterization
The PT/EQA provider should extensively characterize the test
samples to be used by PT/EQA participants.
Characterization should be performed in expert laboratories
using reference methods and/or methods used by participants.
Need to assess:
• Presence or absence of measurand or mutation and
quantification, as applicable
• Stability
• Homogeneity
• Recoverability
29
Sample Production/Manufacture
(cont’d)
PT Material Stability
PT/EQA providers should conduct stability testing to ensure that
samples perform properly throughout the challenge period.
Real-time stability testing
• Test samples held at recommended storage conditions at
periodic time points.
Open vial stability testing
• Evaluate stability of products subjected to a variety of
environmental stress conditions that the sample might
encounter during shipping and handling.
30
15
5/3/2013
Sample Production/Manufacture
(cont’d)
PT/EQA providers should retain residual product for
use in:
• Corrective action plans (both PT provider and laboratory)
• Competency assessments
• Replacement of damaged or degraded samples
31
Sample Transport
United States
• All hazardous material packages are subject to US
Department of Transportation regulations.
International
• Hazardous materials are subject to requirements
of the United Nations International Civil Aviation
Organization and the International Air Transport
Association (IATA).
See MM14 Appendix C for useful websites for
information on shipping PT/EQA samples.
32
16
5/3/2013
Sample Transport (cont’d)
Issues to consider:
•
•
•
•
Will the package be irradiated?
What is the transport time?
What is the transport temperature?
What are the regulations and requirements for
customs, etc. (which vary by country)?
33
Sample Transport (cont’d)
IATA Specimen Types
Infectious substances – known to contain or may contain
pathogens that can cause disease in humans or animals
• Category A – is capable of causing permanent disability
or life-threatening or fatal disease
• Category B – infectious substances not in Category A
34
17
5/3/2013
Sample Transport (cont’d)
PT/EQA programs need to:
•
•
•
•
Categorize the specimens according to the correct IATA
Category A or B.
Package samples appropriately according to IATA
regulations.
Obtain the necessary permits for customs clearance.
Determine appropriate sample rejection criteria by the
recipient and procedures for correction.
35
Information Exchange
The PT/EQA provider needs to develop the appropriate
documents to exchange information with the participants,
assessors, and other parties. These include:
•
•
•
•
•
•
General information about PT/EQA scheme
Timeline (see MM14 Appendix B)
Registration and confirmation (see MM14 Appendix D)
Letter of instruction for participants
Result reporting form
Report from provider to individual participant with
laboratory performance scores
• Summary report from provider to all participants
36
18
5/3/2013
Information Exchange (cont’d)
General information about PT/EQA scheme includes:
•
•
•
•
•
Aims of PT/EQA scheme
Frequency of scheme (number of events per year)
Details of scheme design and procedures
Terms and conditions
How to apply
This information may be provided on the program's website
or in its catalog.
37
Information Exchange (cont’d)
Letter of instruction for participants includes information about:
•
•
•
•
•
Measurand, nature of sample
Testing methods that can be used by participants
Product reconstitution and storage
Safety warnings
Warnings against sharing PT samples and results with other
laboratories
• Instructions for analysis
• Instructions for reporting, date to submit results
• Clinical information about PT/EQA sample
See MM14 Appendix E for an example instructions manual.
See MM14 Appendix F for examples of clinical data.
38
19
5/3/2013
Information Exchange (cont’d)
Result reporting form can be electronic or hard copy and should
include:
•
•
•
•
•
Sample identifier
Test method used
Assay limitations (limit of detection [LoD], mutations tested, etc.)
Results and interpretation if required
Reporting units
Result reporting form should require accepted and appropriate
nomenclature.
PT/EQA provider should validate, protect, and back up
electronic records and systems.
See MM14 Appendix G for an example
of a result reporting form.
39
Information Exchange (cont’d)
Reports to Participants
Summary report to all participants should include:
• Information about the PT/EQA scheme and provider
• Description of PT/EQA sample
• Participants’ results, both individual method/peer group and
aggregate group results
• Statistical data and summaries, assigned values, range of
acceptable results
• Procedures used to establish assigned values
• Summary statistics for test methods/procedures used by
participants
• Comments on performance by PT/EQA provider
See MM14 Appendix I for an example of a participant summary report.
40
20
5/3/2013
Information Exchange (cont’d)
Summary report to individual participants should include
comments on:
• Variation within and between participants, comparisons to
previous PT
• Possible sources of error, suggestions for improving
performance
• Advice and educational feedback
• Situations in which unusual factors makes evaluation
impossible
41
Provider Results Review and
Evaluation
How is the “correct” result defined?
If the sample is well characterized, participant results can
be compared to the expected values.
If the sample is not well characterized, programs may:
• Define correct result as a consensus of participants
results (eg, 80% consensus, may need to group by test
method).
• Define correct result using referee laboratories.
Both of these approaches may be difficult if the survey
material is not homogenous.
42
21
5/3/2013
Provider Results Review and
Evaluation (cont’d)
PT/EQA results may not be graded if:
• There are too few participants in a method group to
make a valid statistical assessment.
• The consensus among participants is not high enough
(eg, <80%).
• The participants use a variety of assays (eg, laboratorydeveloped tests).
• The survey material is too heterogeneous (eg, tumor
tissue).
• A very new measurand is tested.
43
Provider Results Review and
Evaluation (cont’d)
Qualitative Results
The range of acceptable positive results may depend on the
method used and the intended use of the test.
• Do participating laboratories use methods able to
detect measurands targeted for the PT challenge?
• What are the stated limitations of the assay (eg, does
the PT sample have a measurand concentration below
the LoD of the assay)?
The report to participants should note when clinical practice
has an accepted minimum level of performance.
44
22
5/3/2013
Provider Results Review and
Evaluation (cont’d)
Quantitative Results
•
•
•
•
Quantitative results should be evaluated based on acceptable
ranges statistically determined by participant/referee
responses or expected absolute target value.
Acceptable ranges can be determined using statistical
calculations or predetermined ranges.
Stated limitations of assays and clinical significance of values
should be taken into account.
Participants should be required to report results using widely
accepted units of measure so results can be compared easily.
45
Complementary and Supplementary
Approaches to PT/EQA
PT/EQA is available for only a small percentage of molecular
genetic tests.
• In the absence of a formal PT/EQA program, laboratories use
alternative methods of assessment including sample
exchange (described in CLSI document GP29*).
• When laboratories exchange samples, they become PT/EQA
providers! (Information from MM14 is useful and can
improve the quality of the exchange.)
• Some PT/EQA providers have programs to assist laboratories
with sample exchange.
* CLSI document GP29, Assessment of Laboratory Tests When Proficiency
Testing Is Not Available.
46
23
5/3/2013
Complementary and Supplementary
Approaches to PT/EQA (cont’d)
Method-based PT/EQA assesses the test performance, not
specific measurands, and can help to evaluate tests and
procedures not included in PT/EQA schemes.
Method-based PT/EQA can be used to evaluate basic procedures
common to many tests.
•
•
•
•
•
•
•
•
DNA extraction
Polymerase chain reaction (PCR) amplification
Cycle sequencing
Next-generation sequencing
MM14 provides examples and
Quantitative real-time PCR
references describing a variety of
Electrophoresis
method-based PT/EQA schemes
DNA/RNA quantification
and their results.
Interpretation and reporting
47
Authors of
CLSI Document MM14
Lisa Kalman, PhD
Roberta M. Madej, CLS, MS, MBA
Elisabeth Dequeker, Prof Dr, PhD
Marie C. Earley, PhD
Vivianne Gomo, MPH, BSc
Cynthia L. Jackson, PhD
Lawrence J. Jennings, MD, PhD
Mario Pazzagli, PhD
Ted E. Schutzbank, PhD, D(ABMM)
Heather Stang, MS
Paul Wallace, PhD
Amanda E. Weiss, MLS(ASCP)
48
24
5/3/2013
Polling Question
Do you think that using the new edition of MM14 would help
you to improve your use or provision of PT/EQA?
A. Yes
B. No
49
Questions?
Thanks!
Lisa Kalman, PhD
[email protected]
50
25