2015 Webinar Series
Biochemical Genetic Test Establishment and Verification
4/8/2015
Speakers
Tina M. Cowan, PhD, Associate Professor of Pathology and Director, Clinical Biochemical Genetics Laboratory,
Stanford University, Palo Alto, CA
Tina Cowan is an Associate Professor of Pathology at Stanford University Director of the Stanford Clinical
Biochemical Genetics Laboratory. She received her PhD in Genetics from UCLA and postdoctoral fellowship training
in Medical Genetics at the Universtiy of Maryland Baltimore. She is certified by the American Board of Medical
Genetics and Genomics in Clinical Biochemical Genetics, has served on a number of national committees and
boards for medical genetics, education, and laboratory quality.
Chunli Yu, M.D., FACMG, Associate Professor, Director of Biochemical Genetics Division, Mount Sinai Genetic
Testing Laboratory, Department of Genetics and Genomic Sciences, Icahn School of Medicine at Mount Sinai, New
York, NY
Dr. Yu is currently an Associate Professor of Genetics and Genomic Sciences Department at Icahn School of
Medicine at Mount Sinai and Director of Biochemical Genetics Division at Mount Sinai Genetic Testing Laboratory.
Dr. Yu was board certified in clinical biochemical genetics in 1999 and has extensive experiences in directing large
academic biochemical genetics laboratories. Dr. Yu’s professional career has been focused on the laboratory
diagnosis and monitoring of patients with inborn errors of metabolism of carbohydrates, fatty acids, amino acids, and
organic acids, as well as porphyrias and lysosomal storage diseases. She is particularly interested in applications of
LC-MS/MS technology in biochemical genetic testing and new assay development.
Objectives
At the conclusion of this program, participants will be able to:

Describe the basic elements of analytical test validation and their application to biochemical genetics testing.

Recognize the importance of addressing clinical validity in addition to analytical validity in the development of new biochemical
genetics tests.
Continuing Education Credit
4/8/15 – 10/8/15
The Association of Public Health Laboratories (APHL) is approved as a provider of continuing education programs in the clinical
laboratory sciences by the ASCLS P.A.C.E.® Program. Participants who successfully complete each program will be awarded 1.0
®
contact hours. P.A.C.E. is accepted by all licensure states except Florida. APHL is a Florida and CPH-recertification approved CE
provider; each course has been approved for 1.0 contact hours.
10/9/15 – 4/8/16
No CEU credit, but after completing the evaluation you will receive a certificate of attendance.
Evaluation/Printing Certificate
1. Go to https://www.surveymonkey.com/s/588-622-15nd and complete the evaluation.
2. Select: P.A.C.E.®, Florida or CPH-recertification credit.
3. Complete the survey.
4. Certificate printing
d. 4/8/15 – 10/8/15
i. PACE or Florida Certificates: add your information in the boxes and click on Submit. If you are requesting
Florida CEU, you must submit a valid Florida license number for the certificate and for us to enter your
information into CE Broker.
1. Please review your certificate. If you need change any information, go to the bottom of the page
and click on here to go back and edit. IMPORTANT: Enable printing of background images in
the print (Firefox) or page setup (Internet Explorer) dialog options.
2. Certificates are available only by selecting the Print button. They will not be emailed or mailed.
a. Print the certificate.
b. We recommend that you also print the certificate to an adobe file and save. There is a
$15 fee if you request a duplicate certificate later. Webinar certificates will no longer be
saved in the Continuing Education Center (CEC).
ii. CPH-recertification credit: you must submit a valid certification number on the evalulation. Your
information will be forwarded to the National Board of Public Health Examiners. At the end of the
evaluation, you will be able to print a Certificate of Attendance. Copy the Certificate of Attendance URL to
you browser, then in the next screen print your certificate.
e.
10/9/15-4/8/16
i. When the evaluation will close to the Certificate of Attendance. Type your name into the file and print.
Archived Program
The archived streaming video will be available within one day after the live program. Anyone from your site can view the Web archived
program and/or complete the evaluation and print the certificate for free. Registration is not necessary for the archive program. For Live
or archived site registrations, the URL will be sent to the site representative who is responsible for distributing the URL.
Comments, opinions, and evaluations expressed in this program do not constitute endorsement by APHL. The APHL does not authorize any program
faculty to express personal opinion or evaluation as the position of APHL. The use of trade names and commercial sources is for identification only and
does not imply endorsement by the program sponsors.
© This program is copyright protected by the speaker(s) and APHL. The material is to be used for this APHL program only. It is strictly
forbidden to record the program or use any part of the material without permission from the author or APHL. Any unauthorized use of the
written material or broadcasting, public performance, copying or re-recording constitutes an infringement of copyright laws.
Faculty Disclosure
Analysis. Answers. Action.
www.aphl.org
Biochemical Genetic Test
Establishment and Verification
The Association of Public Health Laboratories adheres to established standards
regarding industry support of continuing education for healthcare professionals.
The following disclosures of personal financial relationships with commercial
interests within the last 12 months as relative to this presentation have been
made by the speaker(s):
“Nothing to disclose”
Chunli Yu, MD
Tina Cowan, PhD
Analysis. Answers. Action.
1
Objectives
2
www.aphl.org
Regulations and Guidelines
for Test Establishment
• Describe the basic elements of analytical test validation
and their application to biochemical genetics testing.
• Recognize the importance of addressing clinical validity
in addition to analytical validity in the development of
new biochemical genetics tests.
CLSI
CLIA
ACMG
CAP
MMWR
Analysis. Answers. Action.
3
www.aphl.org
LDTS: Validation vs. Verification
www.aphl.org
ANALYTIC VALIDATION:
1. Establishment of performance specifications:
A type of in vitro diagnostic test that is designed,
manufactured and used within a single laboratory
• Validation
Process to confirm with objective evidence that a
laboratory-developed or modified FDA-cleared/approved
test performs as intended
• Verification
a.
b.
c.
d.
e.
f.
g.
Accuracy
Precision
Analytical sensitivity
Analytical specificity to include interfering substances
Reportable range of test results for the test system
Reference intervals (normal values)
Any other performance characteristic required for test performance
2. Determination of calibration and control procedures
3. Documentation
Process to confirm with objective evidence that
performance claims of a test have been met
5
4
Test Establishment: 42 CFR §493.1253
• Laboratory Developed Test (LDT)
Analysis. Answers. Action.
Analysis. Answers. Action.
[68 FR 3703, Jan. 24, 2003; 68 FR 50724, Aug. 22, 2003]
www.aphl.org
Analysis. Answers. Action.
6
www.aphl.org
Test Validation Elements
Clinical validity
Analytic validity
Analysis. Answers. Action.
•
•
•
•
•
•
•
•
Clinical Validity – MMWR Guidelines
Laboratories should ensure that tests are clinically relevant
and can be interpreted for specific clinical situations
Sensitivity
Specificity
Positive predictive value
Negative predictive value
• Documentation of clinical validity from available information
sources
• Establishment of clinical validity based on internal studies
using previously characterized positive and normal samples
Accuracy/Precision
Analytic sensitivity
Analytic interference
Reportable range
7
• Determination of test results that suggest imminent or
potentially life-threatening conditions, or of critical values or
alert values that warrant immediate medical attention
www.aphl.org
Clinical Validity Elements
Analysis. Answers. Action.
8
www.aphl.org
Definition of Terms
accuracy with which a test identifies a particular condition
Sensitivity
• proportion of people with condition
who test positive
Specificity
• proportion of people without
condition who test negative
Test Result
Positive
Positive
of people with positive
predictive value • proportion
result who have the condition
(PPV)
Test Result
Negative
Negative
of people with negative
predictive value • proportion
result who do not have the condition
(NPV)
Analysis. Answers. Action.
9
Burke W. Clinical validity and clinical utility of
genetic tests. Curr Protoc Hum Genet. 2009
Jan;Chapter 9:Unit 9.15.
www.aphl.org
Condition
Positive
Condition
Negative
True
Positive
False
Positive
PPV
False
Negative
True
Negative
NPV
Sensitivity
Specificity
Analysis. Answers. Action.
10
www.aphl.org
Factors Influencing Clinical Performance
Genetic Influences on Test Performance
• Analytic test performance (more later)
• Intermittent or subtle nature of some abnormalities
Certain fatty acid oxidation disorders (MCAD, VLCAD)
Intermittent maple syrup urine disease
Glutaric acidemia type I (low-excretor variant)
• Other clinical considerations
Diet (e.g., TPN, protein restriction, MCT)
Medication (e.g., antibiotics, antiepileptics)
Other therapies (e.g., dopamine, arginine, IVIG)
Clinical status (e.g., hypoxia, renal or hepatic failure,
pregnancy, catabolic state)
Analysis. Answers. Action.
11
• Enzyme pseudodeficiencies
e.gαglucosidase, β-hexosaminidase
www.aphl.org
Analysis. Answers. Action.
12
www.aphl.org
The Interpretation – General Guidelines
F1.2 In most cases, the analytical methods used by
biochemical genetics laboratories are similar to
those of standard clinical chemistry laboratories.
Accordingly, procedures for test validation, quality
control, quality assurance, and monitoring of safety
and equipment performance are generally the
same in both settings.
https://www.acmg.net/
Importantly, the biochemical genetics laboratory
differs from the clinical chemistry laboratory in the
extent of interpretation required in order to provide
a valid and meaningful result. Interpretation of
biochemical genetics tests should be provided by
an American Board of Medical Genetics (ABMG)certified clinical biochemical geneticist, ideally
taking into consideration the clinical history, results
of other tests, and other relevant parameters.
Ensuring a Meaningful Interpretation
Information provided to the laboratory
• Indication for testing
– Clinical presentation, abnormal NBS, positive family history, follow-up
of known condition
• Clinical history
– Diet, medications, nutritional status, other relevant factors
• Labs should seek information when needed
Knowledge of normal and disease states
• Elevated BCAA in catabolism vs. MSUD
• Enzyme pseudodeficiency
• Experience with as many conditions as possible
– Testing a broad range of known positives (challenging)
– Staying current with literature
– Collaboration is essential!
Analysis. Answers. Action.
13
www.aphl.org
Recognition of a Medical Emergency
MMWR: Determination of test results that suggest
imminent or potentially life-threatening conditions, or of
critical values or alert values that warrant immediate
medical attention
• Setting guidelines (vs. critical values)
– Determined by Medical Director
– Examples:
• New diagnosis in a previously unknown patient
• Elevated galactose-1-phosphate in a newborn
• Very high leucine or citrulline
Analysis. Answers. Action.
15
www.aphl.org
Pre-Validation Considerations (LC-MS/MS)
•
Thorough optimization of LC and mass spec parameters
•
Qualifier/quantifier SRM transition ratios
•
Selection of appropriate internal standards and concentrations
– Stable isotope labeled
– Structural analog
•
Evaluation of carryover
•
Evaluation of background noise level
17
Analysis.
Answers.
Action.
CLSI C62-A
www.aphl.org
14
Analysis. Answers. Action.
www.aphl.org
Analytic Validity
performance characteristics with which a test measures a particular analyte or analytes
Accuracy
Precision
• Closeness of agreement between an
individual value and a true value
• Closeness of agreement between multiple
independent test results
Analytic
sensitivity
• Lower limit of detection (LLOD)
• Lower limit of quantification (LLOQ)
Analytic
interference
• The laboratory must be aware of common
interferences by performing studies or
having available studies performed
elsewhere
Reportable
range
• Analytic measurement range (AMR), the
range of analyte values that a method can
directly measure on the specimen without
any dilution or concentration
Analysis. Answers. Action.
16
CLSI Statispro software
www.aphl.org
Quantitative Evaluation of Matrix Effects
(LC-MS/MS)
• Matrix effects (ME)
ME (%)= B/A x 100
• Recovery (RE)
RE (%) = C/B x 100
• Process efficiency (PE)
PE (%) = C/A x 100
• A: Neat standards
• B: Standards spiked to
the extracted matrix
• C: Standards spiked to
matrix and then
processed
Matuszewski BK, et al, Strategies for the assessment of matrix effect in quantitative bioanalytical methods
based on HPLC-MS/MS. Anal Chem. 2003; 75 (13): 3019-3030.
Analysis. Answers. Action.
18
www.aphl.org
Analyte and Matrix Stability
Analytic Accuracy
• No absolute guidance on experiment design
• Stability of analyte and sample matrix from collection to
result
• Testing homogenous aliquots under different conditions
over time
e.g. extreme temp, room temp, 4C, -20C
• Amino acids
• Acylcarnitines
Analysis.
• Enzymes
Answers.
Action.
– Biotinidase, GALT
closeness of agreement between an individual value
and a true value
• Approaches to evaluating accuracy:
– Measure recovery of spiked samples
– Split samples with an established laboratory
(CLSI EP09-A3: n = 40)
– Test known PT samples, compare to participant consensus
ERNDIM (http://www.erndimqa.nl/)
CAP BGL (carnitine)
CDC NSQAP
• Evaluate throughout entire analytic measurement range
Low (near LLOQ), mid, and high (near ULOQ)
19
www.aphl.org
Analytic Precision
Analysis. Answers. Action.
20
www.aphl.org
Analytic Sensitivity
closeness of agreement between independent test
results
• Lower limit of detection (LLOD)
Signal/noise ratio >3.0
• Repeat testing over time
Patient samples, standards, QC material
• Lower limit of quantification (LLOQ)
• Within-run precision (repeatability)
 20% precision;  20% bias
e.g., 20 replicates of same prep
(CLSI EP17-A2)
• Between-run precision (reproducibility)
e.g., repeat entire testing process daily for 20 days
• Calculation of coefficient of variation (CV%)
• Clinical implications of LLOQ
Careful validation of metabolites with clinical significance
at very low concentrations (e.g., citrulline in proximal urea
cycle defects)
= SD/mean x 100
• Comparison to acceptance criteria
e.g.,   10-15%,   20% at LLOQ or ULOQ
Analysis. Answers. Action.
21
www.aphl.org
Reportable Range
Analytic Measurement Range (AMR)
Analysis. Answers. Action.
22
www.aphl.org
Analytic Interference
• Identification and characterization of interferents
• Analyze replicates throughout concentration range
• Calculate precision and bias at each concentration level
Acceptance criteria, e.g., CV  20%, Bias  20%
• Evaluate linearity, compare to criteria
e.g., R2>0.995
Literature search
Testing samples with known or potential sources of
interference
Hemolysis, icterus, lipemia, various collection tube additives
• Specific examples for biochemical genetics:
Amino acid analysis
Hemolysis (glu, asp, tau); ampicillin (phe)
• Other considerations
Matrix effects
Clinical reportable range
Dilution protocol
23
Analysis.
Answers.
Action.
Acylcarnitine profile
Pivalic acid containing antibiotics (C5); cefotaxime and IV dextrose
(C16-OH)
Biotinidase
Sulfa drugs
www.aphl.org
Analysis. Answers. Action.
24
www.aphl.org
Considerations of Enzyme Testing
Reference Interval
• Analytic specificity
• Pediatric samples might be challenging to obtain
• Certain specimen types are difficult to obtain
– CSF, muscle, fibrablasts
• Normal ranges vs disease ranges
– Affected range (enzyme)
– Carrier range (Hex A%)
• Comparing to published literature ranges
• Periodic verification after test implementation
Analysis. Answers. Action.
25
– Use of specific inhibitors
• Pompe disease - acarbose
• Fabry disease – α-N-acetylgalactosamine
• Interference from clinical status
– Pregnancy, diabetes, prematurity
• Reference interval
– Normal range and affected range
– Non-carrier range and carrier range
www.aphl.org
Test Implementation – The Basics
•
•
•
Clinical validity (literature references or internal studies)
Analytic validity
Quality plan
•
•
•
•
Standard operating procedure
Proficiency testing procedures
Personnel training and competency
Documentation and review of validation study
Analysis. Answers. Action.
www.aphl.org
Test Implementation – Other Details
•
•
•
•
•
•
– QC material, frequency, thresholds, QA review and documentation
26
Practitioner and patient education materials
Sample collection and handling instructions
Test requisition form
Billing codes
Lab website, test directory, CAP activity menu
Results reporting
– Format
– Disclaimer (COM.40630)
Reports for laboratory-developed tests (LDTs) contain a statement that the assay was developed
by the laboratory.
Example: "This test was developed and its performance characteristics determined by <Laboratory X>. It
has not been cleared or approved by the FDA. The laboratory is regulated under CLIA as qualified to
perform high-complexity testing. This test is used for clinical purposes. It should not be regarded as
investigational or for research.”
Analysis. Answers. Action.
27
www.aphl.org
Last But Not Least….
www.aphl.org
• New tests must be completely validated, even if they
have already been validated by another laboratory
There is a summary statement, signed by the laboratory director (or designee who
meets CAP director qualifications) prior to use in patient testing, that includes the
evaluation of validation/verification studies and approval of each test for clinical
use.
The summary statement must include a written assessment of the validation/verification study, including the
acceptability of the data. The summary must also include a statement approving the test for clinical use with the
approval signature such as, "This validation study has been reviewed, and the performance of the method is considered
acceptable for patient testing."
For … LDTs, the summary must address analytical sensitivity, analytical specificity and any other parameter that is
considered important to assure that the analytical performance of a test (e.g. specimen stability, reagent stability,
linearity, carryover, and cross-contamination, etc.), as appropriate and applicable.
If the laboratory makes clinical claims about its tests, the summary must address the validation of these claims.
29
28
In Summary
COM.40000
Method Validation/Verification Approval
Analysis. Answers. Action.
Analysis. Answers. Action.
www.aphl.org
• The laboratory is responsible for assuring and
documenting clinical claims and analytic performance
• Clinical validation should be performed and
documented wherever possible, but can be
challenging for some areas of biochemical genetics
• Analytic validation requirements do not differ from
other laboratory sections, but approaches for
implementation may be unique
Analysis. Answers. Action.
30
www.aphl.org
Analysis. Answers. Action.
www.aphl.org
Thank you!!
31
Comments/concerns about this program? Email [email protected]