How to Evaluate new Instruments Carol Briggs Department of Haematology University College London Hospitals ICSH Session, Toronto, May 2013 Levels of evaluation • Complete operational evaluation This should be done at a national evaluation centre or an independent organisation • Local laboratory evaluation/validation An abbreviated assessment appraising limited aspects of the equipment in the its intended location and by staff who will be operating the instrument Complete operational evaluation • Type of evaluation – National (e.g. FDA,) – Beta site Planning the Evaluation • Analyser availability • Time required • Staff time • Reagent quantities/Batches • Training • Contingency plan Preliminary assessment • Requirements: Space, power, operating environment… • Safety – Mechanical – Electrical – IEC 61010-1 – Chemical – Microbiological – aerosol, splashing, waste disposal National/Complete evaluation • • • • • • • Precision/Reproducibility Carryover Linearity Stability Comparability (accuracy) Efficiency Storage of results and quality control Precision testing 1. Within run precision. The method usually consists of a single run of 20 measurements on the same sample, all reported parameters should be analysed. Normal, abnormal low and abnormal high samples for WBC, Hb and platelets should be sought. The results for mean, SD and CV% should be reported. Precision testing 1 2 3 4 5 6 7 8 9 10 MEAN SD 2SD CV MIN MAX DIFF WBC 4.63 4.54 4.57 4.53 4.49 4.48 4.55 4.57 4.33 4.44 4.51 0.08 0.16 1.76 4.33 4.63 0.30 NEUT# 2.6 2.62 2.6 2.58 2.52 2.53 2.52 2.64 2.41 2.41 2.54 0.08 0.15 3.04 2.41 2.64 0.23 LYMPH# 1.45 1.4 1.46 1.42 1.4 1.39 1.41 1.29 1.36 1.4 1.40 0.05 0.09 3.23 1.29 1.46 0.17 MONO# 0.39 0.34 0.35 0.38 0.38 0.39 0.44 0.45 0.38 0.41 0.39 0.03 0.07 8.44 0.34 0.45 0.11 EOSI# 0.16 0.15 0.13 0.13 0.16 0.14 0.14 0.14 0.14 0.17 0.15 0.01 0.03 8.77 0.13 0.17 0.04 BASO# 0.03 0.03 0.03 0.02 0.03 0.03 0.04 0.05 0.04 0.05 0.04 0.01 0.02 26.34 0.02 0.05 0.03 RBC 4.22 4.3 4.27 4.3 4.31 4.26 4.28 4.33 4.28 4.28 4.28 0.03 0.06 0.67 4.22 4.33 0.11 HGB 12.9 13.1 13.1 12.9 13.1 12.9 12.9 12.9 12.9 13 12.97 0.09 0.18 0.69 12.90 13.10 0.20 HCT 41.1 41.9 41.6 41.8 41.9 41.4 41.6 42 41.6 41.5 41.64 0.26 0.52 0.62 41.10 42.00 0.90 MCV 97.4 97.4 97.4 97.2 97.2 97.2 97.2 97 97.2 97 97.22 0.14 0.28 0.14 97.00 97.40 0.40 MCH 30.6 30.5 30.7 30 30.4 30.3 30.1 29.8 30.1 30.4 30.29 0.27 0.54 0.89 29.80 30.70 0.90 MCHC 31.4 31.3 31.5 30.9 31.3 31.2 31 30.7 31 31.3 31.16 0.24 0.47 0.76 30.70 31.50 0.80 PLT 203 211 207 207 216 211 214 201 199 201 207.00 5.60 11.21 2.71 199.00 216.00 17.00 Precision testing 2. Between batch precision. Between batch precision may be affected by calibration or drift. A single measurement on a sample repeated each day for a period 20-30 days is used to measure total between day (batch) precision for all parameters Carryover • Carry-over is defined as the contamination of a sample by the sample analysed immediately preceding it. • Carry-over from a high sample to a low sample should be assessed by running sample A (high sample) three times, A1, A2, A3, followed by sample B (low sample) three times B1, B2, B3. Carry-over % is calculated by: B1 – B3 A3 – B3 x 100 Linearity • There should be a linear relationship between the parameter measured and at various dilutions over a large a range as possible. • Dilutions should be chosen to include the entire pathological range, from the highest counts to the lowest. • In laboratories where very low WBC and platelet counts are encountered it is advisable to examine linearity in the low range separately. • Replicate tests should be performed to give results at evenly spaced concentrations. • Certain haematological parameters, e.g. red cell indices will not be affected by dilution of a sample. Linearity WBC LINEARITY (100) PLT-O LINEARITY (4000) y = 23.766x - 0.0779 R2 = 0.9996 120 PLT-O X 10^9/L WBC X 10^9/L 100 80 60 40 20 0 0.0 1.0 2.0 3.0 4.0 4500 4000 3500 3000 2500 2000 1500 1000 500 0 5.0 y = 358.52x - 124.48 R2 = 0.9955 0.0 2.0 HB CONCENTRATION g/dL 10.0 12.0 14.0 y = 5.1884x + 0.6034 R2 = 0.9964 60 50 PLT-O X 10^9/L WBC X 10^9/L 8.0 LOW PLT-O LINEARITY. y = 0.0956x - 0.0101 R2 = 0.9915 0.8 6.0 HB CONCENTRATION g/dL LOW WBC LINEARITY 1 4.0 0.6 0.4 0.2 0 40 30 20 10 0 0 2 4 6 8 HB CONCENTRATION g/dL 10 12 0.0 2.0 4.0 6.0 8.0 HB CONCENTRATION g/dL 10.0 12.0 Linearity NRBC LINEARITY y = 0.8952x - 0.0532 R2 = 0.9971 NRBC x 10^9/L 80 60 40 20 0 0 100 90 80 70 60 50 40 30 20 10 0 10 20 30 40 50 60 70 80 90 100 y = 0.7603x + 0.6831 R2 = 0.7872 0 Hb CONCENTRATION y = 0.9769x + 0.5791 R2 = 0.7557 11 10 9 8 7 6 5 4 3 2 1 0 0 1 10 20 30 40 50 60 70 80 90 100 Hb Concentration WBC Linearity on NRBC Sampes. WBC x 10^9/L NRBC X 10^9/L 100 NRBC Linearity 2 3 4 5 6 7 Hb Concentration 8 9 10 Linearity Flow v Analyser 2 NRBC/100 WBC 700 700 600 600 Analyser 2 NRBC/100 WBC Analyser 1 NRB C/100 WBC Flow v Analyser 1 NRBC/100 WBC y = 0.8723x + 1.8398 R2 = 0.9862 500 400 300 200 100 0 y = 0.5419x + 20.971 R2 = 0.6299 500 400 300 200 100 0 0 100 200 300 400 500 Flow NRBC/100 WBC 600 700 0 100 200 300 400 Flow NRBC/100 WBC 500 600 700 Linearity Sample Stability • Ability of a sample to retain the same value for a measured quantity for a defined period. • 5 normal and 5 abnormal samples should be analysed at time zero and then analysed over time for up to 72 hours. • Samples should be stored at room temperature and 40C. Sample Stability 100 2 90 3 80 4 70 5 60 MCV LONG TERM STABILITY 4 DEG C. ABNORMALS 1 2 3 4 5 80 75 70 65 60 55 50 45 40 65 33 60 70 60 50 3 4 5 MCHC pg 80 2 32 31 30 29 28 1 2 3 4 5 RDW-SD 34 100 1 12 1 11 Series1 2 10 Series2 9 Series3 8 Series4 7 Series5 3 4 5 6 MPV LONGTERM STABILITY 4 DEG C. ABNORMALS RDW-SD LONG TERM STABILITY 4 DEG C. ABNORMALS MCHC LONG TERM STABILITY 4 DEG C. ABNORMALS 110 90 MCV fl 35 34 33 32 31 30 29 28 27 26 25 MPV fL 1 RDW-SD fL 110 MCHC g/dL MCV fl 120 MPV LONGTERM STABILITY ROOM TEMP ABNORMALS RDW-SD LONG TERM STABILITY ROOM TEMP. ABNORMALS MCHC LONG TERM STABILITY ROOM TEMP. ABNORMALS 1 55 2 50 3 45 40 12 11 4 5 MPV fL MCV LONG TERM STABILITY ROOM TEMP. ABNORMALS Series1 10 9 8 Series2 7 6 Series5 Series3 Series4 Comparability Comparability rather than accuracy • • Accuracy implies a ‘true’ value Not applicable to many tests • Reference values are available for - Hb - RBC - PCV - WBC - Retics - Leucocyte Differential - Platelets Reference platelet count Reference platelet count Platelets Debris Plt/RBC Coincidence RBC Reference Leucocyte Differential • This is the CLSI H20-A2 method • 200 x 200 cell differential • Referee performs a 200 cell differential if there is disagreement between the first differentials Reference Leucocyte Differential • Classification of cells subjective Rümke Table – 95% CI a n=100 n=200 n=500 n=1,000 n=10,000 0 0-3.6 0-1.8 0-0.7 0-0.4 0-0.1 1 0.0-5.4 0.1-3.6 0.3-2.3 0.5-1.8 0.8-1.3 5 1.6-11.3 2.4-9.0 3.3-7.3 3.7-6.5 4.5-5.5 10 4.9-17.6 6.2-15.0 7.5-13.0 8.2-12.0 9.4-10.7 15 8.6-23.5 10.4-20.7 12.0-18.4 12.8-17.4 14.3-15.8 20 12.7-29.2 14.7-26.2 16.6-23.8 17.6-22.6 19.2-20.8 30 21.2-40.0 23.7-36.9 26.0-34.2 27.2-32.9 29,1-31,0 40 30.3-50.3 33.2-47.1 35.7-44.4 36.9-43.1 39.0-41.0 50 39.8-60.2 42.9-57.1 45.5-54.5 46.9-53.1 49.0-51.0 70 60.0-78.8 63.1-76.3 65.8-74.0 67.1-72.8 69.0-70.9 80 70.8-87.3 73.8-85.3 76.2-83.4 77.4-82.4 79.2-80.8 90 82.4-95.1 85.0-93.8 87.0-92.5 88.0-91.8 89.3-90.6 100 96.4-100 98.2-100 99.3-100 99.6-100 99.9-100 ICSH Reference flow differential – Dr Brent Wood, USA and Mikael Roussel, France • Replace morphology (CLSI H20-A2) with flow cytometry as the reference method for performing nucleated cell identification in blood • Must be able to be used on common instruments in use in the clinical laboratory • 5 colour or more the requirement Candidate Protocols • French – Faucher, et al. Cytometry A (2007) 71A; 934-944 – 5 color single tube – Lyse (Versalyse), no-wash • Swedish – Bjornsson, et al. Cytometry B (2008) 74B: 91-103 – 5 color single tube – Lyse (formic acid), no-wash • Seattle – Wood, et al. Unpublished – 5 color dual tube – Lyse (NH4Cl+formaldehyde or Versalyse), no-wash Core Populations French Swedish French Seattle Swedish Seattle Lymphocytes Lymphocytes ● ● ● Monocytes ● ● ● T cells Neutrophils ● ● ● NK cells Immature grans ● ● ● B cells ● ● ● Plasma cells ● ● pDCs ● ● ● Blasts ● ● ● CD16+ Monocytes Mast cells nRBCs ● ● ● ● ● Basophils ● Reactive Cytotoxic Eosinophils ● ● ● ● ● ● 1 2: A 3: A and C 4: A and C and F SSCè B D C 5: A and C and E E F G H CD19 ECDè SSCè CD45 Cy7è 6: A and C and F and G 7: A and C and F and G CD45 Cy7è CD36 FITCè CD16+ neutrophils CD16 Cy5 è WBC CD2+CRTH2 PEè A 8: A and C and F and H CD45 Cy7è CD45 Cy7è SSCè CD45 Cy7è CD45 Cy7è Imm Gran CD16pos Mono CD36 FITCè B-cells 11: A and D and I I SSCè CD2 + CRTH2 PEè B-blasts 10: A and D 9: A and B Cytotox My+ blasts SSCè Basophils SSCè Non cytotoxic T-cells CD36 FITCè SSCè CD16neg Monocytes CD45 Cy7è Eosinophils CD2 + CRTH2 PEè Drs. Feuillard, Koksch, Simon-Lopez SWE, US, FR 100 90 100 Neuts 80 80 70 70 60 60 50 y = 1.0026x - 4.3555 R² = 0.93665 40 30 y = 0.9552x - 4.9922 R² = 0.93808 20 0 0 20 40 60 80 50 y = 0.7353x + 1.4586 R² = 0.85695 40 30 y = 0.9804x + 1.5578 R² = 0.8758 y = 0.9026x + 2.5612 R² = 0.86061 20 y = 1.0095x - 7.4966 R² = 0.95214 10 10 0 100 0 60 20 40 60 80 100 70 50 FCM % Ly 90 Imm Gran 40 Blast 60 50 40 30 30 20 20 10 10 0 0 10 20 Manual Dif 30 40 50 60 0 0 10 20 30 40 50 60 70 Comparability • Comparison of the evaluation instrument’s results, or assay, to those obtained by another designated method should be examined for as many samples as possible. • These should include normal samples, abnormal samples, covering the entire pathological range, and samples with interfering substances. Abnormal WBC Abnormal RBC Abnormal Platelets Interfering Substances Extreme leucocytosis Sickle cells Giant platelets Haemolysis Extreme leucopenia Target cells Platelet clumps Cryoglobulins Blast cells Fragmented cells Paraproteins Atypical lymphocytes Microcytic cells High bilirubin Smear/Smudge cells Macrocytic cells Lipaemia Immature Granulocytes Spherocytes Left shift/band neutrophils Extreme polycythaemia Extreme anaemia Nucleated red blood cells Reticulocytosis Howell Jolly bodies Heinz bodies Pappenheimer bodies Malarial parasites Comparability • The paired t-test (when results are normally distributed) • For non-Gaussian data the Wilcoxon Rank Sum test or Mann-Whitney U test should be used • A p-value of less than 0.05 is usually considered as statistically significant Comparability • When discrepant results are found between the instruments, where possible, the samples should be measured using the reference methods previously described • All samples must have the CLSI 2 x 200 cell reference leucocyte differential Comparability • The morphology of cells need to be assessed in order to compare the efficiency of the suspect abnormal cell flags generated by the instruments. • The sensitivity, specificity, positive predictive value, negative predictive value and overall efficiency should be calculated for each individual flag Comparability Results from Gold Standard Reference Method True i.e. abnormality present False i.e. abnormality absent Positive for abnormality True Positive (TP) False Positive (FP) PPV = TP / (TP + FP) Negative for abnormality False Negative (FN) True Negative (TN) NPV = TN / (TN + FN) Sensitivity = TP / (TP + FN) Specificity = TN / (FP + TN) New Method Results Comparability RED CELL FRAGMENT FLAG: XE 26 XT 17 LH750 28 MANUAL MORPHOLOGY POSITIVE RED CELL FRAGMENTS = 15 MANUAL MORPHOLOGY NEGATIVE RED CELL FRAGMENTS = 351 XE true positive = 8 XE false negative = 7 XE false positive = 18 XE true negative = 333 XT true positive = 7 XT false negative = 8 XT false positive = 10 XT true negative = 341 LH750 true positive = 9 LH750 false negative = 6 LH750 false positive = 19 LH750 true negative = 332 Sensitivity – XE 53% XT 47% LH750 40% Specificity - XE 98% XT 98% LH750 98% Predictive value of a positive result – XE 31% XT 41% LH750 32% Predictive value of a negative result – XE 98% XT 98% LH750 98% Overall efficiency – XE 93% XT 95% LH750 93% Comparability • Some parameters are only available on a single or limited number of instrument types - % Hypochromic red cells - Reticulocyte haemoglobin content - Immature platelet fraction • Results should be assessed to determine if they are consistent and appropriate with the diagnosis and clinical condition of the patient. • Where routine haematology analysers use monoclonal antibodies and flow cytometry methods for the measurement of some cells, (e.g. platelets labelled with anti-CD61 and lymphocyte subsets with anti-CD4 and antiCD8) the results should be compared to the results from a dedicated flow cytometer Local evaluation/validation WHO defines validation as the action (or process) of proving that a procedure, process, system, equipment, or method used works as expected and achieves the intended result (WHO-BS/95.1793). CPA definition • Validation - confirmation, through the provision of objective evidence, that the requirements for a specific intended use or application have been fulfilled CPA Standards • Examination procedures, including those for sampling, shall meet the needs and requirements of users. • Examination procedures shall be validated for their intended use prior to introduction, and the methods used and results obtained, recorded. • The laboratory shall determine the uncertainty of results, where relevant and possible. Why validate an analyser? • It is a CPA requirement • CE mark not sufficient • Manufacturer claims may be overestimated Local evaluation/validation • Price: Instrument, reagents, consumables, leasing… • Volume of sample needed, automatic and manual mode, types of sample tubes accepted • The units used for reporting results. • Service contract and response times • Interface • Installation and calibration • Training Local evaluation/validation: Blood samples • The new instrument will be compared to the current instrument in the lab and the manual differential • Samples should be carefully selected and follow the collection, processing, transportation, and storage guidelines established • The anticoagulant used should be recorded (K2EDTA or K3EDTA) • The range of samples tested should cover the entire clinical range and include the most severe abnormalities encountered by the laboratory. • Samples should be included with possible interfering substances, such as lipid, high bilirubin concentration, haemolysis or the presence of cryoglobulins. • One third to half of samples should be normal • A minimum of 50 samples are required to test for statistical significance Local evaluation/validation: Analysis of results • • • • • Samples with extreme results should not be included as this influences statistical analysis. Linear regression Bland Altman The paired t-test (when results are normally distributed) For non-Gaussian data the Wilcoxon Rank Sum test or MannWhitney U test should be used Sensitivity and specificity of abnormal cell flags 550 500 450 400 350 300 250 200 150 100 50 0 y = 0.8502x + 2.4549 R2 = 0.9861 0 50 100 150 200 250 300 350 400 450 500 550 FLOW NRBC/I00 WBC FLOW V MANUAL NRBC/100 WBC 175 MAN NRBC/100 WBC • MAN NRBC/100 WBC FLOW V MANUAL NRBC/100 WBC 150 125 100 75 50 25 0 0 25 50 75 100 125 FLOW NRBC/I00 WBC 150 175 Local evaluation/validation: • Between batch precision. Reference Intervals • Reference intervals, specific to the instrument, for all components of the FBC should be calculated during the instrument evaluation. • At least 30 apparently healthy individuals of each sex should ideally be tested, if appropriate, ranges for children of different ages should also be tested. • Samples should be tested within four hours of venesection. Reference Intervals Reference Intervals Reference Intervals • Hb to reported as g/l • All reference ranges to be the same for all labs and across all instrument types • MCHC to be a non-reportable parameter Reference Intervals Instrument ABX Instruments ABX Pentra series Siemens Advia Cell Dyn 1600 & 1700 Cell Dyn 3000 series Cell Dyn 3200 & Ruby Cell Dyn 4000 & Sapphire Coulter S Plus series Coulter T series Sysmex K series Sysmex X-Class Sysmex SF3000 series Sysmex pocH-100i Sysmex XT series 0902FB1 88.90 84.90 88.96 95.26 98.41 85.66 90.40 94.82 94.01 93.47 102.82 94.37 97.06 103.16 0902FB2 85.61 80.23 86.94 91.52 94.98 84.07 88.13 90.51 90.05 90.95 99.78 92.02 93.65 99.14 Reference ranges • Where ever possible local reference ranges should be established for any new method • Fresh or frozen plasmas from at least 30 healthy individuals should be tested • Sex specific ranges may be required (e.g. protein S assays) Efficiency Throughput Real situation Sick patients Wide range of tests performed simultaneously Reflex tests Can several test methodologies run at once? • Efficiency • • • • Start up and shutdown Sample identification Presentation and storage of results Reliability Acceptability • Staff opinions and preferences should be taken into consideration. • An assessment of the level of expertise required for the operation of the instrument should be determined. • The impact of the instrument on the workflow and organisation of the laboratory. • Any modifications to the laboratory design should be considered. Performance assessment Good precision and accuracy Good precision, poor accuracy Reasonable accuracy, reasonable precision Poor accuracy, poor precision
© Copyright 2024