Editor’s Note: This online data supplement contains supplemental material that was not included with the published article by Caroline Lubick Goldzweig and colleagues, “Costs and Benefits of Health Information Technology: New Trends from the Literature,” Health Affairs 28, no. 2 (2009): w282-w293 (published online 27 January 2009; 10.1377/hlthaff.28.2.w282) Figure 1. Article Flow 4,683 Titles Ide ntifie d for Titl e R eview 4.035 R ejected on Title R eview 648 Titles C onsi der ed R elevant 400 R ejected on A bstract R eview 248 A r ticles R equested 5 R ejected: 5 N ot Found 243 A r ticles S cr eened 64 15 4 15 5 23 2 R ejected: Topic (N ot H IT) D esign – O ther P urpose D escriptive Q ualitative N on-systematic review D escriptive D esign/N o B arriers or Facilitators R obotics article 179 A r ticles Q uality R eview ed and i nclude d in t he H IT int er active data base 182 studies foun d References Asaro PV, Sheldahl AL, Char DM. “Embedded guideline information without patient specificity in a commercial emergency department computerized orderentry system.” Academic Emergency Medicine 2006; 13: 452-8. Boissy P, Jacobs K, Roy SH. “Usability of a barcode scanning system as a means of data entry on a PDA for self-report health outcome questionnaires: a pilot study in individuals over 60 years of age.” BioMed Central Medical Informatics and Decision Making 2006; 6: 42. Boldrick JC, Layton CJ, Nguyen J, Swetter SM. “Evaluation of digital dermoscopy in a pigmented lesion clinic: clinician versus computer assessment of malignancy risk.” Journal of the American Academy of Dermatology 2007; 56: 417-21. Butler J, Speroff T, Arbogast PG, Newton M, Waitman LR, Stiles R, Miller RA, Ray W, Griffin MR. “Improved compliance with quality measures at hospital discharge with a computerized physician order entry system.” American Heart Journal 2006; 151: 643-53. Cavanagh K, Shapiro DA, Van Den Berg S, Swain S, Barkham M, Proudfoot J. “The effectiveness of computerized cognitive behavioural therapy in routine care.” British Journal of Clinical Psychology 2006; 45: 499514. Chaudhry B, Wang J, Wu S, Maglione M, Mojica W, Roth E, Morton SC, Shekelle PG. “Systematic review: impact of health information technology on quality, efficiency, and costs of medical care.” Annals of Internal Medicine 2006; 144: 742-52. Cintron A, Phillips R, Hamel MB. “The effect of a webbased, patient-directed intervention on knowledge, discussion, and completion of a health care proxy.” Journal of Palliative Medicine 2006; 9: 1320-8. Cunningham JA, Humphreys K, Kypri K, van Mierlo T. “Formative evaluation and three-month follow-up of an online personalized assessment feedback intervention for problem drinkers.” Journal of Medical Internet Research 2006; 8: e5. Cutler DM, Feldman NE, Horwitz JR. “U.S. adoption of computerized physician order entry systems.” Health Affairs (Millwood) 2005; 24: 1654-63. Del Beccaro MA, Jeffries HE, Eisenberg MA, Harry ED. “Computerized provider order entry implementation: no association with increased mortality rates in an intensive care unit.” Pediatrics 2006; 118: 290-5. Dexter PR, Perkins SM, Maharry KS, Jones K, McDonald CJ. “Inpatient computer-based standing orders vs physician reminders to increase influenza and pneumococcal vaccination rates: a randomized trial.” Journal of the American Medical Association 2004; 292: 2366-71. Downer SR, Meara JG, DaCosta AC. Use of SMS text messaging to improve outpatient attendance. Med J Aust 2005;183(7):366-8 Downer SR, Meara JG, Da Costa AC, Sethuraman K. “SMS text messaging improves outpatient attendance.” Australian Health Review 2006; 30: 389-96. Feldstein A, Elmer PJ, Smith DH, Herson M, Orwoll E, Chen C, Aickin M, Swain MC. “Electronic medical record reminder improves osteoporosis management after a fracture: a randomized, controlled trial.” Journal of the American Geriatrics Society 2006; 54: 450-7. Fung V, Ortiz E, Huang J, Fireman B, Miller R, Selby JV, Hsu J. “Early experiences with e-health services (1999-2002): promise, reality, and implications.” Medical Care 2006; 44: 491-6. Galanter WL, Polikaitis A, DiDomenico RJ. A trial of automated safety alerts for inpatient digoxin use with computerized physician order entry. Journal of the American Medical Informatics Association 2004; 11: 270-7. Garrido T, Jamieson L, Zhou Y, Wiesenthal A, Liang L. “Effect of electronic health records in ambulatory care: retrospective, serial, cross sectional study.” British Medical Journal 2005; 330: 581. Glanz K, Murphy S, Moylan J, Evensen D, Curb JD. “Improving dietary self-monitoring and adherence with hand-held computers: a pilot study.” American Journal of Health Promotion 2006; 20: 165-70. Glasgow RE, Nutting PA, King DK, Nelson CC, Cutter G, Gaglio B, Rahm AK, Whitesides H. “Randomized effectiveness trial of a computer-assisted intervention to improve diabetes care.” Diabetes Care 2005; 28: 33-9. Glasgow RE, Strycker LA, King DK, Toobert DJ, Rahm AK, Jex M, Nutting PA. “Robustness of a computer-assisted diabetes self-management intervention across patient characteristics, healthcare settings, and intervention staff.” American Journal of Managed Care 2006; 12: 137-45. Green MJ, Peterson SK, Baker MW, Harper GR, Friedman LC, Rubinstein WS, Mauger DT. “Effect of a computer-based decision aid on knowledge, perceptions, and intentions about genetic testing for breast cancer susceptibility: a randomized controlled trial.” Journal of the American Medical Association 2004; 292: 442-52. Grime PR. “Computerized cognitive behavioural therapy at work: a randomized controlled trial in employees with recent stress-related absenteeism.” Occupational Medicine 2004; 54: 353-9. Gustafson DH, McTavish FM, Stengle W, Ballard D, Hawkins R, Shaw BR, Jones E, Julesberg K, McDowell H, Chen WC, Volrathongchai K, Landucci G. “Use and Impact of eHealth System by Low-income Women With Breast Cancer.” Journal of Health Communication 2005; 10 Suppl 1: 195-218. Han YY, Carcillo JA, Venkataraman ST, Clark RS, Watson RS, Nguyen TC, Bayir H, Orr RA. “Unexpected increased mortality after implementation of a commercially sold computerized physician order entry system.” Pediatrics 2005; 116: 1506-12. Harno K, Kauppinen-Makelin R, Syrjalainen J. “Managing diabetes care using an integrated regional e-health approach.” Journal of Telemedicine and Telecare 2006; 12 Suppl 1: 13-5. Holden MK, Dyar TA, Dayan-Cimadoro L. “Telerehabilitation using a virtual environment improves upper extremity function in patients with stroke.” IEEE Transactions on Neural Systems and Rehabilitation Engineering 2007; 15: 36-42. Jacobi C, Morris L, Beckers C, Bronisch-Holtze Winzelberg AJ, Taylor CB. “Maintenance of based prevention: a randomized controlled International Journal of Eating Disorders 114-9. J, Winter J, internettrial.” 2007; 40: Jha AK, Ferris TG, Donelan K, DesRoches C, Shields A, Rosenbaum S, Blumenthal D. “How common are electronic health records in the United States? A summary of the evidence.” Health Affairs (Millwood) 2006; 25: w496507. Kaner E, Heaven B, Rapley T, Murtagh M, Graham R, Thomson R, May C. “Medical communication and technology: a video-based process study of the use of decision aids in primary care consultations.” BioMed Central Medical Informatics and Decision Making 2007; 7: 2. Kemper AR, Uren RL, Clark SJ. “Adoption of electronic health records in primary care pediatric practices.” Pediatrics 2006; 118: e20-4. Lester WT, Grant RW, Barnett GO, Chueh HC. “Randomized controlled trial of an informatics-based intervention to increase statin prescription for secondary prevention of coronary disease.” Journal of General Internal Medicine 2006; 21: 22-9. Lindenauer PK, Ling D, Pekow PS, Crawford A, NaglieriPrescod D, Hoople N, Fitzgerald J, Benjamin EM. “Physician characteristics, attitudes, and use of computerized order entry.” Journal of Hospital Medicine 2006; 1: 221-30. Lium JT, Laerum H, Schulz T, Faxvaag A. “From the front line, report from a near paperless hospital: mixed reception among health care professionals.” Journal of the American Medical Informatics Association 2006; 13: 668-75. Mattocks K, Lalime K, Tate JP, Giannotti TE, Carr K, Carrabba A, Blum T, Meehan TP. “The state of physician office-based health information technology in Connecticut: current use, barriers and future plans.” Connecticut Medicine 2007; 71: 27-31. McInnes DK, Saltman DC, Kidd MR. “General practitioners' use of computers for prescribing and electronic health records: results from a national survey.” The Medical Journal of Australia 2006; 185: 88-91. McMahon GT, Gomes HE, Hickson Hohne S, Hu TM, Levine BA, Conlin PR. “Web-based care management in patients with poorly controlled diabetes.” Diabetes Care 2005; 28: 1624-9. Menachemi N. “Barriers to ambulatory EHR: who are 'imminent adopters' and how do they differ from other physicians?” Informatics in Primary Care 2006; 14: 101-8. Murray MD, Harris LE, Overhage JM, Zhou XH, Eckert GJ, Smith FE, Buchanan NN, Wolinsky FD, McDonald CJ, Tierney WM. “Failure of computerized treatment suggestions to improve health outcomes of outpatients with uncomplicated hypertension: results of a randomized controlled trial.” Pharmacotherapy 2004; 24: 324-37. O'Neill L, Klepack W. “Electronic medical records for a rural family practice: a case study in systems development.” Journal of Medical Systems 2007; 31: 2533. Ovretveit J, Scott T, Rundall TG, Shortell SM, Brommels M. “Improving quality through effective implementation of information technology in healthcare.” International Journal for Quality in Health Care 2007; 19: 259-66. Ozanne EM, Annis C, Adduci K, Showstack J, Esserman L. “Pilot trial of a computerized decision aid for breast cancer prevention.” Breast Journal 2007; 13: 147-54. Ozdas A, Speroff T, Waitman LR, Ozbolt J, Butler J, Miller RA. “Integrating ‘best of care’ protocols into clinicians' workflow via care provider order entry: impact on quality-of-care indicators for acute myocardial infarction.” Journal of the American Medical Informatics Association 2006; 13: 188-96. Palen TE, Raebel M, Lyons E, Magid DM. “Evaluation of laboratory monitoring alerts within a computerized physician order entry system for medication orders.” American Journal of Managed Care 2006; 12: 389-95. Pizzi LT, Suh DC, Barone J, Nash DB. “Factors related to physicians' adoption of electronic prescribing: results from a national survey.” American Journal of Medical Quality 2005; 20: 22-32. Potts AL, Barr FE, Gregory DF, Wright L, Patel NR. “Computerized physician order entry and medication errors in a pediatric critical care unit.” Pediatrics 2004; 113: 59-63. Pretty IA, Ellwood RP. “Comparison of paired visual assessment and software analyses of changes in caries status over 6 months from fluorescence images.” Caries Research 2007; 41: 115-20. Proudfoot J, Ryden C, Everitt B, Shapiro DA, Goldberg D, Mann A, Tylee A, Marks I, Gray JA. “Clinical efficacy of computerised cognitive-behavioural therapy for anxiety and depression in primary care: randomised controlled trial.” The British Journal of Psychiatry 2004; 185: 46-54. Robertson L, Smith M, Castle D, Tannenbaum D. “Using the Internet to enhance the treatment of depression.” Australasian Psychiatry 2006; 14: 413-7. Rothschild JM, McGurk S, Honour M, Lu L, McClendon AA, Srivastava P, Churchill WH, Kaufman RM, Avorn J, Cook EF, Bates DW. “Assessment of education and computerized decision support interventions for improving transfusion practice.” Transfusion 2007; 47: 228-39. Roumie CL, Elasy TA, Greevy R, Griffin MR, Liu X, Stone WJ, Wallston KA, Dittus RS, Alvarez V, Cobb J, Speroff T. “Improving blood pressure control through provider education, provider alerts, and patient education: a cluster randomized trial.” Annals of Internal Medicine 2006; 145: 165-75. Schinke S, Di Noia J, Schwinn T, Cole K. “Drug abuse risk and protective factors among black urban adolescent girls: a group-randomized trial of computer-delivered mother-daughter intervention.” Psychology of Addictive Behaviors 2006; 20: 496-500. Schmittdiel J, Bodenheimer T, Solomon NA, Gillies RR, Shortell SM. “Brief report: The prevalence and use of chronic disease registries in physician organizations. A national survey.” Journal of General Internal Medicine 2005; 20: 855-8. Sequist TD, Gandhi TK, Karson AS, Fiskio JM, Bugbee D, Sperling M, Cook EF, Orav EJ, Fairchild DG, Bates DW. “A randomized trial of electronic clinical reminders to improve quality of care for diabetes and coronary artery disease.” Journal of the American Medical Informatics Association 2005; 12: 431-7. Sheridan SL, Shadle J, Simpson RJ Jr, Pignone MP. “The impact of a decision aid about heart disease prevention on patients' discussions with their doctor and their plans for prevention: a pilot randomized trial.” BioMed Central Health Services Research 2006; 6: 121. Smith DH, Perrin N, Feldstein A, Yang X, Kuang D, Simon SR, Sittig DF, Platt R, Soumerai SB. “The impact of prescribing safety alerts for elderly persons in an electronic medical record: an interrupted time series evaluation.” Archives of Internal Medicine 2006; 166: 1098-104. Steele AW, Eisert S, Witter J, Lyons P, Jones MA, Gabow P, Ortiz E. “The effect of automated alerts on provider ordering behavior in an outpatient setting.” PLoS Medicine 2005; 2: e255. Stevenson KB, Barbera J, Moore JW, Samore MH, Houck P. “Understanding keys to successful implementation of electronic decision support in rural hospitals: analysis of a pilot study for antimicrobial prescribing.” American Journal of Medical Quality 2005; 20: 313-8. Tideman RL, Chen MY, Pitts MK, Ginige S, Slaney M, Fairley CK. “A randomised controlled trial comparing computerassisted with face-to-face sexual history taking in a clinical setting.” Sexually Transmitted Infections 2007; 83: 52-6. Tierney WM, Overhage JM, Murray MD, Harris LE, Zhou XH, Eckert GJ, Smith FE, Nienaber N, McDonald CJ, Wolinsky FD. “Can computer-generated evidence-based care suggestions enhance evidence-based management of asthma and chronic obstructive pulmonary disease? A randomized, controlled trial.” Health Services Research 2005; 40: 477-97. Toth-Pal E, Nilsson GH, Furhoff AK. “Clinical effect of computer generated physician reminders in health screening in primary health care--a controlled clinical trial of preventive services among the elderly.” International Journal of Medical Informatics 2004; 73: 695-703. Vilella A, Bayas JM, Diaz MT, Guinovart C, Diez C, Simo D, Munoz A, Cerezo J. “The role of mobile phones in improving vaccination rates in travelers.” Preventive Medicine 2004; 38: 503-9. Wagner B, Knaevelsrud C, Maercker A. “Internet-based cognitive-behavioral therapy for complicated grief: a randomized controlled trial.” Death Studies 2006; 30: 429-53. Walshe K. “Understanding what works – and why – in quality improvement: the need for theory-driven evaluation.” International Journal for Quality in Health Care 2007; 19: 57-59. Table 1. Evidence Table for studies from the HIT leaders HIT Element(s) Healthcare Setting Butler J, Speroff TArbogast PG, Newton M, Waitman LR, Stiles R, Miller RA, Ray W, Griffin MR. “Improved compliance with quality measures at hospital discharge with a computerized physician order entry system.” American Heart Journal 2006; 151: 643-53. Study Purpose To assess the effect of CPOE discharge tools on quality measures for acute myocardial infarction (AMI) and congestive heart failure (CHF) CPOE, clinical reminders during patient encounter, electronic prescribing Vanderbilt University Medical Center, Nashville, TN Dexter PR, Perkins SM, Maharry KS, Jones K, McDonald CJ. “Inpatient computer-based standing orders vs physician reminders to increase influenza and pneumococcal vaccination rates: a randomized trial.” Journal of the American Medical Association 2004; 292: 2366-71. To evaluate the effect of computerized standing nurse orders for influenza and pneumococcal vaccines as compared to physician computerized reminders CPOE, EHR, clinical reminders during patient encounter University hospital in Indianapolis, including all patients discharged from general medicine wards during the study periods who were eligible for the vaccines Author Implementation Factors Evaluated CPOE associated reminders were revised up to weekly based on provider feedback (Implementation strategy) Potential barriers to adoption included confusion between providers and coders on criteria used to assign primary discharge diagnoses and a lack of . regular feedback on results of intervention Program sustained indefinitely at study institution for all inpatient wards other than intensive care Source: Authors’ analysis of results from 2007 Systematic Review of the HIT Literature Findings/Outcomes Smoking cessation counseling increased from 1% to 43% of smokers with CHF and from 21% to 62% of smokers with AMI. Discharge instructions increased from 3% to 56% of CHF patients. The proportion of patients who received discharge prescriptions did not change. Nurse electronic standing orders resulted in greater rates of vaccination compared to electronic physician reminders (for influenza 42% vs. 30%; for pneumococcal vaccination 51% vs. 31%) 1 Author HIT Element(s) Healthcare Setting Lester WT, Grant RW, Barnett GO, Chueh HC. “Randomized controlled trial of an informaticsbased intervention to increase statin prescription for secondary prevention of coronary disease.” Journal of General Internal Medicine 2006; 21: 22-9. Study Purpose To evaluate whether a disease management system outside a clinical encounter would reduce delays in appropriate medication adjustment EHR, electronic prescribing Academic primary care adult practices that relied on an electronic health record for the majority of care Murray MD Harris LE, Overhage JM, Zhou XH, Eckert GJ, Smith FE, Buchanan NN, Wolinsky FD, McDonald CJ, Tierney WM. “Failure of computerized treatment suggestions to improve health outcomes of outpatients with uncomplicated hypertension: results of a randomized controlled trial.” Pharmacotherapy 2004; 24: 324-37. To assess physician and pharmacist compliance with electronic hypertension treatment suggestions and the effect on patient health-related quality of life, and clinical and utilization measures. CPOE, medical prescribing support Large, inner-city academic internal medicine practice affiliated with the Indiana University School of Medicine Implementation Factors Evaluated Initial differences in statin prescription changes were not sustained at one year Findings/Outcomes Implementation strategy included dissemination of locally approved guidelines for managing hypertension; presentations at medical grand rounds; and one-onone meeting with each intervention pharmacist and physician regarding the intervention Compliance with treatment suggestions did NOT differ significantly among groups nor were there differences in quality of life, blood pressure measurements or emergency room or hospital visits. Compared with controls, more intervention patients had statin prescription changes at 1 month postrandomization (15.3% vs 2%, p=0.001). Providers could bypass suggestions by using the “escape” key (barrier) Source: Authors’ analysis of results from 2007 Systematic Review of the HIT Literature 2 Author Ozdas A, Speroff T, Waitman LR, Ozbolt J, Butler J, Miller RA. “Integrating ‘best of care’ protocols into clinicians' workflow via care provider order entry: impact on quality-of-care indicators for acute myocardial infarction.” Journal of the American Medical Informatics Association 2006; 13: 188-96. Potts AL, Barr FE, Gregory DF, Wright L, Patel NR. “Computerized physician order entry and medication errors in a pediatric critical care unit.” Pediatrics 2004; 113: 5963. Study Purpose To evaluate the impact of a decision support tool on a) the integration of cardiology order sets and b) resulting medication orders HIT Element(s) Healthcare Setting CPOE, clinical reminders during patient encounter, clinical guidelines/protoc ols Vanderbilt University Hospital, a 630-bed academic tertiary care hospital To evaluate the impact of CPOE on the frequency of errors in the medication ordering process in a pediatric critical care unit CPOE All patients admitted to a 20-bed multidisciplinary pediatric ICU at an academic institution in a large city Implementation Factors Evaluated Relevant order sets were reviewed and revised by cardiology (implementation strategy) System penetration was complete for non-ER sites but system could not be used in the ER, where many cardiac patients first present, because CPOE was not in use (barrier) None reported Source: Authors’ analysis of results from 2007 Systematic Review of the HIT Literature Findings/Outcomes ACS order set use increased from 60% in the preintervention period to 70% after the program was installed (p=0.009). For all suspected acute myocardial infarction (ACI) admissions, CS order set use led to a significant increase in early aspirin ordering (p=0.001) and a trend toward increase in beta-blocker ordering (p=0.07). Rates of potential adverse drug events (ADEs), medication prescribing errors (MPs) and rule violations (RVs) all decreased significantly after implementation of the CPOE system (40.9% reduction in potential ADEs, 99.4% reduction in MPEs and 97.9% reduction in RVs, p< 0.001 for all comparisons) 3 Author Rothschild JM, McGurk S, Honour M, Lu L, McClendon AA, Srivastava P, Churchill WH, Kaufman RM, Avorn J, Cook EF, Bates DW. “Assessment of education and computerized decision support interventions for improving transfusion practice.” Transfusion 2007; 47: 228-39. Roumie CL, Elasy TA, Greevy R, Griffin MR, Liu X, Stone WJ, Wallston KA, Dittus RS, Alvarez V, Cobb J, Speroff T. “Improving blood pressure control through provider education, provider alerts, and patient education: a cluster randomized trial.” Annals of Internal Medicine 2006; 145: 16575 Study Purpose RCT of decision support intervention with computerized physician order entry (CPOE) for red blood cell, platelet, and freshfrozen plasma orders. HIT Element(s) Healthcare Setting Decision support system Academic medical center with an established CPOE system. To compare the effect of different elements of a multifactorial intervention on quality of care for hypertensive patients EHR, clinical reminders during patient encounter, clinical guidelines/protoc ols Implementation Factors Evaluated System penetration limited to areas with established CPOE Facilitated by guideline training Potential barrier related to physician concerns that the goal of the guidelines was cost and usage reduction Patients with hypertension and their providers associated with the VA Tennessee Valley Healthcare System None reported Source: Authors’ analysis of results from 2007 Systematic Review of the HIT Literature Findings/Outcomes Rates of inappropriate transfusion orders for all staff decreased after the initial education (from 72.6% to 63.3%, p<0.0001) and continued to decrease in the decision support group (59.6% p<0.0001) while slipping back to baseline in the control group (67.4%). The proportion of patients achieving a systolic blood pressure goal of <140 was greatest when patients received an educational letter in addition to their providers receiving education and electronic alerts. 4 Author Sequist Sequist TD, Gandhi TK, Karson AS, Fiskio JM, Bugbee D, Sperling M, Cook EF, Orav EJ, Fairchild DG, Bates DW. “A randomized trial of electronic clinical reminders to improve quality of care for diabetes and coronary artery disease.” Journal of the American Medical Informatics Association 2005; 12: 431-7. Tierney WM, Overhage JM, Murray MD, Harris LE, Zhou XH, Eckert GJ, Smith FE, Nienaber N, McDonald CJ, Wolinsky FD. “Can computer-generated evidence-based care suggestions enhance evidence-based management of asthma and chronic obstructive pulmonary disease? A randomized, controlled trial.” Health Services Research 2005; 40: 47797. Study Purpose To assess the effect of computerized clinical reminders on processes of care for diabetes and coronary artery disease HIT Element(s) Healthcare Setting Clinical reminders during patient encounter Partners HealthCare System, 20 outpatient clinics affiliated with Brigham and Women’s Hospital or Massachusetts General Hospital To determine whether guidelinebased suggestions presented to physicians and pharmacists improves the care of asthma and COPD CPOE, EHR, medication prescribing support, clinical reminders during patient encounter, clinical guidelines/protoc ols 4 hospital-based primary care practices (Indiana University Medical Group-Primary Care) with associated outpatient pharmacies. Patients were >18 with diagnosis of asthma or COPD based on diagnosis, medication or xray result Implementation Factors Evaluated System penetration was universal for all sites Barriers most commonly identified were lack of time, patient noncompliance and lack of knowledge Findings/Outcomes Intervention physicians required to view all care suggestions and both intervention and control physicians required to enter all orders electronically (facilitators) There were no differences between intervention and control groups in adherence to care suggestions, impact on quality of life, medication adherence or patient satisfaction Reminders increased the use of recommended processes for diabetes by an average of 5% (from 14% to 19%) compared to control, ; and for coronary artery disease by 5% (from 17% to 22%) Physicians were skeptical of benefits and intentions of guidelines (barrier) Source: Authors’ analysis of results from 2007 Systematic Review of the HIT Literature 5 Table 2. Evidence Table of Commercial HIT Applications Author Study Purpose HIT Element(s) Healthcare Setting Asaro PV, Sheldahl AL, Char DM. “Embedded guideline information without patient specificity in a commercial emergency department computerized orderentry system.” Academic Emergency Medicine 2006; 13: 452-8 To compare acute coronary syndrome (ACS) guideline compliance before and after both paper and then electronic implementation CPOE Emergency medicine division at Washington University hospital Del Beccaro MA, Jeffries HE, Eisenberg MA, Harry ED. “Computerized provider order entry implementation: no association with increased mortality rates in an intensive care unit.” Pediatrics 2006; 118: 290-5. To determine whether there were changes in risk-adjusted mortality in a pediatric ICU after implementation of a CPOE system CPOE, results reporting / viewing systems Pediatric ICU of a tertiary referral center in the Northwest US Implementation Factors Evaluated A paper-based guideline with order sets for ED patients with ACS was developed by a multidisciplinary team over a six-month period. The guideline was initially introduced as a set of four preprinted paper forms which were later adapted for inclusion as CPOE order sets. (implementation strategy) Clinical leader identified from each clinical division to design order sets; all clinical staff required to attend role-specific training; and, each discipline had "superusers"(implementati on strategy) Findings/Outcomes In the last phase of the study, betablocker use was significantly associated with use of the guidelines. There was no association between aspirin or heparin use and use of the guidelines. Despite electronic implementation of guidelines, there was no improvement in compliance with any of the guideline recommendations. There was a nonsignificant reduction in the risk of mortality in the 13 months before and after implementation (4.22% v 3.36%; RR 0.82, 95% CI 0.55 – 1.21) Go-live support provided 24/7 for 2 weeks, after which providers could not “opt out” (facilitators). Source: Authors’ analysis of results from 2007 Systematic Review of the HIT Literature 6 Author Study Purpose HIT Element(s) Healthcare Setting Feldstein A, Elmer PJ, Smith DH, Herson M, Orwoll E, Chen C, Aickin M, Swain MC. “Electronic medical record reminder improves osteoporosis management after a fracture: a randomized, controlled trial.” Journal of the American Geriatrics Society 2006; 54: 450-7. To evaluate whether an EHR reminder to the primary care provider, with or without a patient reminder, increases the proportion of women who received bone mineral density (BMD) measurement or a medication for osteoporosis after a fracture. EHR, clinical guidelines/proto cols Nonprofit group model HMO in he Pacific Northwest Implementation Factors Evaluated Reminder was signed by (the chair of the osteoporosis quality improvement committee and patient-specific advice, as opposed to general advice, was provided. (facilitators) Source: Authors’ analysis of results from 2007 Systematic Review of the HIT Literature Findings/Outcomes 6 months after initiation of provider and patient reminders, rates of BMD measurement or receipt of osteoporosis medications were 51.5% in the provider reminder group, 43.1% in the patient reminder group, and 5.9% in the usual care group Total calcium intake increased significantly in both intervention groups, compared to usual care. 7 Author Study Purpose HIT Element(s) Healthcare Setting Galanter WL, Polikaitis A, DiDomenico RJ. A trial of automated safety alerts for inpatient digoxin use with computerized physician order entry. Journal of the American Medical Informatics Association 2004; 11: 270-7. To compare the time for physician action regarding digoxin use before and after implementation of synchronous alerts (occurring at the time of ordering) and asynchronous alerts (presented as print-outs for responsible providers) CPOE, Clinical reminders during patient encounter University of Illinois Hospital and Medical Center Implementation Factors Evaluated None reported Source: Authors’ analysis of results from 2007 Systematic Review of the HIT Literature Findings/Outcomes There was a significant improvement in time to ordering appropriate lab tests after implementation of the DSS. Compliance with supplementation of untreated hypokalemia and hypomagnesemia was improved with the asynchronous alerts but not with the synchronous alerts. 8 Author Study Purpose HIT Element(s) Healthcare Setting Garrido T, Jamieson L, Zhou Y, Wiesenthal A, Liang L. “Effect of electronic health records in ambulatory care: retrospective, serial, cross sectional study.” British Medical Journal 2005; 330: 581. To assess the effect of the introduction of a multifunctional EHR on health care utilization and processes of care CPOE, EHR, results reporting/viewin g systems Kaiser Permanente Northwest and Colorado Implementation Factors Evaluated Qualitative interviews indicated that strong leadership support, an organizational structure supporting free flow of information, and efficiency gains were key to successful implementation. Source: Authors’ analysis of results from 2007 Systematic Review of the HIT Literature Findings/Outcomes Four years after implementation in both the Northwest and Colorado, ambulatory care service use declined and laboratory use decreased by 18%,radiology services decreased by 14% in the Northwest. Measures of advice on smoking cessation, cervical cancer screening, or retinal examination in diabetics either were unchanged or improved slightly over time. 9 Author Study Purpose HIT Element(s) Healthcare Setting Han YY, Carcillo JA, Venkataraman ST, Clark RS, Watson RS, Nguyen TC, Bayir H, Orr RA. “Unexpected increased mortality after implementation of a commercially sold computerized physician order entry system.” Pediatrics 2005; 116:150612. To assess the effect of CPOE on outcomes in a pediatric hospital CPOE University-affiliated 235 bed regional pediatric referral center with 12,000 annual admissions Implementation Factors Evaluated Implementation strategy included a mandatory 3 hour computer tutorial and practice session 3 months prior to CPOE implementation for all hospital health care personnel, a.hospital-wide implementation over a 6day period with experts initially available to provide hand-on consultations followed by telephone support Source: Authors’ analysis of results from 2007 Systematic Review of the HIT Literature Findings/Outcomes Risk-adjusted mortality increased form 2.8% (average) in the 4 quarters prior to CPOE implementation to 6.6% (average) in the 2 quarters after implementation. 10 Author Study Purpose HIT Element(s) Healthcare Setting O'Neill L, Klepack W. “Electronic medical records for a rural family practice: a case study in systems development.” Journal of Medical Systems 2007; 31: 25-33. Describe the implementation and impact of an EHR system. EHR Rural family practice. Implementation Factors Evaluated There were 3 phases over 2+ years: a) building EHRs and learning the system, b) connecting with clinical partners, and c) quality management / disease management (implementation strategy) Findings/Outcomes Case mix increased 10% in 2 years Patient volume per physician did not change, so time spent with each patient did not decrease. All staff in this small clinic (4 MDs) used system for ALL visits (system penetration) Office manager took lead in educating the other staff. Used vendor who had previously supplied billing and scheduling systems.(facilitators) Older physicians were less familiar with computers. (barriers) Source: Authors’ analysis of results from 2007 Systematic Review of the HIT Literature 11 Author Study Purpose HIT Element(s) Healthcare Setting Palen TE, Raebel M, Lyons E, Magid DM. “Evaluation of laboratory monitoring alerts within a computerized physician order entry system for medication orders.” American Journal of Managed Care 2006; 12: 389-95. To evaluate whether reminders during CPOE for medications would increase compliance with guidelines for laboratory mo nitoring at initiation of therapy CPOE, EHR, medication prescribing support Physicians from 16 ambulatory practice sites of a groupmodel managed care organization Smith DH, Perrin N, Feldstein A, Yang X, Kuang D, Simon SR, Sittig DF, Platt R, Soumerai SB. “The impact of prescribing safety alerts for elderly persons in an electronic medical record: an interrupted time series evaluation.” Archives of Internal Medicine 2006; 166: 1098-104. To examine the effects of CPOE with decision support on reducing the use of potentially contraindicated agents in elderly patients CPOE, medication prescribing support Primary care providers in a group model HMO in the Pacific Northwest Implementation Factors Evaluated Implementation strategy included one-on-one detailing of each intervention clinician to train in where to find the alerts and how to use the information to alter dosing or order labs at the time of medication ordering Clinicians were prompted for the preferred medication and choosing this medication resulted in an automatically populated order ready for signature (facilitator) Decreases in use of nonpreferred agents was sustained over the 2-year post-alert period Source: Authors’ analysis of results from 2007 Systematic Review of the HIT Literature Findings/Outcomes No significant difference found between control and intervention physicians in their overall rate of compliance with ordering recommended laboratory monitoring. A sudden decrease in rates of initial dispensing of nonpreferred agents among elderly patients was observed when the alerts were initiated, from 21.9 to 16.8 per 10,000 (p<0.01) 12 Author Study Purpose HIT Element(s) Healthcare Setting Steele AW, Eisert S, Witter J, Lyons P, Jones MA, Gabow P, Ortiz E. “The effect of automated alerts on provider ordering behavior in an outpatient setting.” PLoS Medicine 2005; 2: e255. To determine the impact of using computerized alerts to improve the prescribing of medications medication prescribing support Large public outpatient clinic in Denver, Colorado Implementation Factors Evaluated No specific provider education was given to staff (implementation strategy) 82% of clinic clients are Hispanic, 42% had Medicaid, 41% were uninsured, 17% had Medicare, or private insurance. (financial context) The content and layout of the specific alert screens were design by IT staff with input from the clinic providers.(facilitators) Source: Authors’ analysis of results from 2007 Systematic Review of the HIT Literature Findings/Outcomes When the alert was for an abnormal lab value, the % of times the medication order was discontinued increased from 5.6% at baseline to 10.9% during the intervention (p=.03). When an alert was triggered for a missing lab test, the % of times the provider ordered the test increased from 43.0% at baseline to 62.0% (p<.001). 13 Author Study Purpose HIT Element(s) Healthcare Setting Toth-Pal E, Nilsson GH, Furhoff AK. “Clinical effect of computer generated physician reminders in health screening in primary health care--a controlled clinical trial of preventive services among the elderly.” International Journal of Medical Informatics 2004; 73: 695-703. To evaluate the effects of a program for computer generated physician reminders, integrated with an EHR system, for health screening in elderly patients. Clinical reminders during patient encounter Four primary health care centers in suburban Sweden Implementation Factors Evaluated None reported Source: Authors’ analysis of results from 2007 Systematic Review of the HIT Literature Findings/Outcomes There was a statistically significant moderate or marked increase in the number of patients who were tested in all five intervention areas. A significant increase in pathological test results (1-8%) was found for hypertension and cobalamin deficiency. 14 Table 3. Evidence Table of Stand-Alone HIT Applications Author Study Purpose HIT Element(s) Healthcare Setting Cavanagh K, Shapiro DA, Van Den Berg S, Swain S, Barkham M, Proudfoot J. “The effectiveness of computerized cognitive behavioural therapy in routine care.” British Journal of Clinical Psychology 2006; 45: 499-514. To study effectiveness of "Beating the Blues", a computerized cognitive behavioral therapy (CCBT) program, when implemented in primary and secondary settings with minimal research support To test whether patient-directed electronic messages improve completion of a health care proxy (HCP) in the medical record and knowledge regarding HCP patient decision support/consumer health informatics 8 general practices, 2 community mental health teams and 1 primary care clinical psychology service in England. Patients had anxiety and/or depression and had been identified by a health professional as likely to benefit from CCBT administrative electronic communication Patients of Beth Israel Deaconess Medical Center who subscribed to PatientSite, an online electronic communication system Cintron A, Phillips R, Hamel MB. “The effect of a webbased, patientdirected intervention on knowledge, discussion, and completion of a health care proxy.” Journal of Palliative Medicine 2006; 9: 1320-8 Implementation Factors Evaluated Implementation strategy included orientation for patients to the computer program at initial visit and support as needed at each subsequent visit None reported Source: Authors’ analysis of results from 2007 Systematic Review of the HIT Literature Findings/Outcomes Compared to pretreatment ratings, patients demonstrated statistically significant improvements in the Clinical Outcomes in Routine EvaluationOutcome Measure (CORE-OM), Work and Social Adjustment Scale (WSA) and selfreported anxiety and depression. There were no differences in completion of an HCP for intervention vs control subjects though intervention patients were more likely to report knowledge of HCPs 15 Author Study Purpose HIT Element(s) Healthcare Setting Cunningham JA, Humphreys K, Kypri K, van Mierlo T. “Formative evaluation and three-month follow-up of an online personalized assessment feedback intervention for problem drinkers.” Journal of Medical Internet Research 2006; 8: e5. To assess the effect of a new module assessing problem drinking, when added to an existing webbased online selfhelp service for problem drinking patient decision support / consumer health informatics 348 registered participants of another online selfhelp center, the Stop Smoking Center with 138 accessing the 3month follow-up survey and 81 providing complete data to test effectiveness. Grime PR. “Computerized cognitive behavioural therapy at work: a randomized controlled trial in employees with recent stress-related absenteeism.” Occupational Medicine 2004; 54: 353-9. To evaluate the effect of an 8 week CCBT program "Beating the Blues" on emotional distress in employees with recent stressrelated absenteeism and to explore reasons for nonparticipation Patient decision support / consumer health informatics NHS and local authority employees recruited by a London NHS occupational health department who were eligible if they had ten or more days of sickness absence due to stress, anxiety or depression in the past 6 months and scored 4 or more on the GHQ-12 (General Health Questionnaire) Implementation Factors Evaluated None reported Differences were not statistically significant at 3 and 6 months posttreatment (sustainability) Source: Authors’ analysis of results from 2007 Systematic Review of the HIT Literature Findings/Outcomes Of the participants who provided complete follow-up data (81 out of the original 388), reductions in drinking were reported. At the end of treatment, adjusted mean depression scores were 3.07 lower and adjusted mean negative attributional style scores were 2.32 lower in the intervention group. At one month posttreatment, adjusted mean anxiety, depression and negative attributional style scores were lower in the intervention group. 16 Author Study Purpose HIT Element(s) Healthcare Setting Jacobi C, Morris L, Beckers C, BronischHoltze J, Winter J, Winzelberg AJ, Taylor CB. “Maintenance of internet-based prevention: a randomized controlled trial.” International Journal of Eating Disorders 2007; 40: 114-9. To evaluate 3 month effects of an internet eating disorder prevention program patient decision support/consumer health informatics Internet-based prevention program for eating disorders in young German females McMahon GT, Gomes HE, Hickson Hohne S, Hu TM, Levine BA, Conlin PR. “Web-based care management in patients with poorly controlled diabetes.” Diabetes Care 2005; 28: 1624-9. To assess the effects of webbased care management on glucose and blood pressure control over 12 months in patients with poorly controlled diabetes patient decision support/consumer health informatics Veteran patients with poorly controlled diabetes who had a VA Boston Healthcare System primary care provider and access to a telephone Implementation Factors Evaluated Overall usability was rated as “good” (mean 2.00 with 1.00 being best (facilitator). Study provided computers and internet access to participants (facilitators) Source: Authors’ analysis of results from 2007 Systematic Review of the HIT Literature Findings/Outcomes Intervention produced significant and sustained effects in knowledge test, shape concerns, and eating restraint. Web site use compliance was 72%. Greater decline in A1c over time for the web-based group (1.6 +/- 1.4%) compared to usual care (-1.2 +/- 1.4%). Persistent users had greater declines (-1.9 +/- 1.2%) compared to intermittent users (-1.2 +/- 1.4%) or usual care. Hypertensive patients in the webbased group had greater decline in systolic BP (-10 +/17 mmHg) than usual care (-7 +/- 21 mmHg). 17 Author Study Purpose HIT Element(s) Healthcare Setting Proudfoot J, Ryden C, Everitt B, Shapiro DA, Goldberg D, Mann A, Tylee A, Marks I, Gray JA. “Clinical efficacy of computerised cognitive-behavioural therapy for anxiety and depression in primary care: randomised controlled trial.” The British Journal of Psychiatry 2004; 185: 46-54. To determine the impact of clinical and demographic variables on the efficacy of “Beating the Blues”, a CCBT program for anxiety and depression Patient decision support / consumer health informatics Patients aged 18-75 with depression, mixed anxiety and depression or anxiety disorder not currently receiving psychological treatment recruited from general practices in London and southeast England Implementation Factors Evaluated A nurse checked that the patient had logged on successfully at the beginning of each session and ensured that the patient had the necessary printouts at the end of the session (facilitator) Source: Authors’ analysis of results from 2007 Systematic Review of the HIT Literature Findings/Outcomes The computerized CBT program improved depression, negative attributional style, and work and social adjustment and was not affected by whether patients were prescribed drugs or by length or severity of illness. Anxiety and positive attributional style also improved with CBT and improved more for more disturbed patients. 18 Author Study Purpose HIT Element(s) Healthcare Setting Wagner B, Knaevelsrud C, Maercker A. “Internet-based cognitive-behavioral therapy for complicated grief: a randomized controlled trial.” Death Studies 2006; 30: 429-53. To evaluate an Internet-based cognitive behavioral therapy program for complicated grief electronic communication German-speaking participants from around the world. Implementation Factors Evaluated None reported Source: Authors’ analysis of results from 2007 Systematic Review of the HIT Literature Findings/Outcomes The decrease in intrusion and avoidance symptoms in the treatment group was significantly larger than in the control group. 85% of patients rated the email contact with therapist pleasant (versus unpleasant or don’t know). 82% described the therapeutic contact as personal; 85% had positive attitudes towards internet treatment. 20% missed face-to-face contact with a therapist. 19 Table 4. Evidence Table of Implementation Factor/Barriers Studies Author Study Purpose HIT Element(s) Healthcare Setting Cutler DM, Feldman NE, Horwitz JR. “U.S. adoption of computerized physician order entry systems.” Health Affairs (Millwood) 2005; 24: 1654-63. To evaluate the types of hospitals that have implemented CPOE and potential reasons for low rates of implementation CPOE Leapfrog Group's Hospital Patient Safety Survey, Version 1, which surveyed hospitals in 22 geographic regions. More than 60% were from California, New York, and New Jersey Jha AK, Ferris TG, Donelan K, DesRoches C, Shields A, Rosenbaum S, Blumenthal D. “How common are electronic health records in the United States? A summary of the evidence.” Health Affairs (Millwood) 2006; 25: w496-507 To estimate EHR adoption in the US by systematically reviewing existing surveys of EHR adoption and use EHR, hospital/inpatient, outpatient/ambulatory pediatrics; included only those surveys in which the underlying population of interest consisted of physicians, physician group practices or hospitals Implementation Factors Evaluated Government hospitals more likely to report full implementation of CPOE than private nonprofit or forprofit hospitals; teaching hospitals and large hospitals are more likely to invest in CPOE; hospital profitability is not associated with CPOE investment About 24% of physicians use an EHR; about 5% have CPOE (system penetration) Findings/Outcomes None reported None reported Large physician offices were much more likely to have an EHR than solo or small physician practices (39% v. 16%) (barriers) Source: Authors’ analysis of results from 2007 Systematic Review of the HIT Literature 20 Author Study Purpose HIT Element(s) Healthcare Setting Kemper AR, Uren RL, Clark SJ. “Adoption of electronic health records in primary care pediatric practices.” Pediatrics 2006; 118: e20-4. To assess adoption of EHR EHR outpatient / ambulatory pediatrics - random sample of primary care pediatricians Lindenauer PK, Ling D, Pekow PS, Crawford A, NaglieriPrescod D, Hoople N, Fitzgerald J, Benjamin EM. “Physician characteristics, attitudes, and use of computerized order entry.” Journal of Hospital Medicine 2006; 1: 221-30. To assess the adoption of CPOE and the predictors of use of CPOE by attending physicians at 2 community hospitals where the use of CPOE is voluntary CPOE one 600-bed teaching hospital (where residents but not attendings are required to use CPOE) and one 125-bed community hospital (where use of CPOE is voluntary) in Massachusetts Implementation Factors Evaluated Cost was the main barrier identified to adoption of an EHR by pediatric practices that did not have an EHR (n=415), named in 94% of practices, followed by inability to find an EHR that met the practice needs (80%) and physician resistance (77%) Belief that CPOE may lead to fewer errors (facilitator) Belief that CPOE may not support workflow, may be slower than written orders and may not result in faster implementation (barriers) Source: Authors’ analysis of results from 2007 Systematic Review of the HIT Literature Findings/Outcomes None reported Medium and high utilizers of the CPOE system were more likely to believe that CPOE led to fewer medication and non-medication errors, and helps to ensure that important aspects of care do not slip through the cracks. 21 Author Study Purpose HIT Element(s) Healthcare Setting Lium JT, Laerum H, Schulz T, Faxvaag A. “From the front line, report from a near paperless hospital: mixed reception among health care professionals.” Journal of the American Medical Informatics Association 2006; 13: 668-75. To assess effects on use and efficiency of continued use of an EHR EHR community hospital serving a population of about 100,000 people in southern Norway McInnes DK, Saltman DC, Kidd MR. “General practitioners' use of computers for prescribing and electronic health records: results from a national survey.” The Medical Journal of Australia 2006; 185: 88-91. To describe how general practitioners use computers for clinical functions CPOE, EHR, electronic prescribing stratified random sample of Australian general practitioners Implementation Factors Evaluated None reported About 90% of Australian general practitioners were using a clinical software packages, the most common use being prescribing & checking for drugdrug interactions. (system penetration) Source: Authors’ analysis of results from 2007 Systematic Review of the HIT Literature Findings/Outcomes Compared with 2002, there was a significant increase in the use of the EHR for 11 of 19 tasks relevant to physicians, such as check and sign discharge summaries (which was at 100% in 2005), obtain results of new tests, ordering tests, etc. Nurses also reported increased use in 13 of 19 tasks relevant to them, including obtaining of results of investigations, reviewing the patient's problem, and collecting data for the nurses' summarizing report. None reported 22 Author Study Purpose HIT Element(s) Healthcare Setting Menachemi N. “Barriers to ambulatory EHR: who are 'imminent adopters' and how do they differ from other physicians?” Informatics in Primary Care 2006; 14: 101-8. To determine the characteristics of imminent adopters EHR probability sample of Florida primary care and specialist physicians, with a 28% response rate Implementation Factors Evaluated "Upfront" costs of hardware and software were the most commonly named barrier to adoption by both "imminent adopters" and physicians not considering EHR adoption, being cited by 69% and 70%, respectively. Ongoing maintenance costs, inadequate return on investment, perceived difficulties in data entry and lack of uniform standards were also commonly named as barriers Source: Authors’ analysis of results from 2007 Systematic Review of the HIT Literature Findings/Outcomes NR 23 Author Study Purpose HIT Element(s) Healthcare Setting Pizzi LT, Suh DC, Barone J, Nash DB. “Factors related to physicians' adoption of electronic prescribing: results from a national survey.” American Journal of Medical Quality 2005; 20: 2232. To identify factors related to physicians' adoption of electronic prescribing for outpatients electronic prescribing any outpatient setting Implementation Factors Evaluated Only 19% of responding MDs used electronic prescribing. (system penetration) Findings/Outcomes None reported Electronic prescribers were significantly more likely to be general practitioners versus specialists and working at academic or publicly-funded centers. Electronic prescribers were younger. (facilitators) The most commonly reported barriers included: system costs; time required to install the system, train, and change prescribing behavior; and uncertainty about which local pharmacies accept electronic prescriptions. 52% of traditional (paper) prescribers either Source: Authors’ analysis of results from 2007 Systematic Review of the HITwere Literature considering or ready to implement electronic 24 Author Study Purpose HIT Element(s) Healthcare Setting Schmittdiel J, Bodenheimer T, Solomon NA, Gillies RR, Shortell SM. “Brief report: The prevalence and use of chronic disease registries in physician organizations. A national survey.” Journal of General Internal Medicine 2005; 20: 855-8. To determine the prevalence of disease registries in physician organizations and the extent to which they are used to improve care data collection/data summary systems Medical groups and independent practice associations (IPAs) in the United States Implementation Factors Evaluated 47% of physician groups had at least one chronic disease registry. Diabetes registries were the most common. (system penetration) Findings/Outcomes Organizations with registries were more likely to provide feedback to MDs on patients’ use of medications and visits, as well as reminders to patients for retinopathy exams. Physician organizations that had better IT and greater incentives for quality were far more likely to have registries. Larger organizations (200 or more MDs) were more likely to have registries. (facilitators) Source: Authors’ analysis of results from 2007 Systematic Review of the HIT Literature 25 Author Study Purpose HIT Element(s) Healthcare Setting Stevenson KB, Barbera J, Moore JW, Samore MH, Houck P. “Understanding keys to successful implementation of electronic decision support in rural hospitals: analysis of a pilot study for antimicrobial prescribing.” American Journal of Medical Quality 2005; 20: 313-8. To evaluate the impact of an internet-based clinical DSS on empiric antimicrobial prescribing for community acquired pneumonia in rural hospitals in Idaho Medication prescribing support Patients with community-acquired pneumonia admitted to five rural hospitals in Idaho that were involved in a local rural health network Implementation Factors Evaluated Providers were unwilling to enter data directly in the Antibiotic Assistant because of perceived time requirements so each hospital was asked to organize an AMT to enter data (barrier) Source: Authors’ analysis of results from 2007 Systematic Review of the HIT Literature Findings/Outcomes There was a significant improvement in agreement with all treatment recommendations and agreement with recommended dose or antimicrobial agent subsequent to the intervention, but this improvement was primarily accounted for by the performance of one hospital 26
© Copyright 2024