How to use propensity scores in the analysis of nonrandomized designs Patrick G. Arbogast Department of Biostatistics Vanderbilt University Medical Center 2007Jan05 GCRC Research-Skills Workshop 1 Publications in Pub Med with phrase "Propensity Score" 180 140 120 100 80 60 40 20 Year 2007Jan05 GCRC Research-Skills Workshop 2 2006 2005 2004 2003 2002 2001 2000 1999 1998 1997 1996 1995 1994 1993 1992 1991 1990 1989 1988 1987 1986 1985 1984 0 1983 Number of publications 160 Motivation • Randomized clinical trials: randomization guarantees that on avg no systematic differences in observed/unobserved covariates. • Observational studies: no control over tx assignments, and E+/E- groups may have large differences in observed covariates. • Can adjust for this via study design (matching) or during estimation of tx effect (stratification/regression). 2007Jan05 GCRC Research-Skills Workshop 3 Analysis limitations • <10 events/variable (EPV), estimated reg coeff’s may be biased & SE’s may be incorrect (Peduzzi et al, 1996). – Simulation study for logistic reg. • Harrell et al (1985) also advocates min no. of EPV. • A solution: propensity scores (Rosenbaum & Rubin, 1983). – Likelihood that patient receives E+ given risk factors. 2007Jan05 GCRC Research-Skills Workshop 4 Intuition • Covariate is confounder only if its distribution in E+/E- differ. • Consider 1-factor matching: low-dose aspirin & mortality. – Age, a strong confounder, can be controlled by matching. • Can extend to many risk factors, but becomes cumbersome. • Propensity scores provide a summary measure to control for multiple confounders simultaneously. 2007Jan05 GCRC Research-Skills Workshop 5 Propensity score estimation • Identify potential confounders. – Current conventional wisdom: if uncertain whether covariate is confounder, include it. • Model E+ (typically dichotomous) as function of covariates using entire cohort. – – – – E+ is outcome for propensity score estimation. Do not include D+. Logistic reg typically used. Propensity score = estimated Pr(E+|covariates). 2007Jan05 GCRC Research-Skills Workshop 6 Counterintuitive? • Natural question: why estimate probability that a patient receives E+ since we already know exposure status? • Answer: adjusting observed E+ with probability of E+ (“propensity”) creates a “quasi-randomized” experiment. – – For E+ & E- patients with same propensity score, can imagine they were “randomly” assigned to each group. Subjects in E+/E- groups with equal (or nearly equal) propensity scores tend to have similar distribution in covariates used to estimate propensity. 2007Jan05 GCRC Research-Skills Workshop 7 Balancing score • For given propensity score, one gets unbiased estimates of avg E+ effect. • Can include large no. of covariates for propensity score estimation. – In fact, original paper applied propensity score methodology to observational study comparing CABG to medical tx, adjusting for 74 covariates in propensity model. 2007Jan05 GCRC Research-Skills Workshop 8 Applications • Matching. • Regression adjustment/stratification. • Weighting. 2007Jan05 GCRC Research-Skills Workshop 9 Propensity score matching • Match on single summary measure. • Useful for studies with limited no. of E+ patients and a larger (usually much larger) no. of E- patients & need to collect add’l measures (eg, blood samples). 2007Jan05 GCRC Research-Skills Workshop 10 Matching techniques • Nearest available matching on estimated propensity score. – – – – Select E+ subject. Find E- subjecdt w/ closest propensity score. Repeat until all E+ subjects matched. Easiest in terms of computational considerations. • Others: – Mahalanobis metric matching. – Nearest available Mahalanobis metric matching w/ propensity score-based calipers. 2007Jan05 GCRC Research-Skills Workshop 11 Illustrative example • Consider an HIV database: – E+: patients receiving a new antiretroviral drug (N=500). – E-: patients not receiving the drug (N=10,000). – D+: mortality. • Need to manually measure CD4. • May be potential confounding by other HIV drugs as well as 10 prognostic factors, which are identified & stored in the database. 2007Jan05 GCRC Research-Skills Workshop 12 Illustrative example (2) • Option 1: – Collect blood samples from all 10,500 patients. – Costly & impractical. • Option 2: – For all patients, estimate Pr(E+|other HIV drugs & prognostic factors). – For each E+ patient, find E- patient with closest propensity score. – Continue until all E+ patients match with Epatient. – Collect blood sample from 500 propensity-matched pairs. 2007Jan05 GCRC Research-Skills Workshop 13 The effectiveness of right heart catheterization in the initial care of critically ill patients (Connors et al, 1996) Objective Examine association between RHC use during 1st 24 hrs of ICU care & survival, length of stay, intensity of care, & cost of care. Design Prospective cohort study. Setting 5 US teaching hospitals, 1989 – 1994. Subjects Critically ill adult patients receiving care in an ICU for 1 of 9 prespecified disease categories (N=5735). Exposure RHC. Outcome(s) Survival, cost of care, intensity of care, length of stay in ICU & hospital. 2007Jan05 GCRC Research-Skills Workshop 14 RHC: add’l background • Teaching hospitals: – – – – – Beth israel Hospital, Boston. Duke University Medical Center, Durham. Metro-Health Medical Center, Cleveland. St Joseph’s Hospital, Marshfield, WI. UCLA. • Prespecified disease categories: – – – – – – – – Acute respiratory failure. COPD. CHF. Cirrhosis. Nontraumatic coma. Colon cancer metastatic to liver. Non-small cell cancer of lung. Multiorgan system failure with malignancy or sepsis. 2007Jan05 GCRC Research-Skills Workshop 15 RHC: differential E+/E• Decision to use RHC left to discretion of physician. • Thus, tx selection may be confounded with patient factors related to outcome. – eg, patients with low BP may be more likely to receive RHC, & such patients may also be more likely to die. 2007Jan05 GCRC Research-Skills Workshop 16 RHC: propensity score estimation • Panel of 7 specialists in critical care specified variables related to decision to use RHC. • Cpt propensity score, Pr(RHC|covariates), via logistic regression. • Covariates: – age, sex, yrs of education, medical insurance, primary & secondayr disease category, admission dx, ADHL & DASI, DNR status, cancer, 2-month survival probability, acute physiology component of APACHE III score, Glasgow Coma Score, wt, temparature, BP, respiratory rate, heart rate, PaO2/FiO2, PaCO2, pH, WBC count, hematocrit, sodium, potassium, creatinine, bilirubin, albumin, urine output, comorbid illnesses. 2007Jan05 GCRC Research-Skills Workshop 17 RHC: propensity score assessment • Adequacy of propensity score to adjust for effects of covariates assessed by testing for differences in individual covariates between RHC+/RHCpatients after stratifying by PS quintiles. Model each covariate as function of RHC & PS quintiles. – Covariates balanced if not related to RHC after PS adjustment. – 2007Jan05 GCRC Research-Skills Workshop 18 RHC: propensity score matching • For each RHC+, RHC- w/ same disease category & closest PS (+/- 0.03) identified. • Continued until all pairs identified. • PS difference for each pair calculated. Each pair w/ positive difference matched with pair w/ negative difference closest in magnitude. – Assure equal no.’s of pairs w/ positive & negative PS differences. • Final matched set: 1008 matched pairs. 2007Jan05 GCRC Research-Skills Workshop 19 RHC: PS-matched analysis of RHC & survival Survival Interval Survival, n(%) RHC- RHC+ OR (95% CI) 30 d 677 (67.2) 630 (62.5) 1.24 (1.03-1.49) 60 d 604 (59.9) 550 (54.6) 1.26 (1.05-1.52) 180 d 522 (51.2) 464 (46.0) 1.27 (1.06-1.52) Hospital 629 (63.4) 565 (56.1) 1.39 (1.15-1.67) 2007Jan05 GCRC Research-Skills Workshop 20 RHC: PS-matched analysis of RHC & resource use RHC-* RHC+* P Resource utilization 35.7 49.3 (cost/$100) (11.3, 20.6, 39.2) (17.0, 30.5, 56.6) 0.001 Avg TISS** 30 (23, 29, 38) 34 (27, 34, 41) 0.001 ICU stay, d 13.0 (4, 7, 14) 14.8 (5, 9, 17) 0.001 Total stay, d 23.8 (9, 15, 28) 25.1 (9, 16, 31) 0.14 * Mean (25th, 50th, 75th %-tiles); ** Therapeutic Intervention Scoring System. 2007Jan05 GCRC Research-Skills Workshop 21 Regression adjustment/stratification • Stratification on PS alone can balance distributions of covariates in E+/E- groups w/o exponential increase in no. of strata. • Rosenbaum & Rubin (1983) showed that perfect stratification based on PS will produce strata where avg tx effect w/i strata is unbiased estimate of true tx effect. 2007Jan05 GCRC Research-Skills Workshop 22 RHC: regression adjustment • Full cohort: N=5735. • PH regression: – Adjusted for PS, age, sex, no. of comorbid illnesses, ADL & DASI 2 wks prior to admission, 2month prognosis, day 1 Acute Physiology Score, Glasgow Coma Score, & disease category. • Question: why include covariates in main model in addition to PS (especially covariates already used to estimate PS)? 2007Jan05 GCRC Research-Skills Workshop 23 RHC: 30-day survival, entire cohort Disease cat.* RHC-, n(%) RHC+, n(%) HR (95% CI) 3551 2184 1.21 (1.09-1.25) ARF 1200 (34) 589 (27) 1.30 (1.05-1.61) MOSF 1245 (35) 1235 (57) 1.32 (1.11-1.57) CHF 247 (7) 209 (10) 1.02 (0.55-1.89) Other 859 (24) 151 (7) 1.06 (0.80-1.41) Overall ARF – acute respiratory failure, MOSF – multiorgan system failure. 2007Jan05 GCRC Research-Skills Workshop 24 RHC: resource utilization Mean (SE) P-value 7900 (3900) 0.001 Greater intensity of care, TISS 7.0 (0.3) 0.001 Longer ICU stay, d 2.2 (0.5) 0.001 Hospital length-of-stay, d 1.5 (0.8) 0.07 Higher cost, $ 2007Jan05 GCRC Research-Skills Workshop 25 Propensity score weighted regression adjustment • Weight patient’s contribution to reg model. • Inverse-probability-of-tx-weighted (IPTW) estimator (Robins et al, 2000): – – Estimates tx effect in pop whose distribution of risk factors equals that found in all study subjects. Wts: 1/PS(X) for E+ & 1/(1-PS(X)) for E-. • Standardized mortality ratio (SMR)-weighted estimator (Sato et al, 2003): – – Estimates tx effect in pop whose distribution of risk factors equals that found in E+ subjects only. Wts: 1 for E+ & PS(X)/(1-PS(X)) for E-. 2007Jan05 GCRC Research-Skills Workshop 26 Comparison of propensity score methods • Example: tissue plasminogen activator (t-PA) in 6269 ischemic stroke patients (Kurth et al, 2005): Multivariable logistic reg. – Logistic reg after matching on PS +/- 0.05 – Logistic reg adjusting for PS (linear term & deciles). – IPTW. – SMR. – 2007Jan05 GCRC Research-Skills Workshop 27 Propensity score distribution by t-PA+/t-PA- 2007Jan05 GCRC Research-Skills Workshop 28 Propensity analysis results 2007Jan05 GCRC Research-Skills Workshop 29 Propensity analyses restricting to PS 0.05+ 2007Jan05 GCRC Research-Skills Workshop 30 Propensity score vs other methods • Matching on individual factors: – Too cumbersome (eg, matching on 10 factors, each having 4 categories, resulting in ~1,000,000 combinations of patient characteristics). • Stratified analyses: same problem. • Regression (Cepeda et al, 2003): – <7 events/confounder – PS less biased, more robust, & more precise. – 8+ events/confounder – multiple reg preferable: • • 2007Jan05 Bias from multiple reg goes away, but still present for PS analysis (eg, ~25-30% bias when OR=2.0). Coverage probability (% of 95% CI’s containing true OR) decreases for PS analysis. GCRC Research-Skills Workshop 31 Benefits: • Useful when adjusting for large no. of risk factors & small no. of EPV. • Useful for matched designs (saving time & money). • Can be applied to exposure with 3+ levels (Rosenbaum, 2002). 2007Jan05 GCRC Research-Skills Workshop 32 Limitations • Can only adjust for observed covariates. • Propensity score methods work better in larger samples to attain distributional balance of observed covariates. – In small studies, imbalances may be unavoidable. • Including irrelevant covariates in propensity model may reduce efficiency. • Bias may occur. • Non-uniform tx effect. 2007Jan05 GCRC Research-Skills Workshop 33 Sample propensity analysis: RHC • E+: RHC use. – swang1 (0=RHC-, 1=RHC+) • D+: time-to-death, min(obs time, 30d). – Events after 30d censored. • • – RHC could not have a long-term effect. Such ill patients more affected by later tx decisions. t3d30, censor var=censor • N=5735 patients, N=1918 deaths w/i 30d. • 38.0% RHC+ & 30.6% RHC- died w/i 30d. 2007Jan05 GCRC Research-Skills Workshop 34 Kaplan-Meier plot by RHC status 0.40 0.30 0.20 0.10 log-rank: P<0.001 0.00 At risk: No RHC 3551 RHC 2184 0 2963 1721 10 Follow-up Time (days) No RHC 2007Jan05 GCRC Research-Skills Workshop 2654 1486 2480 1363 20 30 RHC 35 Propensity model Logistic reg: RHC+/- dependent var. • Adjusts for 50 risk factors. • Propensity score distribution by RHC groups: • 1 Propensity Score .8 .6 .4 .2 0 No RHC RHC RHC Status 2007Jan05 GCRC Research-Skills Workshop 36 Confounders related to RHC after propensity score (quintiles) adjustment (selected risk factors)? Propensity-adjusted, p-value No Yes Age 0.026 0.945 Gender 0.001 0.731 APACHE score <0.001 0.100 Weight (kg) <0.001 0.530 Mean BP <0.001 0.255 Respiratory rate <0.001 0.531 0.002 0.604 <0.001 0.470 WBC Creatinine 2007Jan05 GCRC Research-Skills Workshop 37 RHC & survival, entire cohort Model HR (95% CI) Unadjusted 1.30 (1.19 – 1.43) Multivariable 1.24 (1.12 – 1.38) Propensity score (linear) 1.22 (1.10 – 1.36) Propensity score (quintiles) 1.24 (1.11 – 1.37) 2007Jan05 GCRC Research-Skills Workshop 38 References • • • • • • • • • • • • Cepeda MS, Boston R, Farrar JT, Strom BL. Comparison of logistic regression versus propensity score when the number of events is low and there are multiple confounders. Am J Epidemiol 2003; 158: 280-287. Connors Jr AF, Speroff T, Dawson NV, et al. The effectiveness of right heart catheterization in the initial care of critically ill patients. JAMA 1996; 276: 889-897. D’Agostino Jr, RB. Tutorial in biostatistics: propensity score methods for bias reduction in the comparison of a treatment to a non-randomized control group. Stat Med 1998; 17: 2265-2281. Gum PA, Thamilarasan M, Watanabe J, Blackstone EH, Lauer MS. Aspirin use and all-cause mortality among patients being evaluated for known or suspected coronary artery disease. JAMA 2001; 286: 11871194. Harrell FE, Lee KL, Matchar DB, Reichart TA. Regression models for prognostic prediction: advantages, problems, and suggested solutions. Cancer Treatment Reports 1985: 69: 1071-1077. Kurth T, Walker AM, Glynn RJ, Chan KA, Gaziano JM, Berger K, Robins JM. Results of multivariable logistic regrssion, propensity matching, propensity adjustment, and propensity-based weighting under conditions of nonuniform effect. Am J Epidemiol 2006; 163: 262-270. Peduzzi P, Concato J, Kemper E, Holford TR, Feinstein AR. A simulation study of the number of events per variable in logistic regression analysis. J Clin Epidemiol 1996; 49: 1373-1379. Robins JM, Hernan MA, Brumback B. Marginal structural models and causal inference in epidemiology. Epidemiology 2000; 11: 550-560. Rosenbaum PR. Observational Studies. New York, NY: Springer-Verlag, 2002. Rosenbaum PR, Rubin DB. The central rol of the propensity score in observational studies for causal effects. Biometrika 1983; 70: 41-55. Rubin DB. Estimating causal effects from large data sets using propensity scores. Annals of Internal Medicine 1997; 127: 757-763. Sato T, Matsuyama Y. Marginal structural models as a tool for standardization. Epidemiology 2003; 14: 680-686. 2007Jan05 GCRC Research-Skills Workshop 39