Reducing Selection Bias in Quasi-Experimental Educational Studies Christopher Brooks Omar Chavez School of Information University of Michigan Department of Statistics and Data Sciences University of Texas at Austin [email protected] Jared Tritz [email protected] Stephanie Teasley School of Information University of Michigan School of Information University of Michigan [email protected] [email protected] ABSTRACT In this paper we examine the issue of selection bias in quasiexperimental (non-randomly controlled) educational studies. We provide background about common sources of selection bias and the issues involved in evaluating the outcomes of quasi-experimental studies. We describe two methods, matched sampling and propensity score matching, that can be used to overcome this bias. Using these methods, we describe their application through one case study that leverages large educational datasets drawn from higher education institutional data warehouses. The contribution of this work is the recommendation of a methodology and case study that educational researchers can use to understand, measure, and reduce selection bias in real-world educational interventions. 1. INTRODUCTION Evaluating the impact of novel educational pedagogies, strategies, programs, and interventions in quasi-experimental studies can be highly error-prone due to selection biases. The effect of these errors can be significant, and can lead to harm being done to learners, instructors, and institutions through misinformed decision-making. Further, the lack of confidence researchers have in their analyses of realworld deployments can lead to a decrease in situated experimentation. In this paper we describe a methodology to understand and correct for selection bias, restoring the confidence researchers and policy-makers can have in the results of quasi-experimental studies. A quasi-experimental study is one in which there is no randomized control population. This design contains the potential for selection bias of learners and is a principal challenge for measuring the effectiveness of the intervention delivered. Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. LAK ’15 March 16 - 20, 2015, Poughkeepsie, NY, USA ACM 978-1-4503-3417-4/15/03 ...$15.00. http://dx.doi.org/10.1145/2723576.2723614. In educational studies selection bias is often very difficult to eliminate: there are ethical considerations around the equal access to new learning technologies and programs, as well as pragmatic considerations such as dealing with open recruitment of learners where the bias is the result of self-selection. For instance, one would expect a series of workshops aimed at helping non-traditional learners excel in first year university would increase the grades of these students. But students may not elect to attend workshops randomly – students with existing strong study skills may be more predisposed to attending the workshops, and this latent variable may be more explanatory of the outcome than the workshop itself. The big data culture that has permeated academic (as well as other) institutions offers a solution to issues of selection bias in quasi-experimental interventions. Instead of limiting learner access to a technology or program to form a control group a priori, a subset of the overall population of learners is selected post hoc such that it best matches the group of learners who received the intervention. This creates a matched sample, and allows for an “apples-to-apples” comparison of outcomes between the two groups of learners while contextualizing how those groups might differ with respect to selection bias. The work presented here describes a process for identifying a matched sample of learners and contextualizing how the matched sample differs from those learners who have received educational interventions. This technique is especially important when communicating research results to decision makers within the higher education institution. By comparing the results of a treatment (a learning technology or program) effect on a group of learners against a similarlymatched sample, researchers can control for selection bias and make a more compelling argument about the impact (or lack there of) of their intervention. The contributions of this work are three fold: 1. A process for evaluating educational programs and interventions using subset matching, including an understanding of the important statistical tests that must be considered when contextualizing how good (unbiased with respect to some attributes) of a match is achieved. 2. A case study demonstrating how this method can be applied to reduce selection bias. 3. A free and open source software toolkit1 that allows educational researchers to execute this process directly, complete with the reporting of contextual statistics about the matched populations. 2. 2.1 ADDRESSING SELECTION BIAS Selection Bias To identify methods to deal with the problem of selection bias, we first describe what causes an educational intervention (treatment effect) to become biased. This bias is typically due to members of the treatment group voluntarily selecting themselves to participate in a given intervention. Other exogenous factors may exist, such as access to technology to engage in the treatment (e.g. Internet access, access to a smart phone) or selection that is based in part on demographic features. Consequently one might ask whether there is some sort of important difference in the individuals who elect to participate in a particular treatment or whether this a completely random phenomenon that is unrelated to the particular individuals. Regardless, we can mitigate the effects that various factors (the observed covariates) might have on the outcome of interest. Specifically, we want to estimate the extent to which the measured covariates influence our estimate of the average treatment effect. In general, our estimates of the average treatment effect involve matching members of the treatment condition to members of the comparison group (control) and relies on the “strong ignorability assumption” [2]. To state plainly: If we observe two individuals with the same base set of covariates or same propensity score, then the likelihood that either one would participate in the intervention is the same for both. Thus one electing to use the intervention and the second not is purely coincidental, hence comparing the two students’ outcomes is valid approach. Without this assumption it is impossible to infer all the selection bias has been removed from the estimated treatment effect [5, 2]. This inference in practice is limited to the covariates we are able to measure which inevitably have limitations either due to resources, time or ethical constraints limiting our ability to develop a “complete” set of potentially relevant factors to control. A natural consequence of this is explained in [3]: “It is important to realize, however, that whether treatments are randomly assigned or not, and no matter how large a sample size one has, a skeptical observer could always eventually find some variable that systematically differs in the E trials and C trials (e.g., length of longest hair on the child) and claim the average difference estimates the effect of this variable rather than the causal effect of Treatment. Within the experiment there can be no refutation of this claim; only a logical argument explaining that the variable cannot causally affect the dependent variable or additional data outside the study can be used to counter it.” This statement again points to the need for a researcher, when attempting to establish a causal relationship between 1 Available at https://github.com/usaskulc/population_ matching the application of a treatment and some sort of measured outcome, to use as complete a data set as possible. Covariates that are both related to an outcome of interest (such as test scores) as well as the covariates that effect the likelihood of an individual opting to participate in the treatment or intervention are relevant and discussed in [1]. Thus we would say selection bias due to some collection of variables X, is the bias that is introduced into our estimate of the average treatment effect, when we fail to account or control for X. Mathematically we can state this in the following way: Suppose the true treatment effect is Ttrue . Let T(−X) be the estimated treatment effect when we do not use X and T(+X) be the estimated treatment effect when we do use X, then: bias X removed = T(−X) − T(+X) (1) This equation allows us to calculate how much selection bias is introduced as the result of failing to account for a particular covariate or set of covariates X from the data we have available. For example, suppose we were interested in measuring the selection bias a variable such as Math SAT2 would introduce into our estimated treatment effect. We would first take our two groups, treatment and control, and match them based on some list of covariates (e.g. gender, socio-economic status, residency status) and not include Math SAT scores. Our estimated Treatment effect would be T1. We then would repeat the analysis but this time include Math SAT scores to get a second estimate of the treatment effect T2. The difference between T1 and T2 is our point estimate of the bias introduced by failing to include Math SAT as a control variable. For a full discussion of how various types of coavariates in education can account for selection bias see [5]. To summarize their findings: When it comes to which variables to use in one’s analysis, it is important to select data that was known before the treatment was administered, or at least could have been known before the treatment was administered. Using data collected after treatment assignment that can be influenced (changed in value) by the treatment itself is not useful since they can introduce bias either by causing an over or underestimate of the treatment effect. In educational research, variables relating to demographics, pretest information, prior academic achievement, topic and subject matter preferences, as well as psychological and personality predispositions, have been shown to affect either observed performance or propensity to participate in interventions. For instance, of these variables Shadish et al. [4] have found when it comes to interventions variables on proxy-pretests and topic preference together with demographics reduce nearly all bias for language-related outcomes and variables related to demographics, pretests. Further, prior academic achievement reduced about 75% of selection bias in a mathematics related intervention. It is worth noting however, the actual reductions in bias could also be due to the specific context of the intervention. However, the findings do provide supporting evidence that estimating treatment effects with observational data is an appropriate approach. 2 The Math SAT is a standardized test measuring the mathematics ability of entry level college students in the United States. 2.2 Methods for Subset Matching The question then arises as to which matching method best deals with the problem of selection bias. Should we match equally across all of the covariate measures that we have available, or should we use a univariate statistic that describes the propensity by which an individual relates to the treatment group? Rosenbaum and Rubin [2] provide advice on this issue, and suggest that using the covariates directly or propensity scores are both sufficient and neither is clearly better than the other on this matter. Instead, it is how related the set of all covariates which are used for matching is to the treatment assignment or outcome of interest which is important. With this caveat in mind, we outline two popular methods for finding a matched population for a quasi-experimental study. The first is a simple matching strategy, whereby scores are calculated for each covariate and each pair of subjects in the treatment and condition groups. A vector of scores for a particular pair of individuals represents the difference between subjects, and various differencing methods (e.g. Euclidean distance, Mahalanobis distance) may be used depending upon the form and distribution of the data. The second method is to collapse covariates for each individual into a propensity score using a regression approach such as linear or logistic regression. The result is then a single value that describes the likelihood an individual would receive some treatment condition. Care must be taken when forming propensity scores, especially in large matching datasets where the number and diversity of non-treatment individuals outweighs that of the treatment individuals. The difference between two individual’s scores forms a metric by which individuals can be matched. Regardless of the method used, both approaches form a matrix of treatment versus non-treatment individuals where intersection elements hold the similarity two individuals have to one another. This matrix can be solved as a linear assignment problem with the result being globally minimal (most similar) pairwise matches between the treatment and non-treatment populations. 2.3 Reporting on the Effects of Selection Bias While the subset matching technique attempts to minimize the overall difference between the treatment group and a matched sample, such an approach does not guarantee that suitable matches for a given analysis can be found. It is thus important to verify how well matched the treatment group is to the non-treatment group when presenting results on the effect of the treatment. This can be done by considering the similarity of each of the covariate distributions between the treatment and non-treatment groups. While there are several methods that might be used, a practical approach for continuous data is to compare distributions using a two-sampled Kolmogrov-Smirnov test, which is sensitive to both the shape and location of the distributions being compared. A second useful approach is to use the Mann-Whitney test. It has greater efficiency than the t-test on data not sampled from a normal distribution, and it is nearly as efficient as the t-test on normally distributed data. Both are conservative tests that will provide a comprehensive comparison of the distribution of two populations. In our experience, achieving a significant (e.g. p ≤ 0.05) confidence value using the Kolmogrov-Smirnov is difficult unless the non-treatment group is quite large and diverse, leading to excellent matches. Less robust tests of the quality of matches include means-test methods such the students paired t-test. 3. CASE STUDY: LEARNING COMMUNITIES The purpose of this section of the paper is not to outline a particular case study result per se, but to demonstrate how the techniques described can be used by educational researchers to come to conclusions about the effect of their interventions, by reducing the possible selection bias of the participatory sample. Learning communities programs3 at our institution group some students into residences based upon students’ interest in pursuing a particular domain or discipline. The goal of the learning communities programs is to provide students with a peer group for support, as well as provide opportunities for academic development. These programs have existed for more than a decade in various forms, and there is a strong interest in understanding the effect these programs have on student success and achievement. Current learning communities include women students in science and engineering programs; students who are interested in the health sciences; students pursuing the visual arts; students who are interested in social justice and community; and students who are interested in research. Students are not chosen at random for participation in learning communities programs. There is both self-selection bias (e.g. students who are interested in being in the learning community) as well as a formal selection phase (e.g. application forms included essays which are judged). Students may apply to many learning communities, but can only be accepted into one. Learning communities programs are only available for freshman (first year) university students. One common question for program evaluators of learning communities is whether participation in the program raises the overall academic achievement of students. A naive approach to answering this issue would be to conduct a t-test between students who are in a particular learning community and those who were not in any learning community along a particular outcome variable such as overall grade point average. Using one year of such data, the means difference is 0.12 (on a four point scale, see Table 1) suggesting the learning community students actually perform worse than non-learning community students; a t-test confirms significance at p ≤ 0.01. Student Group Learning Community Non-Learning Community N 103 6,090 average GPA 3.13 3.25 Table 1: Comparison of treatment (Learning Community) and non-treatment (Non-Learning Community) groups using a naive analysis. In determining how well matched the comparison groups are, a first step is to consider the list of variables being considered and similarity of the distribution of those variables within each group. This can be done with a two-tailed Kolmogrov-Smirnov test, and Table 2 shows the results between the two learning communities groups for a variety of 3 See http://www.lsa.umich.edu/mlc variables that are hypothesized as interacting with cumulative GPA. For the variables that are statistically significant (e.g. p < 0.01) we cannot reject the null hypothesis that the two samples come from different distributions. In this example, we see that only gender meets this criteria, suggesting that the distribution of gender in the two groups is different. Variables Sex Ethnic Group Citizenship Status Standardized Entrance Test Credits at Entry Parental Education Household Income KS Confidence (p) p < 0.001∗ ∗ ∗ p = 0.720 p = 1.000 p = 0.987 p = 0.164 p = 0.953 p = 0.661 Table 2: Comparison of the treatement and nontreatment groups across seven demographic and performance features before matching. In this case it was the treatment group that had a higher number of women than the non-treatment group. To reduce this bias a matched set can be created. Using the equal covariate matching method described at the beginning of Section 2.2, it is possible to minimize the bias that may exist. Balancing across the variables listed, a paired treatment–non-treatment dataset of 206 individuals can be created. Application of a two-tailed Kolmogrov-Smirnov test shows no significance at p = 0.01 level, though one variable (Household Income) is significant at the p = 0.05 level. The high confidence of all other p-values suggests this dataset is well balanced, except perhaps with respect to parental income. Variables Sex Ethnic Group Citizenship Status Standardized Entrance Test Credits at Entry Parental Education Household Income KS Confidence (p) p = 1.000 p = 1.000 p = 1.000 p = 0.996 p = 1.000 p = 1.000 p = 0.036∗ Table 3: Comparison of the treatment and nontreatment groups across seven demographic and performance features after matching. the p=0.036 level) has been eliminated.4 4. 4 The result of the matching process are two populations of the same size with individuals in the first directly matched to individuals in the second. Thus, a paired t-test for statistical significance can be used on outcome variables. Considering GPA, the paired t-test returns a statistically significant difference (p = 0.003), with the means difference between the groups being 0.18 points in favor of the non-treatment group. In short, the researcher can now say with greater certainty that there is a difference between the treatment and non-treatment students and that bias introduced because of an observed variables (except perhaps Household Income, at CONCLUSIONS Selection bias in quasi-experimental studies can undermine the confidence decision-makers have in the results of analyses, and lead to possible misunderstandings and poor policy decisions. Yet institutions, researchers, and practitioners, are often unable to run randomized controlled experiments of learning innovations based on pragmatic or ethical concerns. This paper has introduced a methodology by which researchers can use contextualize the results of their analysis and reduce selection biases. Leveraging big educational datasets and institutional data warehouses, researchers can often mitigate selection bias by finding a comparison group of learners who did not undergo a particular treatment. Learners can be compared equally across all covariates (e.g. demographics, previous performances, or preferences), or covariates can be collapsed into a single propensity score which can be used as the basis for matching. The end result of the matching process is a paired dataset of learners who have undergone a treatment and similar learners who did not receive the treatment. The researcher can then apply post-hoc analysis as appropriate. In this work we have included an example of this method applied to a case study educational program which is particularly affected by selection bias: university learning communities. These learning communities are heavily biased based on the sex of participants (Table 2). After controlling for this bias, an increase of 66% is seen in the means differences between the treatment and control groups (from 0.12 to 0.18 in GPA units). Whether this is significant enough to change policy or deployment of the program depends on how decision makers weight this particular outcome. There may be alternative student outcomes such as satisfaction, time to degree completion, or co-curricular achievements that influence policy in this area. What is important here is that the researcher can feel confident that these results more accurately reflect the effects on the treatment population given the kinds of learners who would opt-in to the treatment. In our experience, however, creating matched pairs of learners rarely results in perfect results, thus contextualizing the goodness of fit between the two groups of learners is important. This can be done both before the matching as well as afterwards, using the Kolmogrov–Smirnov statistic. This technique can describe which covariates may not be possible to match on; an insight which is essential when forming educational policy. As the level of significance goes up (a declining p value), the more likely it is that variability (noise) in the data will cause for a rejection of a particular hypothesis. As more variables are considered, the chance of spurious correlation at a well-accepted level such as p = 0.05 or p = 0.01 for one variable increases. A more conservative value for a given confidence level can be achieved by dividing the alpha (e.g. 0.05) necessary for statistical significance by the number of variables being considered (7), a Bonferroni correction which controls the family-wise error levels. In this example, at p = 0.05, one would then expect only values of p ≤ 0.0083 to be consider statistically significant. Thus the appearance of Household Income being statistically significantly different between the two distributions should be questioned as to whether it is a spurious result. 5. ACKNOWLEDGEMENTS Thanks to Dr. Jim Greer from the University of Saskatchewan for motivating earlier work in this area. Also, thanks to Dr. Ben Hansen at the University of Michigan for insights on using propensity and prognostic scores and their application to matching problems. Finally, thanks to Dr. Brenda Gunderson from the University of Michigan for her support in investigating these issues in the E2 Coach framework. 6. REFERENCES [1] B. Hansen. The prognostic analogue of the propensity score. Biometrika, pages 1–17, 2008. [2] P. Rosenbaum and D. Rubin. The central role of the propensity score in observational studies for causal effects. Biometrika, 70(1):41–55, 1983. [3] D. Rubin. Estimating causal effects of treatments in randomized and nonrandomized studies. Journal of educational Psychology, 1974. [4] W. R. Shadish, M. H. Clark, and P. M. Steiner. Can Nonrandomized Experiments Yield Accurate Answers? A Randomized Experiment Comparing Random and Nonrandom Assignments. Journal of the American Statistical Association, 103(484):1334–1344, Dec. 2008. [5] P. M. Steiner, T. D. Cook, W. R. Shadish, and M. H. Clark. The importance of covariate selection in controlling for selection bias in observational studies. Psychological methods, 15(3):250–67, Sept. 2010.
© Copyright 2024