Running head: TUTORIAL IN STATISTICS: SAMPLE SIZE DETERMINATION
Tutorial in Statistics: Sample Size Determination for ANOVA and MANOVA
Amy Williams
University of Calgary
1
TUTORIAL IN STATISTICS: SAMPLE SIZE DETERMINATION
2
Tutorial in Statistics: Sample Size Determination for ANOVA and MANOVA
Introduction
ANOVA and MANOVA are two forms of statistical analysis that are at the forefront of
statistical research today. ANOVA, which stands for analysis of variance, is an approach to
analyzing data for two or more groups that involves breaking down the dependent variable into
between-group and within-group variances (Kerlinger & Lee, 2000). MANOVA, or multivariate
analysis of variance, is similar to ANOVA; this approach, however, involves the analysis of two
or more groups on two or more dependent variables, as well as the examination of the
correlations that exist between these variables (Stevens, 2009). In order for researchers to carry
out either ANOVA or MANOVA effectively, certain factors must be taken into consideration.
One of these factors is the determination of an appropriate sample size, or the number of
participants needed to take part in a particular study. This tutorial sheds light on the importance
of sample size in research, highlights important theoretical information about this topic, and
gives an overview of how appropriate sample size determination is carried out for both ANOVA
and MANOVA.
The Importance of Sample Size Determination
Determining sample size is an important aspect of research for several reasons. If a
sample size is too small or too large, this can have economic implications for researchers in
terms of wasted or overused resources, respectively (Lenth, 2001). Conducting a study can be
expensive, so knowing ahead of time how many participants are needed allows researchers to
anticipate potential costs (Kerlinger & Lee, 2000). There are also ethical concerns related to
sample size. According to Lenth (2001),
TUTORIAL IN STATISTICS: SAMPLE SIZE DETERMINATION
3
An undersized experiment exposes the subjects to potentially harmful treatments without
advancing knowledge. In an oversized experiment, an unnecessary number of subjects
are exposed to a potentially harmful treatment, or are denied a potentially beneficial one
(p. 187).
For these reasons, researchers and statisticians are advised to invest time in planning statistical
studies and ensure that the number of subjects involved will provide them with statistically
significant results. Utts and Heckard (2006) encourage researchers to ask two important
questions to help guide research design planning: how precise will my results be if my study
contains a particular number of participants? And how large does my sample need to be in order
to obtain statistically significant results?
Important Theoretical Information
Despite the importance of sample size determination, the literature related to this topic is
not extensive (Lenth, 2001). Introductory statistics textbooks generally provide comprehensive
overviews of this topic; the authors of these books tend to emphasize the use of larger sample
sizes over smaller sample sizes. Kerlinger and Lee (2000) give an important reason for this
preference: “the smaller the sample the larger the error, and the larger the sample the smaller the
error” (p. 175). Utts and Heckard (2006) shed light on the greater accuracy and minimized
uncertainty that having larger sample sizes brings to research design and statistical studies, but
they also caution students and researchers about the statistically significant results that even
small effects of large samples can elicit. Sample size determination is therefore not just a matter
of simply choosing to use a large sample because more benefits are associated with this choice;
careful consideration of the overall statistical study is imperative.
TUTORIAL IN STATISTICS: SAMPLE SIZE DETERMINATION
4
How to Determine Sample Size
There are several steps involved in determining the sample size for a statistical study.
Lenth (2001) sheds light on the power approach, which involves five important steps: first, the
researcher must identify both the null and the alternative hypothesis; second, he or she must
decide on a significance level; third, the researcher must also decide on an effect size; fourth, he
or she must gather missing values from related studies or published literature; and fifth, he or she
must then decide on the power value for the study. Kerlinger and Lee (2000) focuses on three
similar aspects of sample size determination, beginning with a calculation of the actual or
estimated population standard deviation value (this information can come from previous studies)
and identifying the amount of error that will be tolerated, and then estimating the probability of
making a Type I error. In addition, Kerlinger and Lee (2000) offer a formula for sample size
determination which is given in Table 1.
Lenth (2001) stresses the importance of power analysis in estimating an appropriate
sample size, stating that this method is “one of the most popular approaches to sample size
determination” (p. 187). Power, after all, is an integral component of statistical analysis because
it is what analysis is based upon: the ability to detect statistical differences when they in fact
exist. Stevens (2009) refers to power as “the probability of making a correct decision” (p. 162).
Table 1
Sample Size Estimation Equation__________________________________________
n = ( Z2 * σ2 ) / d2
Z2 = standard score corresponding to the specified probability of risk
σ = the standard deviation of the population
d = specified deviation
TUTORIAL IN STATISTICS: SAMPLE SIZE DETERMINATION
5
Note. From Foundations of Behavioral Research (4th ed.) by F.N. Kerlinger and H.B. Lee
(2000). Belmont, CA: Cengage Learning, p. 297. Copyright 1992 by Fred N. Kerlinger.
Sample Size Determination Example: ANOVA
Description of the Study
A group of educational researchers in Kuwait wishes to conduct an experiment involving
fifth grade students that attend English-speaking private schools. This study would require
students – both boys and girls – to take part in a two-week (one 90-minute period a day) Social
Studies instructional session in which they would be randomly placed in one of three groups:
Group 1: No Study Plan (Control Group)
Group 2: Implementation of Social Studies “Study Plan”
Group 3: Memorization
The purpose of this study is to determine if having students take part in a Social Studies
“Study Plan” prior to writing a curriculum-based assessment would enhance their performance
on this assessment compared to those students who do not take part in a study plan or those who
are simply encouraged to memorize notes that they have taken during the two-week period.
A study plan involves a contract that students sign in class and then take home to have
their parents sign; it involves students making a commitment to review their notes using different
strategies several times a week before a scheduled test. Strategies include, but are not limited to,
comparing and contrasting ideas that students have learned using a Venn diagram, summarizing
main ideas, or illustrating an important concept. The study plan is meant to elicit higher-level
thinking and make students aware of their own learning styles in an attempt to discourage them
from simply memorizing their notes.
How many students should be placed in each group in order for the group of researchers
to detect a statistically significant result? To answer this question, the group of researchers will
TUTORIAL IN STATISTICS: SAMPLE SIZE DETERMINATION
6
follow the aforementioned steps outlined in Lenth’s (2001) power approach and calculate the
sample size using Kerlinger and Lee’s (2000) sample estimation formula (the researchers could
also use one of the online sample calculators listed in the Resource section of this tutorial to
calculate the sample size).
Power Analysis
First, the group of researchers will propose a hypothesis. For this particular study, they
predict that the mean score for both Group 1 and Group 3 will be lower than that for Group 2.
The null hypothesis, of course, states that a difference between the groups will not be present.
Next, the researchers will choose a significance level. In the social sciences, a significance level
of .05 is usually chosen for statistical studies, which means that the researcher has less that a 5%
chance of making a Type 1 error, or choosing to reject the null hypothesis when it is in fact true
(Stevens, 2009). The researchers will also choose an effect size. According to Stevens (2009),
the effect size refers to “how much of a difference the treatments make, or the extent to which
the groups differ in the population on the dependent variable(s) (p. 163). Generally, small or
medium effect sizes are chosen in research pertaining to the social sciences (Stevens, 2009). A
medium effect size is 0.5. Next, the population standard deviation is either estimated or derived
from past research; for this particular study, the researchers estimate a standard deviation of 0.75.
Finally, they decide on the power value. Lenth (2001) considers a power of .80 common in
statistical studies, and the researchers of this particular study have chosen it as their target.
n = ( Z2 * σ2 ) / d2
According to the above formula, Z represents the standard score associated with the level
of significance, or risk (Kerlinger & Lee, 2000). In the t-distribution probability chart in the
appendix of Kerlinger and Lee’s book entitled Foundations of Behavioral Research, Z = 1.96.
TUTORIAL IN STATISTICS: SAMPLE SIZE DETERMINATION
7
The aforementioned standard deviation of the population (σ) is 0.75 and the specified deviation
(d) which identifies the precision the researcher hopes to obtain is set at 0.3.
n = 1.96 (0.75) = 3.842 (0.56) = 2.16 = 24
0.3
0.09
0.09
The number of participants needed for each group for this particular ANOVA is 24 (or 72
participants in total). This formula is an effective method of obtaining the required sample size
for a study involving random sampling; a variation of this formula exists for studies in which the
population size from which the sample will be taken is known (Kerlinger & Lee, 2000).
(Lauter, 1978)
Sample Size Determination Example: MANOVA
Stevens (2009) presents a condensed table that indicates the number of subjects needed
per group in a MANOVA depending on the desired effect size (see Table 2). More
comprehensive tables for three, four, five, and six-group MANOVA, however, have been
adapted from those Lauter (1978) created and are included at the back of Stevens’ book entitled
Applied Multivariate Statistics for the Social Sciences; these tables are invaluable to both
students and researchers as they minimize the estimation and calculations the formulas such as
the one mentioned above requires.
In order to effectively use these statistical tables, a researcher must make three decisions:
the number of groups required for the study, the effect size, and the significance level (either
0.05 or 0.01). Once these decisions have been made, sample size for MANOVA can be
determined.
Table 2
Sample Size Estimation for MANOVA
Groups
Effect Size
3
4
5
6
TUTORIAL IN STATISTICS: SAMPLE SIZE DETERMINATION
8
Very Large
12-16
14-18
15-19
16-21
Large
25-32
28-36
31-40
33-44
Medium
42-54
48-62
54-70
58-76
Small
92-120
105-140
120-155
130-170
From Applied multivariate statistics for the social sciences (5th ed.) by J.P. Stevens. New York,
NY: Taylor & Francis Group. Copyright 2009.
Description of the Experiment
A researcher wants to determine the impact that both marital status and living in housing
provided by the school (independent variables) has on overseas teachers’ ratings of job
satisfaction and length of employment in Kuwait (dependent variables). The researcher randomly
obtains participants from this study from the various international schools in Kuwait and has
them fill out a survey that includes questions about job satisfaction (for example, “overall, how
would you rate your experience at your particular school”) and length of employment. Upon
receiving completed surveys, he plans on dividing the participants up into four groups based on
the independent variable:
Group 1: Married and living in school accommodations
Group 2: Single and living in school accommodations
Group 3: Married and living in own accommodations
Group 4: Single and living in own accommodations.
He has chosen a power of .80 and has decided upon a medium effect size (basing both decisions
on what is common in most social science studies; he could have also researched past studies on
a similar topic and chosen an effect size this way). According to the statistical table in the
appendix of Steven’s (2009) book, this researcher would require 50 participants per group (200
participants in total) in order for his study to produce statistically significant results.
If the researcher decides to change the study so that there are now four dependent
variables (‘job satisfaction,’ ‘length of employment,’ ‘overall school rating,’ and ‘work
TUTORIAL IN STATISTICS: SAMPLE SIZE DETERMINATION
environment rating’), and he now anticipates a large effect size, the number of participants
required per group would decrease to 37. Sample size determination for MANOVA is therefore
dependent upon the number of dependent variables as well as the effect size and the chosen
power value. This method of sample size determination is referred to as a priori estimation,
which is a method that relies extensively on power values (Stevens, 2009).
Summary and Conclusions
Determining the sample size for a statistical study is an important aspect of a quality
research design; it is also a difficult process (Lenth, 2001). Several methods for determining
sample sizes for ANOVA, MANOVA, and other analyses exist, including the power approach,
the random sampling formula, and a priori estimation. Each of these methods requires the
researcher’s knowledge of the effect size and power and is used before data for a particular
statistical study is collected. When used with careful consideration and planning, these methods
are effective tools.
The post hoc estimation of power is also an option for determining sample size; this
method, however, is used after a study has actually been carried out and involves the researcher
interpreting the results and identifying the effect sample size and effect size have on the power
(Stevens, 2009). Not all researchers and statisticians favour this method. Lenth (2001) cautions
against retrospective planning such as the post hoc, stating that the goal of this method involves
“collect[ing] enough additional data to obtain statistical significance, while ignoring scientific
meaning” (p. 191).
Regardless of which sample size estimation method a researcher chooses, he or she must
acknowledge that sample size, effect size, and power are dependent on one another and that in
order to estimate or determine one, information about the other two is needed. A wealth of
9
TUTORIAL IN STATISTICS: SAMPLE SIZE DETERMINATION
10
information, statistical software, and web resources are available to help researchers determine
the appropriate sample size needed for their statistical studies so that this important task – as
daunting as it may seem – is not overlooked or neglected altogether.
Online Resources
Java Applets for Power and Sample Size:
http://www.stat.uiowa.edu/~rlenth/Power/
IBM SPSS Sample Power Program:
http://www-01.ibm.com/software/analytics/spss/products/statistics/samplepower/
National Statistical Service Sample Size Calculator:
http://www.nss.gov.au/nss/home.nsf/pages/Sample+Size+Calculator+Description?OpenDocume
nt
TUTORIAL IN STATISTICS: SAMPLE SIZE DETERMINATION
11
References
Kerlinger, F. N., & Lee, H. B. (2000). Foundations of Behavioral Research (4th ed.). Belmont,
CA: Cengage Learning.
Lauter, J. (1978). Sample size requirements for the T2 test of MANOVA (tables for one-way
classification). Biometrical Journal, 20, 389-406.
Lenth, R. V. (2001, August). Some practical guidelines for effective sample size determination.
The American Statistician, 55, 187-193.
Stevens, J. P. (2009). Applied multivariate statistics for the social sciences (5th ed.). New York,
NY: Taylor & Francis Group.
Utts, J. M., & Heckard, R. F. (2006). Statistical Ideas and Methods. United States: Thomson
Brooks/Cole.