Power analysis and sample size calculation Eirik Skogvoll, MD, PhD Professor, Faculty of Medicine Consultant, Dept. of Anaesthesiology and Emergency medicine 1 The hypothesis test framework Rosner tbl. 7.1 P (type II error) Truth Conclusion H0 correct 1– α α H0 accepted H0 rejected P (type I error) Power Type I error: rejecting H0 if H0 is true Type II error: accepting H0 if H0 is wrong 3 H0 wrong β 1–β Hypothesis A statement about one or more population parameters, e.g. μ ≤3 μ1 = μ2 σ2 = 14 π = 0,4 π1 =π2 ψ =1 β1 = 0 ρ >0 (Expected value, or “true mean”) (Two expected values) (variance) (probability of a “success”, in a binomial trial) (two probabilities) (odds ratio, from a contingency table) (regression coefficient) (correlation coefficient) 4 The sample size is a compromise… • Significance level • Required or optimal power • Available resources – Money – Time – Practical issues of recruiting patients or subjects • The problem under investigation 5 Power • The probability that some test will detect a difference, if it exists • With low power, even large differences may go unnoticed: Absence of evidence is not evidence of absence! • Power is usually set to 0.8 – 0.9 (80-90 %) • An infinitely small difference is impossible to detect • The researcher must specify the smallest difference of (clinical) interest • Power calculations requires knowledge of the background variability (usually the standard deviation) 6 Power or significance? • We wish to reduce the probability of both a type I error (α) and a type II error (β) , because… – A small α means that it is diffcult to reject H0 – A small β means that it is easier to reject H0 (and accept H1) – But minimizing α and β at the same time is complicated, because α increases as β decreases, and vice versa • Strategy: – Keep α at a comfortable level (0.10, 0.05, 0.01); the maximum ”acceptable” probability of making a type I error; i.e. rejecting H0 when it is true – Then choose a test that minimizes β, i.e. maximizes power ( =1 - β). Beware of H1: tests may have different properties! 7 Power (Rosner 7.5) One- sided alternative hypothesis H0: μ = μ0 vs. H1: μ = μ1 < μ0 H0 is rejected if zobs < zα and accepted if zobs ≥ zα . The decision dose not depend on μ1 as long μ1 < μ0. (Rosner Fig. 7.5) Power, Pr (reject H0|H0 wrong) = 1 – Pr(type II error) = 1 – β, however, depends on μ1. 8 Rosner Fig. 7.5 H1: μ = μ1 < μ0 9 vs. H 0: μ = μ 0 ⎛ σ2 ⎞ Under H1 , X ~ N ⎜⎜ μ1 , ⎟⎟ n ⎠ ⎝ We reject H0 if the observed statistic z is less than critical value zα , i.e. if zobs < zα . Using the corresponding value for X: ⎛ ⎞ ⎜ X −μ ⎟ σ ⎞ ⎛ 0 < zα ⎟ = Pr ⎜ X < μ0 + zα ⋅ Pr ( Z < zα ) = Pr ⎜ ⎟ σ n ⎝ ⎠ ⎜ ⎟ ⎜ ⎟ n ⎝ ⎠ (zα < 0) 10 ... remember that Z = X − μ1 σ ⇔ X =Z⋅ σ n + μ1 n σ ⎞ σ ⎞ ⎛ ⎛ σ + μ1 < μ 0 + zα thus, Power = P⎜ X < μ 0 + zα ⎟ = P⎜ Z ⎟ n⎠ n n⎠ ⎝ ⎝ 11 σ ⎞ σ ⎞ ⎛ ⎛ zα zα ⎟ ⎜ ⎟ ⎜ μ μ μ n ⎟ = P⎜ Z < 0 − μ1 + n⎟ = P⎜ Z + 1 < 0 + σ σ σ ⎟ σ σ σ ⎟ ⎜ ⎜ ⎟ ⎜ ⎟ ⎜ n ⎠ n n n ⎠ n n ⎝ ⎝ ⎞ ⎛ ⎞ ⎛ ⎟ ⎜ ⎟ ⎜ − ( μ 0 − μ1 ) μ μ μ μ − ⎛ ⎞ 0 1 0 1⎟ ⎜ ⎟ ⎜ = P⎜ Z < zα + ⋅ n⎟ =P Z< + zα = P Z < zα + σ σ ⎟ ⎜ ⎟ ⎜ σ ⎝ ⎠ ⎟ ⎜ ⎟ ⎜ n n ⎠ ⎝ ⎠ ⎝ ( μ 0 − μ1 ) ⎛ ⎞ = Φ⎜ zα + ⋅ n ⎟ ... because P(Z < z ) = Φ ( z ) ... table 3a σ ⎝ ⎠ 12 One-sided alternatives, in general: H0: μ = μ0 vs. As large as possible … H1: μ = μ1 | μ0 − μ1 | | μ0 − μ1 | ⎛ ⎞ ⎛ ⎞ Power = Φ ⎜ zα + ⋅ n ⎟ = Φ ⎜ − z1−α + ⋅ n⎟ σ σ ⎝ ⎠ ⎝ ⎠ (Rosner, Eq. 7.19) (remember that zα = – z1-α) 13 Factors influencing power: • • • • Power is reduced if α is reduced (zα becomes more negative) Power increases with increasing |μ0 – μ1| Power is reduced with increasing σ Power is increased with increasing n 14 Example (Rosner expl. 7.26) Birth weight. H0: μ0 = 120 oz, vs. H1: μ1 = 115 oz, α = 0.05, σ = 24, n = 100. Power = Pr(rejecting H0|H0 wrong) = 120 − 115 5 ⋅10 ⎞ ⎛ = Φ (0.438) Φ⎜ z0.05 + ⋅ 100 ⎟ = Φ (−1.645 + 24 24 ⎠ ⎝ = Pr( Z < 0.438) = 0.66 → Pr(type II error) = Pr(accept H0|H0 wrong) = β= 1– 0.669 = 0,331 15 Example (Rosner expl. 7.29) Birth weight. H0: μ0 = 120 oz, vs. H1: μ1 = 110 oz, α = 0.05, σ = 24, n = 100. 120 − 110 10 ⋅10 ⎞ ⎞ ⎛ ⎛ Φ⎜ z0.05 + ⋅ 100 ⎟ = Φ⎜ − 1.645 + ⎟ = Φ (2.52) 24 ⎠ 24 ⎝ ⎠ ⎝ = Pr( Z < 2.52) = 0.99 16 Power for a one sample test, two sided alternative (in general): H0: μ = μ0 vs. H1: μ = μ1 ≠ μ0 ⎛ ⎞ | μ0 − μ1 | Power ≈ Φ ⎜ − z α + ⋅ n ⎟ (Rosner Eq. 7.21) σ ⎝ 1− 2 ⎠ 17 From Power to Sample size… From Rosner Eq. 7.21 : ⎛ | μ 0 − μ1 | ⋅ n ⎞ ⎟ = 1 − β ... now we set (" fix" ) 1 − β Power = Φ⎜⎜ − z1−α / 2 + ⎟ σ ⎝ ⎠ ⎛ ⎛ | μ 0 − μ1 | ⋅ n ⎞ ⎞⎟ −1 ⎜ ⎟ = Φ −1 (1 − β ) Φ Φ⎜⎜ − z1−α / 2 + ⎟⎟ ⎜ σ ⎠⎠ ⎝ ⎝ − z1−α / 2 + | μ 0 − μ1 | ⋅ n σ = z1− β ⇔ ⇔ μ 0 − μ1 ⋅ n = σ ⋅ (z1− β | μ 0 − μ1 | ⋅ n σ = z1− β + z1−α / 2 σ ⋅ (z1− β + z1−α / 2 ) + z1−α / 2 ) ⇔ n = μ 0 − μ1 18 Sample size for one sample test, two sided alternative: H0: μ = μ0 vs. H1: μ = μ1 ≠ μ0 σ 2 (z1− β + z1−α / 2 )2 n= (Rosner Eq. 7.28) 2 (μ0 − μ1 ) 19 Two sample test: sample size (Rosner 8.10) Given X1 ∼ N(μ1, σ21) and X2 ∼ N(μ2, σ22) H0: μ1 = μ2 vs. H1: μ1 ≠ μ2 (two sided test), significance level α, and Power 1 – β. (σ 12 + σ 22 )( z n= 1− α + z1− β ) 2 2 ( μ2 − μ1 ) 2 …. in each group (Rosner Eq. 8.26) 20 Rules of thumb Lehr’s ”quick” equations for two-sample comparison: • 80 % power, α = 0.05 (two sided), Δ = standardized difference, m = no. in each group: m = 16/Δ2 • 90 % power, α = 0.05 (two sided), Δ = standardized difference, m = no. in each group m = 21/Δ2 21 The standardized difference Machin et al. ”Sample size tables for Clinical studies” 3.ed, 2009 Altman 1991 (figure 15.2) Need to specify the ”standardized difference”: Δ= | μ 2 − μ1 | σ … the least, clinically important difference expressed as standard deviations. May vary 0,1 – 1,0, where Δ= 0,1 – 0,2 constitutes a ”small” difference, while Δ = 0,8 is ”large”. 22 What is the background variability (”noise”)? • • • From other studies Pilot data From the range of possible values (”quick and dirty”): Range SD ≈ 4 • This results stems from the basic properties of the Normal (Gaussian) distribution, in which approx. 95 % of the population are found within ± 2SD of the mean 23 Example • 80 % power, α = 0.05 (two sided), Δ = standardised difference = 0.5 m = 16/Δ2 = 16/0.25 = 4*16 = 64 Total sample size = 64*2 = 128 24 Nomogram (Altman 1991) (total sample size) Standardised difference 25 Example 80 % power, α = 0.05 (two sided), Δ = 0.5 N (total) = 125 (total sample size) Standardised difference 26 Factors influencing power (1-ß) • • • • Power decreases if α is reduced (zα becomes more negative) Power increases with increasing |μ0 – μ1| Power decreases with increasing σ Power increases with increasing sample size ”n” 27 Factors leading to increased sample size (n) • • • • Large standard deviation Strict requirement for α (low level, 0.01- 0.02) Strict requirement for power (1 – β) (high level: 0.90-0.95) Low absolute difference between parameters under H1 and H0 (|μ1 – μ0|) 28 Confidence intervals • Most journals want CI’s reported along with (or instead of) p-values • The expected length of a confidence interval is determined by the sample size: l = 2·tn-1,1-α/2·s/√n …where t= relevant quantile of the t distribution with n-1 degrees of freedom s = sample standard deviation n = no. of observations 29 How to make efficient use of your subjects • Re-use subjects if possible, using a repeated measurements design. Intra-subject variance will be reduced, as each subject acts as his own control. • Enter more than one experimental factor, using a factorial design. Each subject counts twice (or thrice…) ! • Refine your measurement methods to reduce residual error • Never, never, never…! dichotomize a continuous outcome variable to a binary one (i.e. 1 or 0) ! This has the largest variance. 30 Software for sample size calculations • Machin et al. ”Sample Size Software for Clinical studies” 3.ed, 2009 -- but you need to set dot (.) as decimal separator / • Sample Power • R • StatExact (for special procedures) 31 References • • • • • • • • Eide, DM. Design av dyreforsøk (kap. 28) i A. Smith: Kompendium i forsøksdyrlære. http://oslovet.veths.no/dokument.aspx?dokument=262 (2007) Mead R. The Design of Experiments. Statistical Principles for Practical Applications. Cambridge: Cambridge Univ. Press; 1990 Altman DG. Practical statistics for medical research. London: Chapman & Hall; 1991. Altman DG, Bland JM. Absence of evidence is not evidence of absence. BMJ. 1995 Aug 19;311(7003):485. Rosner B. Fundamentals of Biostatistics. Belmont, CA: Duxbury Thomson Brooks/Cole; 2005. Schulz KF, Grimes DA. Sample size calculations in randomised trials: mandatory and mystical. Lancet. 2005 Apr 9-15;365(9467):1348-53. Bland JM. The tyranny of power: is there a better way to calculate sample size? BMJ. 2009 October 6, 2009;339(oct06_3):b3985-. Machin D, Campbell M, Tan SB, Tan SH. Sample Size Tables for Clinical Studies. 3 ed. Oxford: Wiley-Blackwell; 2009. 33
© Copyright 2024