Agreement Indices in MultiLevel Analysis Ayala Cohen Faculty of Industrial Engineering& Management Technion-Israel Institute of Technology May 2007 1 Outline • Introduction ( on Interrater agreementIRA) • rWG(J) Index of agreement • AD ( Absolute Deviation), Alternative measure of agreement -------------------------------Review Our work (2001) (2007) Etti Doveh Etti Doveh Uri Eick Inbal Shani 2 INTRODUCTION Why we need a measure of agreement In recent years there has been a growing number of studies based on multi-level data in applied psychology, organizational behavior, clinical trials. Typical data structure: Individuals within groups ( two levels) Groups within departments (three levels) 3 Constructs • Constructs are our building blocks in developing and in testing theory. • Group-level constructs describe the group as a whole and are of three types (Kozlowski & Klein, 2000): – Global, shared, or configural. 4 Global Constructs • Relatively objective, easily observable, descriptive group characteristics. • Originate and are manifest at the group level. • Examples: – Group function, size, or location. • No meaningful within-group variability. • Measurement is generally straightforward. 5 Shared Constructs • Group characteristics that are common to group members • Originate in group members’ attitudes, perceptions, cognitions, or behaviors – Which converge as a function of socialization, leadership, shared experience, and interaction. • Within-group variability predicted to be low. • Examples: Group climate, norms. 6 Configural Group-Level Constructs • Group characteristics that describe the array, pattern, dispersion, or variability within a group. • Originate in group member characteristics (e.g., demographics, behaviors, personality) – But no assumption or prediction of convergence. • Examples: – Diversity, team star or weakest member. 7 Justifying Aggregation • Why is this essential? – In the case of shared constructs, our construct definitions rest on assumptions regarding withinand between-group variability. – If our assumptions are wrong, our construct “theories,” our measures, are flawed and so are our conclusions. • So, test both: Within group agreement The construct is supposed to be shared, is it really? – Between group variability (reliability) Groups are expected to differ significantly, do they really? 8 Chen, Mathieu & Bliese ( 2004) proposed a framework for conceptualizing and testing multilevel constructs. This framework includes the assessment of inter-group agreement Assessment of agreement is a pre-requisite for arguing that a higher level construct can be operationalized . 9 Distinction should be made between: Interrater reliability (IRR= Interrater Reliability) and Interrater agreement (IRA= Interrater Agreement) Many past studies wrongly used the two terms interchangeably in their discussions. 10 The term interrater agreement refers to the degree to which ratings from individuals are interchangeable ; namely, it reflects the extent to which raters provide essentially the same rating. (Kozlowski & Hattrup,1992;Tinsley&Weiss,1975 . ( 11 Interrater reliability refers to the degree to which ratings of different judges are proportional when expressed as deviations from their means 12 Interrater reliability (IRR) refers to the relative consistency and assessed by correlations Interrater agreement (IRA) refers to the absolute consensus in scores assigned by the raters and is assessed by measures of variability. 13 Scale of Measurement • Questionnaire with J parallel items on a Likert scale with A categories e.g. A=5 1 2 3 4 5 Strongly Disagree Indifferent Agree disagree 14 Strongly agree Example Item Rater1 Rater2 k=3 raters Likert scale A=7 categories J= 5 items 15 Rater 3 Dev from mean 1 7 4 3 1 2 5 2 1 -1 3 5 2 1 -1 4 6 3 2 0 5 7 4 3 1 Prior to aggregation , we assessed within unit agreement on…… To do so, we used two complementary approaches (Kozlowski & Klein, 2000) A consistency based approach ,computation of the intra class correlation coefficient ,ICC(1) A consensus based approach ( index of agreement) 16 How can we assess agreement ? • Variability measures: e.g. Variance MAD( Mean Absolute Deviation) Problem: What are “small / large” values ? 17 The most widely used index of interrater agreement on Likert type scales has been rWG(J), introduced by James ,Demaree & Wolf (1984). J stands for the number of items in the scale 18 Examples when rWG(J) was used to assess interrater agreement Group cohesiveness Group socialization emphasis Transformational and transactional leadership Positive and negative affective group tone Organizational climate 19 This index compares the observed within group variances to an expected variance from “random responding “ In the particular case of one item (stimulus) , (J=1) this index is denoted as rWG and is equal to 2 rWG 20 Sx 1 2 2 rWG Sx Sx 1 2 2 is the variance of ratings for the single stimulus 2 is the variance of some “null distribution” corresponding to no agreement 21 Problem: A limitation of rWG(J) is that there is no clear-cut definition of a random response and the appropriate specification of the null distribution which models no agreement is debatable If the null distribution used to define 2 fails to model properly a random response, then the interpretability of the index is suspect. 22 The most natural candidate to represent non agreement is the uniform (rectangular) distribution, which implies that for an item with number of categories which equals to A, the proportion of cases in each category will be equal to 1/A 23 For a uniform null For an item with A number of categories A 1 12 2 2 null 2 rWG 24 Sx 1 2 How to calculate the sample variance Sx 2 ? We have n ratings and suppose n is “small” 1 n 2 Sx ( Xk X ) n k 1 2 25 Example A=5 k=9 raters: 3 3 3 3 5 5 5 5 4 ( With ( n-1) in the denominator), Sx 2 rW G 1 2 Sx 2 1 2 2 rWG 26 2 null A2 1 12 1 1 1 2 2 James et al. (1984): “ The distribution of responses could be non- uniform when no genuine agreement exists among the judges. The systematic biasing of a response distribution due to a common response style within a group of judges be considered. This distribution may be negatively skewed, yielding a smaller variance than the variance of a uniform distribution”. 27 Slight Skewed Null 1 = .05 2 = .15 3 = .20 Yielding 2 null 4 = .35 5 = .25 = 1.34 Used as a “null distribution” in several studies (e.g., Schreisheim et al., 1995; Shamir et al., 1998). Their justification for choosing this null distribution was that it appears to be a reasonably good approximation of random response to leadership and attitude questionnaires. 28 Null Distributions A=5 0.35 slight skewed 0.3 0.25 0.2 0.15 0.1 0.05 0 1 29 2 3 4 5 James et al )1984( suggested several skewed distributions , (which differ in their skewness and variance) to accommodate for systematic bias . 30 Often, several null distributions (including the uniform) could be suitable to model disagreement. Thus, the following procedure is suggested. Consider the subset of likely null distributions and calculate the largest and smallest null variance specified by this subset. 31 Additional “problem” The index can have negative values rWG Sx 2 1 2 Larger variance than expected from random response 32 Bi-modal distribution: ( extreme disagreement) Example: A=5 Half answer 1 , Half answer 5 Variance: 4 Uniform variance 2 null A2 1 2 12 2 rWG 33 Sx 1 2 1 What to do when rWG is negative? James et al ( 1984) recommended replacing a negative value by zero. Criticized by Lindell et al. ( 1999) 34 For a scale of J items rWG ( J ) s 35 2 J[1 ( s 2 / 2 )] 2 2 2 2 J[1 ( s / )] s / Is the average variance over the J items For a scale of J items 2 s r1 1 2 J[1 ( s / )] rW G( J ) J[1 ( s 2 / 2 )] s 2 / 2 Jr1 Jr1 Jr1 (1 r1) 1 ( J 1)r1 2 36 2 For a scale of J items rWG ( J ) Jr1 1 ( J 1)r1 Spearman Brown Reliability : kr1 1 (k 1)r1 37 Example 3 raters 7 categories Likert scale 5 items rWG( J ) J[1 ( s 2 / 2 )] J[1 ( s 2 / 2 )] s 2 / 2 5[1 ( s 2 / 2 )] 0.66 2 2 1 4[1 ( s / )] 1 ( s 2 / 2 ) 0.2777 38 Item Rater Rater Rater 1 3 2 1 2 3 4 5 7 5 5 6 7 4 2 2 3 4 3 1 1 2 3 Var* 78/27 78/27 78/27 78/27 78/27 Var calculated with n in denominator Since its introduction, the use of rWG(J) has raised several criticisms and debates. It was initially described by James et al. (1984) as a measure of interrater reliability . Schmidt & Hunter (1989) criticized this index claiming that an index of reliability cannot be defined on a single item 39 In response, Kozlowski and Hattrup (1992) argued that it is an index of agreement not reliability. James, Demaree & Wolf (1993) concurred with this distinction, and it has now been accepted that rWG(J) ) is a measure of agreement . 40 Lindell, Brandt and Whitney (1999) suggested, as an alternative to , rWG(J) a modified index which is allowed to obtain negative values (even beyond minus 1) 2 r *WG ( J ) 41 s 1 2 The modified index r*WG(J) provides corrections to two of the criticisms which were raised against rWG(J) . First, it can obtain negative values, when the observed agreement is less than hypothesized. Secondly, unlike rWG(J) it does not include a Spearman-Brown correction and thus it does not depend on the number of items (J( 42 Academy of management Journal 2006 Does Ceo Carisma matter…..Agle et al. 128 Ceo’s 770 team members “Because of strengths and weaknesses of various interrater agreement measures, we computed both the intraclass correlation statistics ICC(1) and ICC(2), and the interrater agreement statistics r*WG(J)……………” 43 Agle et al. 2006 Overall, the very high interrater agreement justified the combination of individual manager’s responses into a single measure of charisma for each CEO…. -----------------They display ICC(1)= ICC(2)= r*WG(J) = One number ? 44 Ensemble of groups(RMNET) Shall we report median, mean? ….” Observed distributions of rWG(J) are often wildly skewed ….medians are the most appropriate summary statistic”….. 45 Ehrhart,M.G.(Personnel Psychology, 2004) Leadership and procedural justice climate –as antecedents of unit-level organizational citizenship behavior Grocery store chain 3914 employees in 249 departments 46 ….”The median rwg values across the 249 departments were : 0.88 for servant leadership, ………… WHAT TO 47 CONCLUDE ?????? Rule-Of-Thumb The practice of viewing rWG in the 0.70’s and higher as representing acceptable convergence is widespread. For example: Zohar (2000) cited rWG values in the .70’s and mid .80’s as proof that judgments “were sufficiently homogeneous for within group aggregation” 48 Benchmarking rWG Interrater Agreement Indices: Let’s Drop the .70 Rule-OfThumb Paper presented in the Annual Conference of the Society for Industrial and Organizational Psychology Chicago April 2004 R.J. Harvey and E. Hollander 49 “It is puzzling why many researchers and practitioners continue to rely on arbitrary rules-of-thumb to interpret rWG, especially the popular rule-ofthumb stating that rWG ≥ 0.70 denotes acceptable agreement”….. 50 “The justification of the rule rests largely on the argument that some researchers ( e.g. James et al., 1984) viewed rater agreement as being similar to reliability, reliabilities as low as .7 are useful ( e.g. Nunnaly,1978) , therefore rWG ≥ 0.7 implies interrater reliability”….. 51 There is little empirical basis for a .70 cutoff and few studies have attempted to determine how various values equate with “real world” levels of interrater agreement 52 The sources of four commonly reported cutoff criteria Lance, Butts, Michels (2006) ORM 1) GFI>.9 Indicates well fitting SEM’s 2) Reliability of .7 or higher is acceptable 3) rWG ‘s >.7 justify aggregation of individual responses to group-level measures 4) Keep the number of factors whose eigenvalues are greater than 1. 53 Rule-Of-Thumb A reviewer … “ I believe the phrase has its origins in describing the size of the stick appropriate for beating one’s wife. A stick was to be no larger in diameter than the man’s thumb….. Thus, use of this phrase might be offensive to some of the readership”… 54 Rule-Of-Thumb Feminists often make that claim that the “rule of thumb” used to mean that it was legal to beat your wife with a rod, so long as that rod was no thicker than the husband’s thumb. But, it turns out to be an excellent example of what may be called fiction…. 55 Rule-Of-Thumb From carpentry: The length from the tip of one’s thumb to the first knuckle was used as an approximation for one inch As such, we apologize to readers who may be offended by the reference to “rule of thumb” but remind them of the mythology surrounding its interpretation. 56 Statistical Tests Test the null hypothesis of no agreement . Dunlop et al. (JAP, 2003) Provided a table of rWG “critical values”. Under the null hypothesis of uniform null ( J=1, one item, different A values of the Likert scale, and different number of judges ) 57 Critical Values of the rWG statistic at the 5% Levels of statistical Significance A 2 3 5 11 n 58 4 1.0 5 .85 .78 8 .61 .59 10 .53 .49 Dunlop et al (2003)…. “Statistical tests of rWG are useful if one’s objective is to determine if any nonzero agreement exists ; although useful, this reflects a qualitatively different goal from determining if reasonable consensus exists for a group to aggregate individual level data to the group level of analysis “ …… 59 Alternative index AD Proposed by Burke, Finkelstein & Dusig Organizational Research Methods, 1999) AD M( j) 1 n | X jk X j | n k 1 ADM ( J ) = 60 ADM ( j ) J Gonzalez-Roma,V. ,Peiro,J.M.,&Tordera,N. (2002). JAP “ The ADM(J) index has several advantages compared with the James, Demaree and Wolf ( 1984) interrater agreement index rwg, see Burke et al. (1999)”….. 61 The ADM(J) index does not require modeling the null response distribution. It only requires an a priori specification of a null response range of interrater agreement. Second, the index provides estimates of interrater agreement in the metric of the original response range. 62 We followed Burke and colleagues’(1999) specification of using a null response range equal to or less than 1 when the response scale is a Likert-type 5 point scale. This value is consistent with our judgement that any two contiguous scale points are somewhat similar for the 5-point Likert-type scales used in the present study 63 …..Organizational commitment Measured by three items ( J=3). Respondents answered using a 5-point scale (A=5) The mean ADM(J) was 0.68 ( SD=0.32) and the ICC(1) was .22. The one –way ANOVA result , F(196,441)=1.7,p<.01, Suggests an adequate between differentiation and supports the validity of the aggregate organizational commitment measure. 64 Statistical Tests Test the null hypothesis of no agreement . Dunlop et al. (JAP, 2003) provided a table of AD “critical values”. Under the null hypothesis of uniform null ( One item, different A values of the Likert scale, and different number of judges ) 65 Critical Values of the AD statistic at the 5% Levels of statistical Significance A n 4 66 2 3 5 11 0.0 .75 5 .40 1.04 8 .69 1.53 10 .74 1.70 Criticism vs Reality • Citations and applications of rWG(J) in more than 450 papers 67 So far, the index rWG(J) has been much more frequently used than ADM(J) . We performed a systematic literature search of organizational multilevel studies, that were published during the years 2000-2006 (ADM(J) was introduced in 1999). Among the 41 papers that included justification of the aggregation from individual to group level, there were 40 (98%) that used the index rWG(J) and only 2 (5%) used the index ADM(J) . One study used both indices. 68 Statistical properties of rWG(J) • Cohen, A., Doveh, E., & Eick, U. (2001). Statistical properties of the rWG(J index of agreement. Psychological Methods, 6, 297310. Studied the sampling distribution of rWG(J) under the null hypothesis 69 Simulations to Study the Sampling Distribution Depends on • J( number of items in the scale) • A Likert scale • n group size • The null variance 2 • Correlation structure between items in 2 2 the scale J[1 ( s / )] rWG ( J ) J[1 ( s 2 / 2 )] s 2 / 2 70 Simulations to Study the Sampling Distribution “Easy” for a single item rWG Sx 1 2 Simulate data for a discrete “null” distribution 71 2 Dependent vs Independent Display of E (rWG(J)) n=3,10,100 corresponding to small,medium,large group sizes J=6,10 Data either uniform or slight skew A=5 Independent or CS ( Compound Symmetry) ρ=0.6 72 Compound Symmetry Structure CS 2 2 2 V33 73 2 2 2 2 2 2 Uniform data , uniform null “Error of first kind” Skew data uniform null “power” 74 CS =0.6 75 Testing Agreement for Multi-item Scales with the Indices rWG(J) and ADM(J) Ayala Cohen ,Etti Doveh & Inbal Shani Organizational Research Methods (2007) 76 Table of .95 RWG Percentile CS correlation structure n Group size 77 J=5 A=5 rho=0 rho= .3 rho= .4 3 0.87 0.90 0.90 5 0.74 0.77 0.78 6 0.70 0.73 0.75 7 0.69 0.70 0.72 8 0.66 0.66 0.67 15 0.52 0.56 0.58 20 0.49 0.50 0.53 Simulation • Software available: Simulate the null distribution of the index for a given n, A, J, k and the correlation structure among the items. 78 Example Bliese et al.'s (2002) sample of 2042 soldiers in 49 U.S. Army Companies. The companies ranged in size from n=10 to n=99. Task significance is a three-item scale ( J=3) assessing a sense of task significance during the deployment (A=5). 79 SimulationSample based values percentiles 80 Rejection area 1=in 0=out size rwg AD .95 rwg .05 AD rwg AD 10 .00 .97 .72 .63 0 0 20 .67 .69 .61 .72 1 1 37 .44 .78 .48 .80 0 1 41 .46 .81 .47 .80 0 0 45 .00 .93 .45 .81 0 0 53 .00 .99 .43 .82 0 0 58 .39 .76 .43 .83 0 1 68 .16 .91 .40 .85 0 0 85 .51 .80 .37 .86 1 1 99 .29 .87 .34 .88 0 1 Inference on the Ensemble • ICC summarizes the whole structure, assuming homogeneity of within variances. • Agreement indices are derived for each group separately. • How to infer on the ensemble? Analogous to regression analysis: F vs individual t tests 81 Based on the rWG(J) tests, significance was obtained for 5 companies, while based on ADM(J) it was obtained for 6 companies. Under the null hypothesis, the probability of obtaining at least 5 (out of 49 independent tests), with a significance level α=0.05, is 0.097 and it is 0.035 for obtaining at least 6. Thus, if we allow a global significance level of no more than 0.05, we cannot reject the null hypothesis based on rWG(J) , but will reject based on ADM(J). 82 Ensemble of groups(RMNET) What shall we do with groups that fail the threshold? 1) Toss them out because something is “wrong” with these groups. The logic is that if they do not agree, then the construct has not solidified with them, may be they are not a collective, they are distinct subgroups…. 83 Ensemble of groups(RMNET) 2) KEEP them… If on average we get reasonable agreement across groups on a construct, it justifies aggregation for all. ( …CFA: We index everyone, even if some people may have answered in a way which is inconsistent across items, but we do not drop them..) 84 Open Questions • Extension of slight skew to A>5 • Power Analysis • Comparison of degree of agreements , non-null cases 85
© Copyright 2024