585192
research-article2015
RAP0010.1177/2053168015585192Research & PoliticsZigerell
Research Article
Is the gender citation gap in international
relations driven by elite papers?
Research and Politics
April-June 2015: 1–7
© The Author(s) 2015
DOI: 10.1177/2053168015585192
rap.sagepub.com
LJ Zigerell
Abstract
This paper reanalyses data from a recent widely-discussed study reporting that female-authored papers published in top
international relations journals received fewer citations than equivalent male-authored papers. The reanalysis indicated
that the gender citation gap is largely limited to elite papers, defined either as papers in the right tail of the citation
distribution or as papers published in the most familiar journals. Results suggest that the original study’s recommendation
to consider the gender citation gap in promotion and review requires more data and a better understanding of the
factors that influence whether a paper enters the discipline’s elite.
Keywords
Gender, bias, citations
Introduction
Researchers often have enough latitude in how data are collected, coded, analysed, and reported to influence inferences about the presence, direction, and size of an effect
(Simmons et al., 2011). Gelman and Loken (2013) offered
a metaphor for this phenomenon, based on the Jorge Luis
Borges story, ‘The Garden of Forking Paths’; the path from
data to an inference can be imagined as a set of forking
paths: path A leads to inference A, but path B might lead to
inference A or to inference B. This forking paths metaphor
clarifies two methods to eliminate researcher latitude: the
first method is to publicly declare an intention to follow a
particular path before data collection, to address scepticism
that research design choices were based on a post hoc analysis of the data; and the second method is to demonstrate
that the same inference is produced from all reasonable
paths that might have been chosen before data collection, to
address scepticism that inferences might differ with a different set of reasonable research design choices (see
Gelman and Loken, 2013: 13–15). This second method is
reflected in the Walter (2013) assertion regarding the data
analysed in the Maliniak et al. (2013) paper, ‘The Gender
Citation Gap in International Relations’:
This paper has garnered a lot of press…, not because it’s telling
us something we hadn’t already suspected but because the
data are incontrovertible. Crunch the numbers in different
ways and the results are always the same: articles written by
women in IR are cited less than men, all else equal [emphasis
added].
Statements such as these must be accepted on faith in the
absence of data; however, recent movement in political science toward journal-required and researcher-volunteered
posting of data and code permit other researchers to reanalyse data to assess whether inferences have been based on a
representative set of reasonable research designs (see
Bueno de Mesquita et al., 2003; King, 1995; Lupia and
Elman, 2014). Such a reanalysis is reported below regarding the claim that the Maliniak et al. (2013) data always
produce the inference that female-authored international
relations papers are cited fewer times than equivalent maleauthored papers.
The reanalysis provided evidence that any gender citation gap in international relations is largely limited to
papers in the right tail of the citation distribution and, for
Illinois State University, USA
Corresponding author:
LJ Zigerell, Illinois State University, Schroeder Hall 401, Normal IL
61790-4540, USA.
Email: [email protected]
Creative Commons CC-BY: This article is distributed under the terms of the Creative Commons Attribution 3.0 License (http://
www.creativecommons.org/licenses/by/3.0/) which permits any use, reproduction and distribution of the work without further
permission provided the original work is attributed as specified on the SAGE and Open Access pages (http://www.uk.sagepub.com/aboutus/
openaccess.htm).
Downloaded from by guest on July 7, 2015
2
Research and Politics
more recent papers, limited to the three most familiar political science journals. This finding is important for at least
two reasons: first, any remediation to address the gender
citation gap in international relations should account for the
finding that there is little if any gender gap in the bottom
90% of cited papers; and second, models presented in
Maliniak et al. (2013) and in the reanalysis contained no
controls that can explain why some papers fall into the right
tail of the citation distribution, so it is an open question
whether a gender gap caused by papers in the right tail of
the citation distribution is due to sex bias in the citation of
highly-cited papers or due only to controls that are absent
from the model.
Is the gender citation gap in
international relations a function of
model specification?
Table 2 of Maliniak et al. (2013) reported results from
several models predicting citation counts to international
relations papers, with models successively adding one or
more controls. The final model in Table 2 is the full
Maliniak et al. (2013) model that contained all controls
from the previous models, with results indicating that
sample female-authored papers received 86% of the citations received by sample male-authored papers, holding
other model variables constant; the two-tailed p-value and
95% confidence interval were 0.093 and [0.72, 1.02],
which suggests a large (0.72) bias against female-authored
papers at one end of plausible estimates and a small (1.02)
bias against male-authored papers at the other end of plausible estimates (for guidelines on interpreting confidence
intervals, see Cumming and Finch, 2005: 174–175).
The full Maliniak et al. (2013) model contained theoretically relevant controls reflecting the expectation that papers
with deeper and wider exposure will receive more citations.
The full Maliniak et al. (2013) model accounted for deeper
exposure over time with controls for paper age and the
square of paper age, reflecting the expectation that older
papers will receive more citations because older papers
have had more opportunity to be cited. The full Maliniak
et al. (2013) model accounted for wider exposure with controls for R1 author status, tenured author status, coauthored
papers, and the journal that published the paper, reflecting
the expectation that potential citers are more likely to be
more aware of papers published by researchers at R1 institutions, published by tenured researchers, published by
multiple researchers, and published in more familiar journals. Reflecting the expectation that some issue areas or
methodological approaches might have a higher opportunity for citation because a larger pool of researchers study a
particular area or use a particular approach, the full Maliniak
et al. (2013) model contained dichotomous controls for the
type of research reported: positivist research, research with
ideational factors, research with material variables, research
in particular issue areas, and research with particular methodological approaches.
However, the full Maliniak et al. (2013) model included
some cases, controls, and restrictions that did not appear
theoretically necessary: the full Maliniak et al. (2013)
model excluded data collected on papers from 2007,
included year controls, included a tenured female control,
and included papers coauthored by men and women, hereafter referred to as coed papers. To assess the robustness of
the full Maliniak et al. 2013 model, the full Maliniak et al.
(2013) model has been re-estimated, removing one of these
controls or a set of related controls. Figure 1 presents point
estimates and 95% confidence intervals for several modified models. Model 2 is the full Maliniak et al. (2013)
model but with papers published in 2007 included in the
analysis, because there does not appear to be a reason to
exclude these data.1 Model 3 is the full Maliniak et al.
(2013) model but without year controls, because there did
not appear to be a reason to expect citation counts to differ
by publication year, given that paper age and the square of
paper age were already included as controls. Model 4 is the
full Maliniak et al. (2013) model but without the control
indicating whether a tenured female was an author on a
paper, because there did not appear to be a reason to expect
tenured status to have a different effect on citation counts
for men and women, and because the full Maliniak et al.
(2013) model included no corresponding control to differentiate the effect of R1 status for men and women. Model 5
is the full Maliniak et al. (2013) model but without coed
papers, because the critical comparison for assessing sex
bias in citations is the comparison of papers with only
female authors to papers with only male authors. Model 6
is the full Maliniak et al. (2013) model with controls for
whether the paper had a restricted temporal focus or a
restricted geographical focus, based on the expectation that
a paper should receive fewer citations if it restricts its focus
to a particular time or a particular non-US region. Model 7
is the full Maliniak et al. (2013) model but with all of the
aforementioned changes: inclusion of sample papers from
2007, removal of year controls, removal of the tenured
female control, removal of coed papers, and inclusion of
the restricted temporal and geographical focus controls.
Models discussed above used only data available in the
Maliniak et al. (2013) dataset. Models 8 and 9 include new
controls drawn from the Web of Science and JStor; data
from the available institutional Web of Science database
were limited to papers published after 1987, so models with
new controls are based on a smaller number of observations
and on more recent cases, compared to the full Maliniak
et al. (2013) models. These new models reflect edits to the
Maliniak et al. (2013) dataset. First, twelve instances of
duplicate papers were discovered and corrected for. Second,
the dataset contained 204 papers intended for the analysis
but without a value for the number of citations to that paper;
if one of these papers was found in the Web of Science
Downloaded from by guest on July 7, 2015
3
Zigerell
Figure 1. Estimates of the gender gap in citations to international relations papers.
Note: The figure depicts point estimates and 95% confidence intervals for the gender gap in citations to international relations papers, based on a
negative binomial regression across 15 models. Estimates less than 1.0 are estimates in which papers authored by only women receive fewer citations than papers authored by only men, holding constant all other model controls. An estimate of 0.90 is interpreted as papers authored by only
women receiving 90% of the citations received by papers authored by only men, holding constant all other model controls.
database, the missing citation count for that paper was
coded with the citation count excluding citations from
books and book chapters (following Maliniak et al., 2013:
896); all but 86 of the post-1987 papers for the twelve sample journals were found in the Web of Science and were
coded.2 Third, publication type was coded for the 2,052
post-1987 publications found in the Web of Science: 1,715
papers, 208 reviews, 53 editorial material, 39 book reviews,
25 letters, 11 notes, and 1 reprint.
The first new control was paper page count, based on the
expectation that longer papers contain more material and
are thus more likely to contain material that others find
worth citing (see Østby et al., 2013). The second new control indicated the number of references that the paper made
to other sources, based on the expectation that a larger number of references reflects the degree to which a paper is
related to other research and thus reflects the potential of
the paper to be cited (see Wang et al., 2011). Table 1 presents selected statistics for the new controls, indicating that
men were less likely than women to author restricted temporal or geographical focused papers but were more likely
to author papers that landed in the right tail of references to
other sources.
Differences between the new models and the full Maliniak
et al. (2013) model are that the new models: include papers
published in 2007; do not include year controls; do not
include a tenured female control; remove coed papers;
include the restricted temporal and geographical focus controls; include a control for paper page count; include a control for the number of references made to other sources;
restrict the dataset to papers published after 1987; and restrict
the dataset to publications coded as a paper, or as a paper or
a review. Table 2 reports exponentiated coefficients and
p-values for key variables for Model 8, which predicts the
number of citations received by publications coded as a
paper, and Model 9, which predicts the number of citations
received by publications coded as a paper or a review.3
Respective point estimates for the gender citation gap are
93% and 97%, indicating gaps of 7% and 3%.
Is the gender citation gap in
international relations a function of
highly-cited papers?
Maliniak et al. (2013: 893) and Simmons (2013) noted that
the gender citation gap disfavouring female-authored
papers appears in the mean but not in the median: the
median citation count to female-authored papers was equal
to or larger than the median citation count to male-authored
papers, in the 1980s, the 1990s and the 2000s. Maliniak
et al. (2013) proposed self-citation and citation cartels as
potential explanations for the citation gap, but the fact that
men are more likely to cite themselves and the possibility
that men are more likely to participate in citation cartels
cannot explain the higher median for papers with only
female authors. Moreover, the higher mean for maleauthored papers and the higher median for female-authored
papers give rise to the possibility that a sex difference in the
citation distribution could influence inferences about sex
differences in citations. The sample for the full Maliniak
et al. (2013) model contained 2,057 papers with only male
authors and 262 papers with only female authors, so allfemale papers accounted for 11.3% of the total sample of
non-coed papers; however, the percentage of all-female
Downloaded from by guest on July 7, 2015
4
Research and Politics
papers decreases further into the right tail of citations: as
displayed in Figure 2, all-female papers account for 8.7%
of non-coed papers above the 90th percentile; 6.1% of noncoed papers above the 95th percentile; and 4.5% of papers
above the 99th percentile.
Maliniak et al. (2013) discussed the possibility that the
gender citation gap might be driven by outliers:
similar. The coefﬁcient on ALL FEMALE is always negative,
with point estimates very close across speciﬁcations. The
consistency of the effect across many speciﬁcations lends
support to the general ﬁnding, as well as the result being a more
general phenomenon across the distribution of articles. Of
course, any subset of articles that garners fewer citations will
decrease the chance of a gap being statistically signiﬁcant thanks
to information loss (905).
Are these ﬁndings driven by a few highly cited, male-authored
articles? When we exclude articles with three standard deviations
above the mean of cites or more, our results are substantively
The mean and standard deviation of citations were
respectively 25 and 50 for papers in the sample for the full
Maliniak et al. 2013 model. Model 10 is the original full
Maliniak et al. (2013) model with the analysis restricted to
papers that received 175 or fewer citations, which eliminated 1.3% of the sample papers. Models 11 to 13 are the
original full Maliniak et al. (2013) model with the sample
restricted to the bottom 98%, bottom 95%, and bottom 90%
of papers based on total citations. Models 14 and 15 respectively represent Models 7 and 13, but excluding the top 2%
and 5% of non-coed cited papers.
Maliniak et al. (2013) correctly noted that removing
cases in the right tail of the citation distribution decreases
the chance of detecting a gender gap at conventional levels
of statistical significance; however, the important change to
note is not in the p-values but in the point estimates, rising
from 0.86 for the whole sample to 0.99 for the bottom 90%
of papers. The reduction of the gender gap when highlycited papers are removed suggests that any sex difference
– and thus any sex bias – is limited or largely limited to
papers in the right tail of citations. But the reduction of the
Table 1. Selected summary statistics for the new controls.
Percentage of papers
Percentage of reviews
Percentage with a restricted temporal
focus, papers only
Percentage with a restricted
geographical focus, papers only
Mean and standard deviation for page
count, papers only
Mean and standard deviation for
references, papers only
Men
Women
83
10
19
82
12
21
51
58
24.5, 9.5
25.3, 9.0
56.8, 31.5
56.8, 25.5
Note: Samples for the percentage of papers and reviews are the 1,612
male-authored publications with a coded publication type and the 237
female-authored publications with a coded publication type. Samples for
the statistics restricted to papers are the 1,337 male-authored publications coded as a paper and the 195 female-authored publications coded
as a paper.
Table 2. Predicting citations to international relations papers.
Model 8
Papers only
Model 9
Papers and reviews only
Incidence
Rate ratio (p-value)
Incidence
Rate ratio (p-value)
Paper with only female authors
At least one author tenured
Multiple authors
At least one author from an R1 school
Paper had a restricted temporal focus
Paper had a restricted geographical focus
Publication type: Review
Paper page count
Number of references to other publications
Number of observations
0.93 (0.405)
1.14 (0.055)
1.07 (0.340)
1.14 (0.061)
0.76 (<0.001)
0.53 (<0.001)
—
1.03 (<0.001)
1.006 (<0.001)
1532
0.97 (0.720)
1.15 (0.028)
1.06 (0.394)
1.13 (0.066)
0.76 (<0.001)
0.57 (<0.001)
0.66 (<0.001)
1.03 (<0.001)
1.005 (<0.001)
1728
Note: Numerical cells indicate exponentiated coefficients and p-values from a negative binomial regression predicting the number of citations a
publication has received, from publication to March 2013, for papers and/or reviews published in sample issues of the dataset journals from 1988
to 2007. The exponentiated coefficients can be interpreted as incidence rate ratios; the estimate of 0.93 is interpreted as papers authored by only
women receiving 93% of the citations received by papers authored by only men, holding constant all other model controls. Bold font indicates coefficients for which p<0.05 for a two-tailed test. The table omits model results for: paper age; paper age squared; positivist research; research with
material variables; research with ideational factors; the set of paradigm variables; the set of issue variables; the set of methodology variables; the
set of journal variables; and the constant. Analyses were restricted to papers in which the set of authors was all men or all women. The excluded
category for publication type in Model 9 is a paper.
Downloaded from by guest on July 7, 2015
5
Zigerell
Figure 2. The gender citation gap for highly-cited international relations papers.
Note: The figure depicts the number of citations to highly-cited sample papers published between 1980 and 2006. Each pink dot above the grey line
represents a single paper authored by only women, and each light blue dot beneath the grey line represents a single paper authored by only men.
gender gap when highly-cited papers are not included in the
analysis presents a problem for causal inference using the
full Maliniak et al. (2013) model because that model lacks
a control that can explain outliers in the right tail of citations; even the new controls for page count and number of
references are unable to explain why some papers receive
an outlier number of citations, such as the 1,084 citations
received by the most highly-cited sample paper; it is thus
unclear to what extent an observed gender citation gap
reflects sex bias and to what extent an observed gender citation gap reflects factors absent from the models.
Is the gender citation gap in
international relations a function of
familiar journals?
Similar research has failed to detect at conventional levels
of statistical significance a gender citation gap to papers
published in the Journal of Peace Research (Østby et al.,
2013). It is thus possible that the gender citation gap is a
function of the type or familiarity of the journal that a paper
is published in. To test this, controls for individual journals
were replaced in model 8 by familiarity scores for the
twelve dataset journals based on Garand and Giles (2003):
the interaction of journal familiarity score and the female
authorship variable was substantively and statistically significant (0.47, p=0.013). Removing the journal familiarity
score in model 8 and restricting the analysis to the three
most familiar journals (APSR, AJPS and JOP) returned a
point estimate for the gender citation gap of 0.58 (p=0.012,
n=165) compared to the gap of 0.95 for the nine residual
journals (p=0.562, n=1367), providing corroborating evidence that the gender citation gap in international relations
is only an elite phenomenon.4
Network analysis
Maliniak et al. (2013) investigated the relative influence of
male- and female-authored papers in international relations, by modelling ‘authority’ scores for each paper, which
ranged from 0 to 1 and reflected the idea that ‘an article that
is cited by many widely cited articles will have a higher
authority score than an article cited by many articles that
themselves are only rarely cited’ (907). Figure 3 presents
results for the models reported above, but replacing the
citation count-dependent variable with the authority scoredependent variable used in Table 4 of Maliniak et al. (2013).
Point estimates fluctuate, but results generally support the
inference that compared to citations to papers authored by
only men, citations to papers authored by only women tend
to come from papers that receive fewer citations and are
thus less central to the international relations literature.
Conclusion
Maliniak et al. (2013) reported evidence that international
relations papers authored by women receive fewer citations
than international relations papers authored by men, even
controlling for many factors that might be proposed as alternative explanations. Maliniak et al. (2013) suggested that:
‘The most important recommendation for departments is to
take the existence of gender bias seriously when evaluating
Downloaded from by guest on July 7, 2015
6
Research and Politics
Figure 3. Estimates of the gender gap in the influence of international relations paper citations.
Note: The figure depicts point estimates and 95% confidence intervals for the gender gap in highest authority score for international relations
papers, based on a least squares regression across 15 models. Estimates less than 0 are estimates in which papers authored by only women have a
lower authority score than papers authored by only men, holding constant all other model controls.
female scholars for promotion and review…We believe it is
better to work with an existing indicator whose bias is
known, than to switch to one that is assumed to be unbiased
but is not’ (919–920). The most obvious interpretation of
this recommendation is that promotion and review standards
should be lower for female international relations researchers than for male international relations researchers, at least
with regard to citations.
Different promotion and review standards for men and
women might be warranted with strong evidence of homogeneous sex bias in citation counts, but the data analysed in
Maliniak et al. (2013) do not appear to provide enough evidence to justify blanket changes to promotion and review
practices, in several senses: first, 95% confidence intervals
for sex bias were nearly 30 percentage points wide and in
most cases contained an estimate of zero bias; second, the
sex difference in citation counts appears restricted to the
right tail of citations and to the most familiar journals, so
any advantage for male researchers and any disadvantage
for female researchers might not be evenly distributed
among male and female researchers; and third, even
restricting the analysis to the right tail of citations and the
most familiar journals, it is unclear whether the estimated
gender citation gap is driven by bias or by unmodelled factors, given that the models contain no controls that can
explain why some papers land in the right tail of the citation
distribution or are placed in the most familiar journals.5
It is important to look for bias and to adopt policies to
eliminate or account for any bias that is detected. Reanalysis
of the Maliniak et al. (2013) data indicated that the inference of a gender citation gap was not always statistically
significant, but 95% confidence intervals always contained
a gender gap that was substantively important. However,
use of the Maliniak et al. (2013) data to inform promotion
or review practices should ensure that the estimates are relevant for the researcher under review, and are not unduly
influenced by bias that might advantage or disadvantage
only those international relations papers that land in the
right tail of the citation distribution or are placed in the discipline’s most familiar journals.
Acknowledgements
The author thanks the editor and anonymous reviewers for helpful
comments, thanks International Organization for helping to make
the reproduction data and code available, and thanks Daniel Maliniak
and Ryan Powers for responding to inquiries regarding their study.
Declaration of Conflicting Interest
The authors declare that there is no conflict of interest.
Funding
This research received no specific grant from any funding agency
in the public, commercial, or not-for-profit sectors.
Supplementary material
The online appendix is available at: http://rap.sagepub.com/
content/by/supplemental-data
The replication files are available at: http://dx.doi.org/10.7910/
DVN/29945
Notes
1. The Maliniak et al. (2013) dataset contained 140 papers
published in 2007 from issues 1 and 2 of World Politics and
issues 1 and 3 of the remaining journals that were coded in_
analysis==1 with a non-missing value for the citation count
variable. See the supplemental material.
Downloaded from by guest on July 7, 2015
7
Zigerell
2. Sixteen of the 86 cases were from the first two volumes of
the European Journal of International Relations, and 70
cases were from the first five volumes of Security Studies;
three other cases missing a citation count were from a journal
that was not included in the set of twelve sample journals.
3. The analysis is restricted to papers and reviews because
reference data appeared less reliable for some of the lessfrequent publication types. See the supplemental material.
4. The full Maliniak et al. (2013) model was re-estimated with a
control for the three most familiar journals. For the nine residual journals, the resulting point estimate, p-value, and 95%
confidence interval for female authorship was: 0.90, p=0.279,
[0.75, 1.09]. For the three most familiar journals, the resulting point estimate, p-value, and 95% confidence intervals for
female authorship was: 0.90, p=0.716, [0.51, 1.58]. There was
more of a difference in estimates when, as in the new models,
cases were restricted to publications between 1988 and 2007:
0.95 as the point estimate for the nine residual journals, and
0.67 as the point estimate for the three most familiar journals.
5. The estimate of a sex difference in citations is neither a
floor nor a ceiling on the size of sex bias. Interpreting the
sex difference estimate as the estimate of sex bias would
be an overestimate of sex bias if the model lacks relevant
controls that would reduce the estimate; but interpreting the
sex difference estimate as the estimate of sex bias would be
an underestimate of sex bias if included model controls capture some of the indirect effect of sex bias, such as if part of
the reduced citation count for restricted geographical focus
papers is because women are more likely to author restricted
geographical focus papers.
References
Bueno de Mesquita B et al. (2003) Symposium on replication in international studies research. International Studies
Perspectives 4(1): 72–107.
Cumming G and Finch S (2005) Inference by eye: Confidence
intervals and how to read pictures of data. American
Psychologist 60(2): 170–180.
Garand JC and Giles MW (2003) Journals in the discipline: A
report on a new survey of American political scientists.
Political Science and Politics 36(2): 293–308.
Gelman A and Loken E (2013) The garden of forking paths: Why
multiple comparisons can be a problem, even when there
is no ‘fishing expedition’ or ‘p-hacking’ and the research
hypothesis was posited ahead of time. Available at: http://
www.stat.columbia.edu/~gelman/research/unpublished/p_
hacking.pdf (accessed 30 June 2014).
King G (1995) Replication, replication. PS: Political Science &
Politics 28(3): 444–452.
Lupia A and Elman C (2014) Openness in political science: Data
access and research transparency. PS: Political Science &
Politics 47(1): 19–42.
Maliniak D et al. (2011) International relations in the US
Academy. International Studies Quarterly 55(2): 437–464.
Maliniak D et al. (2013) The gender citation gap in international
relations. International Organization 67(4): 889–922.
Østby G et al. (2013) Gender gap or gender bias in peace research?
Publication patterns and citation rates for Journal of Peace
Research, 1983–2008. International Studies Perspectives
14(4): 493–506.
Simmons B (2013) The gender citations gap: A glass half-full
perspective. The Monkey Cage. The Washington Post. 4
October. Available at: http://www.washingtonpost.com/
blogs/monkey-cage/wp/2013/10/04/the-gender-citationsgap-a-glass-half-full-perspective/ (accessed 29 April 2015).
Simmons JP et al. (2011) False-positive psychology: Undisclosed
flexibility in data collection and analysis allows presenting anything as significant. Psychological Science 22(11):
1359–1366.
Walter BF (2013) How to reduce the gender gap in one (relatively) easy step. The Monkey Cage. The Washington Post.
1 October. Available at: http://www.washingtonpost.com/
blogs/monkey-cage/wp/2013/10/01/how-to-reduce-thegender-gap-in-one-relatively-easy-step/ (accessed 29 April
2015).
Wang M et al. (2011) Mining typical features for highly cited
papers. Scientometrics 87(3): 695–706.
Downloaded from by guest on July 7, 2015