1. art and entertainment
2. visual art and design
3. art and craft supplies

Can Citizens Be Framed? How Information, Not Emphasis, Changes Opinions
Thomas J. Leeper & Rune Slothuus
Department of Political Science
Aarhus University
Bartholins Alle 7
8000 Aarhus C, Denmark
E-mail: [email protected] / [email protected]
Web: www.thomasleeper.com / www.ps.au.dk/en/slothuus
March 20, 2015
Abstract
Framing refers to a communicator’s presentation of a political issue and is widely seen as an
effective way for elites to influence citizens’ opinions and behavior. The attractiveness of
framing as a political strategy stems from the perception that it is relatively easy to focus people’s
attention on an aspect of the issue, beneficial to the communicator. We present three studies that
question the power of framing to shape public opinion. Existing studies confound framing with
the provision of new information and argumentation. Our novel experimental design allows us—
for the first time—to disentangle the separate effects of framing and information. Despite
adjusting the design of each of our experiments in favor of the framing hypothesis, we find strong
and consistent effects of information on opinions, but we find no effects of framing. These results
have serious normative and practical implications for understanding elite influence on public
opinion.
A basic premise of representative democracy is that citizens can form and express their
preferences for which public policies they want government to pursue and that elected
representatives, in turn, respond to these public preferences (e.g., Dahl 1971; 1989: 109-112). Yet
there might be a tension between the democratic ideal and political reality, because, as Disch
(2011: 101) notes, “citizens’ capacity to form preferences depends on the self-interested
communications of elites.” Indeed, fifty years of research on the influence of mass
communication on opinion formation has demonstrated that politicians and other elites can
powerfully shape citizens’ political preferences (see Kinder 2003).
At the forefront of this literature is the repeated finding that “framing”—understood to
mean a communicator’s selective presentation and interpretation of an issue or event—can have a
sizeable impact on citizens’ opinions. For example, if a hate group rally is framed as a question of
free speech, most citizens will tend to support allowing the rally to be held. In contrast, if the
rally is framed as a threat to public order, citizens will tend to oppose allowing the rally (Nelson,
Clawson, and Oxley 1997). Kinder (2003: 359) summarizes the framing concept clearly: “frames
supply no new information. Rather, by offering a particular perspective, frames organize—or
better, reorganize—information that citizens already have in mind. Frames suggest how politics
should be thought about, encouraging citizens to understand events and issues in particular
ways.”
However, despite a “dramatic growth in framing studies” (Weaver 2007: 143-144), it is far
from clear what this body of research tells us about how easily elites can “frame” public opinion.
The reason for this is a mismatch between the theoretical definition of framing as a
communicator or message recipients’ emphasis on subsets of issue-relevant considerations and
the empirical paradigm for studying framing effects. A defining feature of most, if not all,
existing framing studies is that they confound testing the impact of the framing of an issue with
2
the impact of providing substantive information and arguments (i.e., traditional argument-based
persuasion; Scheufele and Iyengar 2015: 2; de Vreese and Lecheler 2012: 299). Experimental
studies of “framing effects” typically involve messages that both frame an issue along a particular
dimension and supply information and arguments about the issue (e.g., Boudreau 2015; Chong
and Druckman 2007b; Druckman and Leeper 2012; Jerit 2009; Slothuus and de Vreese 2010;
Sniderman and Theriault 2004). The same problem extends to observational studies of framing
which assess frames in news coverage that also contains issue-relevant information (e.g.,
Hopkins 2014; Schemer, Wirth, and Matthes 2012; Slothuus 2010; Schuck, Vliegenthart, and de
Vreese 2015).1
The consequences of this mismatch between theory and empirical practice are far more
than semantic. Indeed, framing research has serious normative and practical implications for
understanding how interactions between elites and the public work. Evidence for framing effects
invites debate about the malleability of public opinion and the ability of citizens to participate
meaningfully in democratic politics: “framing effects suggest that distributions of public
preferences are arbitrary, and that political elites can manipulate popular preferences to serve
their own interests” (Chong and Druckman 2007a: 120). If all a politician must do to influence
citizens’ policy preferences is mention a dimension of the issue (or, in practice, stimulate media
to feature that chosen dimension), elite influence can be achieved with ease.2 In this instance, it
would be “highly rational for elites to pursue strategies based upon issue framing” because
“framing as a political strategy involves minimal costs, and it has the potential to provide sizable
1
Scheufele and Iyengar (2015) describe the framing literature as in a “state of conceptual confusion” where “any
attribute of information is treated as a frame and any response from the audience is deemed a framing effect. From
this perspective, framing cannot be distinguished from other forms of media or social influence such as agenda
setting, learning or persuasion” (also see Chong and Druckman 2007a: 115; de Vreese and Lecheler 2012: 299).
2
Some have interpreted framing effects as a deliberative process where citizens consciously weight alternative
considerations emphasized by frames (Brewer 2001; Druckman 2001; Nelson, Oxley, and Clawson 1997). However,
given the inability of previous framing studies to separate the effects of emphasis from providing substantive
information, this remains unclear.
3
benefits” (Jacoby 2000: 751). In essence, “It may indeed often be easier to change frames of
reference than the beliefs underlying one’s attitude” (Chong 2000: 118). This distinction mirrors
the long-standing debate about the size and type of media effects on citizens’ political thinking
and behavior. It is widely believed that media can influence public opinion only indirectly by
changing the importance of citizens’ existing beliefs through processes of framing, priming, and
agenda-setting whereas the media have limited capacity to provide information that will change
the content of those beliefs. Framing is seen as politically important precisely because direct
media influence is thought to be so difficult, but to date no study has distinguished the effects of
framing from information.
To clarify the effects of framing and information, we present a series of three experimental
studies (combined n=2,299) that enable us to tease apart the separate influences of framing and
issue-relevant information on political opinions. Our results show a strong and consistent effect
of information on opinions but we find no effect of framing. And these results are consistent
across the three studies despite our repeated attempts to carefully adjust the design in favor of the
framing hypothesis. Given the normative and practical implications of framing research, these
results invite us to reconsider our understanding of opinion formation, elite influence, and citizen
competence.
The Theory and (Limited) Evidence for Framing Effects
A frame is generally understood to be “an emphasis in salience of some aspects of a topic”
(de Vreese 2003: 27), it “suggests what the controversy is about, the essence of the issue”
(Gamson and Modigliani 1987: 143), and it stresses “specific elements or features of the broader
controversy, reducing a usually complex issue to one or two central aspects” (Nelson, Clawson,
and Oxley 1997: 568; see also Druckman 2001; Kinder 2003; de Vreese and Lecheler 2012).
Thus, a frame is typically thought of as a message that provides an interpretation of an issue or
4
policy by emphasizing which aspect of the issue is relevant for evaluating it, without the frame
itself providing any new substantive information about the issue (Price and Tewksbury 1997;
Nelson, Oxley, and Clawson 1997).
Even though framing effects are thought to work by merely highlighting and emphasizing
information and beliefs already present in a debate or the public’s minds, theorizing suggests
frames are powerfully impactful. The reason is that citizens often possess a mix of considerations
that could be used to form an opinion on an issue. These considerations might point in different
directions, each pushing the individual to support or oppose the issue (Chong 1993; Feldman and
Zaller 1992), thus “leaving citizens often confused and conflicted about where to stand… Frames
help to resolve this confusion by declaring which of the many considerations is relevant and
important, and which should be given less attention” (Nelson and Kinder 1996: 1058). Frames
are therefore thought to shift public opinion via “framing effects” where during “the course of
describing an issue or event, a speaker’s emphasis on a subset of potentially relevant
considerations causes individuals to focus on these considerations when constructing their
opinions” (Druckman and Nelson 2003: 730; also see Jacoby 2000: 750, 751).
More formally, the psychological theory underlying framing effects draws on the
expectancy-value model of attitude formation. In this model, an opinion towards an object (e.g., a
policy) reflects the weighted sum of a set of evaluative beliefs about that object. Thus, opinion =
Σ beliefi × weighti, where beliefi is the evaluative belief on dimension i and weighti is the
subjective weight or importance the individual attaches to that dimension (Chong and Druckman
2007a: 105-106; Nelson, Oxley, and Clawson 1997: 225-228). For example, an opinion towards a
social welfare policy might be the result of a positive belief that the policy will fight poverty (i.e.,
a reason to support the policy) and a negative belief that it will increase taxes (i.e., a reason to
oppose the policy). Depending on the relative weight or importance an individual attaches to each
5
of these beliefs, overall opinion towards the policy might be positive or negative. Thus the
strategic appeal of framing is that it requires comparatively less effort than following a strategy of
argument-based persuasion aimed at convincing the public to change their beliefs (e.g., that a tax
increase is worth paying for). It is thought to be far easier to remind citizens of particular preexisting beliefs when forming their opinion than it is to change those beliefs (e.g., Chong 2000:
118; Jacoby 2000: 751).
Despite the clarity of emphasis framing theory, there is a remarkable lack of clean
empirical testing for framing effects. In particular, empirical tests of framing rarely, if ever,
follow the strict definition of only changing the emphasis or importance of an issue dimension.
Instead, studies have typically assessed the impact of experimental manipulations of both
information (e.g., factual policy information and persuasive arguments) and emphasis.3 A
paradigmatic example is Druckman and Leeper’s (2012) study of opinion toward the Patriot Act,
where the policy was framed as either about “weakening the protection of citizens’ civil liberties”
or a way “to identify terrorist plots on American soil and to prevent attacks before they occur.” In
addition to emphasizing these alternative dimensions of the issue, the experimental manipulations
also presented study participants with different information about policy content—either that
under the Patriot Act, “the government has access to citizens’ confidential information from
telephone and e-mail communications” or, in the alternative presentation, that “the government
has more resources for counterterrorism, surveillance, border protection, and other security
policies.” This combination of framing and information leaves it an open question to what extent
opinions in the study were swayed by framing (that is, emphasizing civil liberties versus
3
Notable exceptions are Berinsky and Kinder’s (2006) study of journalistic storytelling and Druckman et al.’s (2010)
study of candidate evaluations.
6
emphasizing fighting terrorism) or the substantive information that was also provided along each
of those dimensions.
In another well-known experiment, the framing of a government spending proposal was
varied by either stating that it “means higher taxes” or gives people “a better chance of getting
ahead in life” (Sniderman and Theriault 2004). While these statements refer to different policy
dimensions, it is difficult to know if opinion changes were caused by the mere emphasis on these
dimensions or by potential changes in respondents’ underlying beliefs about the different
consequences of the proposed spending in each condition.4 Reviewing the literature as a whole, it
appears to be challenging to avoid supplying different arguments and additional information to
participants in different framing conditions (e.g., Aarøe 2011; Boudreau 2015; Chong and
Druckman 2007b; Druckman and Nelson 2003; Hartman and Weber 2009; Lecheler and de
Vreese 2013; Nelson, Clawson, and Oxley 1997; Slothuus 2008; Slothuus and de Vreese 2010).
If extant framing experiments show opinion change in response to political
communications, does it matter if that opinion change is due to information and arguments rather
than framing? We do not argue that the empirical findings of previous studies are invalid, but
rather that they entail a mismatch between theory and design. In light of the operational
confounding, interpreting these studies through the lens of framing theory is problematic given
that they do not test framing in the strict sense that “frames organize—or better, reorganize—
information that citizens already have in mind” (Kinder 2003: 359). Consequently, the valueadded of the framing concept itself is unclear from extant research because “we cannot be sure
that there is anything truly ‘unique’ about this phenomenon; that framing cannot be subsumed
4
This operational confounding of framing and information is further confused by the sometimes reference to frames
and arguments interchangeably (e.g., Chong and Druckman 2007b: 641; Druckman et al. 2013: 57; Jerit 2009: 412).
7
under some other generally understood concept, such as persuasion” (Nelson, Oxley, and
Clawson 1997: 223).
Moreover, the normative implications of the current framing literature is unclear. While
influence through emphasis alone (i.e., framing) need not imply that citizens are irrational or that
they fail to deliberate over political issues, opinion change without substantive information or
belief change means the public can be easily coopted by the slightest of rhetorical shifts (see
discussion by Druckman 2014: 474-475). A public that instead requires evidence and
argumentation before changing its views implies a democratic process more closely aligned to
popular conceptualizations of the democratic ideal of citizens forming “enlightened” preferences
(Dahl 1989: 108-112).
Properly evaluating framing requires disentangling the experimental confound. We
therefore lay out two distinct empirical alternatives about elite influence on opinions. One relates
to the effects of information provision or persuasive argumentation on opinions (i.e., changes in
the content of beliefs) and the other relates to the effects of emphasizing particular considerations
(i.e., changing the weights applied to different issue-relevant beliefs). We call the first the
information hypothesis, which expects that information and arguments will shape beliefs about
the policy and influence policy opinion in the direction implied by the evaluative content of the
information or argument (Chong and Druckman 2007a: 115-116; Kinder 2003: 359). In contrast,
the framing hypothesis expects the mere emphasis on a dimension or aspect of a policy will
influence the importance of beliefs and change opinion in the direction implied by the content
(positive or negative) of the evaluative beliefs emphasized (Nelson, Oxley, and Clawson 1997).
Testing these two hypotheses requires a novel paradigm for studying elite influence. It is of
course possible that both forms of communication may be influential; ours is the first study to
provide such a clean test of each.
8
A New Approach for Studying Framing Effects
To advance our knowledge of whether framing is a viable and effective political strategy to
influence public opinion distinct from providing new information, we propose a novel
methodological approach to operationalize and test issue framing in a form true to its theoretical
definition. We argue that to assess the value-added of the framing concept, we need to
independently evaluate the impact on opinions of mere emphasis on an issue dimension (i.e.,
framing) by separating it experimentally from the impact of policy information or
argumentation.5 Very little, if any, extant research does this.6
Our experimental design isolates the manipulation of beliefs and framing at two different
points in time (see Figure 1). In the first stage (t1), we manipulate information using the paradigm
of persuasion research. As Kinder (2003: 367) explains, “Political persuasion entails the supply of
arguments and evidence through which people are induced to change their minds about some
aspect of politics” (emphasis added; see Chong and Druckman 2007a: 115; Zaller 1992: 118).
We provide study participants with information and arguments on a policy issue that should lead
them to form positive or negative evaluative beliefs on the issue. Depending on the evaluative
content (or direction) of those beliefs, participants are expected according to the expectancyvalue model of attitudes to support or oppose the policy (given the change in the content of
beliefs in), hence testing the information hypothesis.
5
As an alternative strategy to avoid the confounds between framing and related concepts, Scheufele and Iyengar
(2015) propose to confine framing research to “equivalence” framing and visual manipulations. Equivalence
framing, however, is not how issues typically are framed in political discourse—“It is difficult to satisfy this
requirement of interchangeability of alternatives outside a narrow range of choices” (Sniderman and Theriault 2004:
135; see de Vreese 2003: 27)—and visual manipulations constitute a limited set of real-world communications.
6
Some might see question wording experiments as the closest empirical approximation of framing true to its name
(Schuman and Presser 1996). The concern with many question wording experiments, however, is that they either (1)
supply different information, as in a paradigmatic “framing” experiment, or (2) effectively ask survey respondents to
evaluate two distinct attitude objects, rather than framing a single issue in two distinct ways (e.g., the seminal
experiment asking about support for “assistance to the poor” versus “welfare” in fact asks about two entirely
different policies; Huber and Paris 2013).
9
In the second step of our experimental design (t2), we manipulate framing in a manner true
to its theoretical definition. Specifically, we manipulate the emphasis placed on alternative
dimensions of consideration in a political debate to adjust which dimension of the policy (i.e.,
which evaluative belief) study participants should most heavily rely upon when forming their
opinion. Because participants hold beliefs formed during t1, the manipulation of framing can be
implemented purely by emphasizing one dimension of the policy and without supplying any new
information or arguments. This way, we avoid the risk of further changing belief contents and
thus create a clean test of the framing hypothesis.7
Based on extant work, we expect both information provision and framing to be effective
communication strategies and we thus except to find support for both of our hypotheses. Yet it is
important to note that, given the problems with previous studies of framing effects discussed
above, we believe we are the first to conduct an empirical test of the framing hypothesis in its
pure form.8
Study 1: A Test of Pure Framing
We recruited participants for our survey experiment online using the Amazon Mechanical
Turk (MTurk) crowdsourcing platform and randomized respondents into experimental conditions
using Qualtrics. A number of recent studies indicate that while MTurk is not a representative
online panel, the pool of respondents is more diverse than traditional convenience samples
7
Importantly, this temporal separation of information supply and framing is critical for disentangling the information
hypothesis and the framing hypothesis. The information stage alone allows us to test the information hypothesis but
also serves to provide an exogenous shock to participants’ issue-relevant beliefs. If information (and thus beliefs)
were not manipulated, then framing effects may not manifest due to participants’ lack of relevant beliefs or beliefs
that are insufficiently heterogeneous to display framing effects. Framing theory requires an individual to have beliefs
in memory that can be emphasized, hence our choice to manipulate information before manipulating framing. For the
same reason, we opted to not separate our information and framing phases by a long time window where the effects
of information on beliefs might have decayed.
8
Perhaps the study most closely related to what we will do is Druckman and Bolsen (2011) who investigated
whether the effect of a frame is enhanced if the frame contains factual information. However, they did not
manipulate information and framing separately and hence did not distinguish the information and framing
hypotheses.
10
(Berinsky et al. 2012) and results from experiments performed on MTurk closely match those
performed on other samples and in other settings (Leeper and Mullinix n.d.; Mullinix, Druckman,
and Freese n.d.). Table A1 in the Online Appendix reports the demographic composition of the
study sample. A total of 750 participants took part in the experiment. The survey took about 10
minutes to complete and respondents were compensated $0.75 for their time.
We next describe our design in more detail, including the selection of policy issue, how we
manipulated information and framing, and how we measured our key variables.
Policy Issue
Our first task was to select a suitable policy issue. Given that our goal is to test the
information and framing hypotheses, we use a real issue of modest political salience on which it
should be possible to observe both an influence of information and an influence of framing. We
intentionally avoid studying an issue on which opinions are likely to be crystallized due to
evidence that these types of opinions are resistant to communication effects (see Druckman and
Leeper 2012). One strategy for circumventing the crystallization problem is to examine opinion
dynamics surrounding a fictional issue. To avoid sacrificing ecological validity, however, we
focus instead on a health care policy topic, like those used in many previous studies (Fowler and
Gollust 2015; Leeper 2014). Specifically, we focus on citizens’ support for the implementation of
electronic medical records, which are a digital replacement for paper records typically maintained
by physicians, clinics, and hospitals.
The issue provides an excellent case for studying the influence of information and framing
for several reasons. First, the topic is a real and substantively important policy issue. In recent
years the estimated number of physicians and hospitals using electronic medical records have
varied from 50% to nearly 80%, meaning that their use is not yet widespread or uncontroversial
(Miller and Sims 2004; Ford et al. 2006). Additionally, Title XIII of the American Recovery and
11
Reinvestment Act of 2009 created financial incentives for the adoption of electronic medical
records through Medicaid reimbursements (Blumenthal and Tavenner 2010). Second, the
issue—at the time of the study—had not received significant, recent media attention. Thus, we
avoid studying a more salient issue where citizens would possess more crystalized opinions.
Third, and most importantly, electronic medical records raise numerous considerations, including
whether they increase or decrease the cost of health care and whether they have any impact on the
quality of patient care. We focus on these important and commonly debated dimensions of the
issue—the policy’s fiscal cost and the policy’s impact on a particular target group—that tend to
also be the important dimensions by which many policies are evaluated (see Jerit 2009).
Time 1 Information Manipulation and Measures
As already stated, the purpose of the first phase of the experiment was to manipulate
information about the issue.9 At the beginning of the survey, experimental participants were told
that they would participate in a study “about the quality of journalism” and that they would “read
a few short excerpts from a news article.”
We randomly assigned participants to one of four treatment conditions or a control group.
In each treatment group, participants then read a sequence of four excerpts of an article created
by the study investigators that described (fictional) discussions by the American Medical
Association about the proposed use of electronic medical records. We intentionally wrote articles
in journalistic style and masked the final purpose of the study so that participants were led to
believe that they were simply evaluating journalistic quality. Each condition contained a different
version of the articles. In the High Cost information condition, participants read information
about the high costs of electronic medical records, while in the Low Cost condition other
participants read information about the low costs of electronic medical records. In the High
9
The full stimulus material and all question wordings for all studies are available in the Online Appendix.
12
Impact condition, participants read information about the high (positive) impact of electronic
medical records on health care, while in the Low Impact condition participants read information
about the low impact of electronic medical records. In the control condition, participants read
about an unrelated topic (an album being produced by a musical group). After each excerpt,
participants were asked a factual question about the text they had just read which serves as an
attention check and a mechanism for increasing participants’ engagement with the stimulus
materials.
As a manipulation check of the information treatments, we asked participants two questions
to gauge their beliefs about the policy. Specifically, the Cost Belief question read: “The article
you read discussed a proposal to standardize electronic medical records. From what you read,
will health care costs be more expensive or less expensive under the proposed changes?” and
recorded their answers on a seven-point scale from “much less expensive” (coded -1) to “much
more expensive” (coded 1). The Impact Belief question asked: “From what you read, will health
care quality be improved or harmed under the proposed changes?” and recorded their answers on
a seven-point scale from “Harmed a lot” (-1) to “Improved a lot” (1).10
Participants then answered some general questions about the written quality of the excerpts
(consistent with our cover story), answered some demographic questions, and then participated in
an unrelated study of preferences for political candidates’ faces (which served as a distractor
activity). We intentionally did not measure policy opinion during the t1 information phase to
avoid any consistency biases in opinion reports during the framing phase of the study.
Time 2 Framing Manipulation and Measures
After completing the intervening material, participants then participated in the final part of
the survey where we exposed them to frames about the policy. This part of the survey was
10
Participants in the control condition were not asked these questions.
13
introduced by telling participants that “The third study asks you for your opinion about a recent
political issue” and then asked them to read a text that emphasized either the costs or health
impact of electronic medical records.11 This framing manipulation was carefully crafted to not
supply any additional information about the issue, but rather only emphasizes one dimension or
the other. This provides a clean test of whether emphasis alone can move opinions. Participants
were randomly assigned to receive either the Cost Frame or the Impact Frame:
Cost Frame: “Recently there has been some debate in Congress about a new proposal
regarding electronic medical records. Some support the proposal. Others are opposed
because they say that we should judge the proposal based on whether it is costly.”
Impact Frame: “Recently there has been some debate in Congress about a new proposal
regarding electronic medical records. Some support the proposal. Others are opposed
because they say that we should judge the proposal based on whether it will affect the
health of average Americans.”
This manipulation is fully crossed with the t1 information conditions, yielding a 5x2 factorial
design with 10 experimental conditions in total. After reading the framing treatment, respondents
were asked for their opinion on the electronic medical records proposal: “Given this information,
to what extent do you favor or oppose this proposal?” and could supply their response on a fully
labeled seven-point scale from “strongly favor” to “strongly oppose” (with “neither favor nor
oppose” as the middle category). In the analysis, this measure was recoded to scale from -1 to 1,
with higher scores indicating support.
We also included a secondary dependent variable as an additional way of gauging framing
effects. This measure allows us to test whether participants in the Cost (Impact) Frame condition
11
We use one-sided frames because we want to test the impact of framing when it is theorized to be most powerful.
Previous work has found that the effects of opposing or competing frames tend to cancel (Chong and Druckman
2007b; Sniderman and Theriault 2004).
14
said that costs (impact on patients’ health) were a more important consideration when thinking
about the issue (Druckman and Nelson 2003; Nelson, Clawson, and Oxley 1997). Specifically,
we provided participants with a list of six ideas (“improving technology,” “costs,” “errors in
medical records,” “the health of average Americans,” “the opinions of doctors,” and “patient
privacy”) and asked them to rate the importance of each idea on a five-point scale from “not at all
important” (coded 0) to “extremely important (coded 1).
As a manipulation check of the framing treatment, we asked all respondents the following:
“Please think again about the proposal regarding electronic medical records. Do opponents say
we should judge the proposal according to its costs, its impact on health, both, or neither?” We
then measured the percentage of respondents correctly identifying the dimension emphasized by
proposal opponents (which is what was manipulated in the framing vignettes). This enables us to
test whether the frames were “received” by the participants, thus allowing us to rule out
inattention as an alternative explanation in the event that we find weak framing effects.
Results
We begin by examining whether the information manipulation influenced participants’
beliefs about electronic medical records. The results are reported in Table 1, with column 1
showing results for the Cost Belief question and column 2 showing results for the Impact Belief
question. As should be immediately clear, those who received High Cost information saw
electronic medical records as much more costly than those in any of the other conditions, but
differed only marginally from those in the Low Cost information condition when it comes to
perceptions of the policy’s impact on health care quality.12
By contrast, individuals in the High Impact condition saw the policy as offering
substantially improved health care quality compared to those in the Low Impact condition. While
12
For clarity of presentation, we place all statistical significance tests in Tables A2-A8 in the Online Appendix.
15
these two groups differed somewhat in their perceptions of the cost of electronic medical records,
both conditions were much closer to neutral on average than either of the costs conditions.13 Yet
the effects of the information manipulations are clear: Participants’ beliefs about the issue were
substantially influenced by exposure to just a few short paragraphs of information. This result
supports the information hypothesis.
Table 1. Beliefs about Cost and Impact by Information Condition (Study 1)
Information Condition
Cost Beliefs
Impact Beliefs
High Cost
0.81 (0.02)
0.22 (0.03)
Low Cost
-0.65 (0.03)
0.33 (0.03)
High Impact
-0.25 (0.04)
0.88 (0.02)
Low Impact
-0.03 (0.03)
0.17 (0.03)
We now turn to analyzing the second phase of the experiment. In particular, we are
interested in the mean level of support for electronic medical records in each of our ten
experimental conditions. If information affects opinions, then these groups should differ based on
what information they received in the information treatment phase. If framing matters, then
groups that received the same information treatment should differ based only on what dimension
is emphasized in the framing treatment phase. We present the results in Figure 2, as pairs of
points representing treatment group means with error bars representing one- and two-standard
errors of the mean. In each pair, the gray bar represents the Cost Frame condition and the black
bar the Impact Frame condition.
The results are easy to interpret: Framing an issue by emphasizing a particular issue
dimension (be it cost or impact) has no effect on participants’ opinions. Only in the control group
(where respondents received no information treatment) do we see anything close to a framing
13
We acknowledge that the impact information manipulations produced a somewhat smaller difference in impact
beliefs than did the cost information for cost beliefs. One possibility for this is that it is hard to imagine serious
policy debate by health care experts (i.e., the AMA) about a policy that would have negative impacts on health, thus
the Low Impact information simply made the policy seem inconsequential as opposed to detrimental to health.
16
effect and it is arguably in the opposite direction of what might be expected given those in the
Cost Frame condition are more supportive.
How should we interpret these results in light of the information and framing hypotheses?
Looking again at Figure 2, we see that information clearly mattered. Individuals in the High
Impact and Low Cost conditions were more supportive on average than those in the Low Impact,
High Cost, or control conditions. When individuals have received favorable information about a
policy, they are more supportive and when they have received unfavorable information about that
policy, they are less so. This lends clear support to the information hypothesis and suggests that
policy information is vital to shaping preferences. Framing, by contrast, seems to matter very
little. Simply because the debate shifts to emphasize one particular feature of the policy does not
mean that the public is easily swayed to change their opinions. Information, not emphasis,
changes opinions.
One could argue, however, that these results do not challenge the framing hypothesis;
rather they simply indicate that the framing manipulation did not “work” experimentally and thus
there is a null finding. It may simply be that participants did not notice that a particular dimension
of the policy was being emphasized. Fortunately, our framing manipulation check (measuring
whether participants correctly perceived the frame used by the proposal’s opposition) allows us to
know whether participants picked up on the frame being emphasized. In the Cost Frame
condition, 63% (s.e. = 2.50) of respondents correctly perceived the debate as emphasizing costs
and only 6% (1.19) perceived the debate as emphasizing impact. Similarly, in the Impact Frame
condition, 34% (2.46) of respondents correctly perceived the debate as emphasizing impact and
only 24% (2.21) of respondents perceived the debate to be emphasizing costs. While this
manipulation check indicates that participants were better able to identify the Cost Frame than the
Impact Frame, the differences in responses indicate that the two framing conditions clearly
17
emphasized different dimensions. The framing treatments therefore seem to have “worked” as
intended and were received by the study participants.
We can further probe the possibility of finding framing effects with our secondary
dependent variable on belief importance. Consistent with the results on opinion, however, we
find that framing a given dimension did not sizably change the importance participants attached
to that dimension. In the Cost Frame condition, respondents rated the importance of costs at 0.72
(0.01) (on the 0 to 1 scale) and rated the importance of health impact at 0.75 (0.01). Similarly, in
the Impact Frame condition, respondents rated the importance of health impact at 0.79 (0.01) and
the importance of impact at 0.76 (0.01). In other words, pure framing in the form of telling
citizens that a dimension is important apparently is not enough for them to substantially value
that dimension more and, consequently, may explain why framing alone did little to change
opinions toward the issue. This is important because it means that framing does not work
empirically as it is thought to work in theory. The result of our first study is clear: Information,
not emphasis, changes opinions.
Study 2: A Stronger Framing Manipulation
Given Study 1 indicated that opinions moved in response to information but not in response
to framing, we modified our experimental protocol to create more favorable conditions for
finding support for the framing hypothesis. Specifically, we changed the framing manipulations
to be longer and more clear, while still not introducing any additional information. The text now
read:
Cost Frame: “Recently there has been some debate in Congress about a new proposal
regarding electronic medical records. Some support the proposal. Others are opposed
because they say that we should judge the proposal based on whether it is costly. Indeed,
much of the debate over the proposal now revolves around the question of costs.”
18
Impact Frame: “Recently there has been some debate in Congress about a new proposal
regarding electronic medical records. Some support the proposal. Others are opposed
because they say that we should judge the proposal based on whether it will affect the
health of average Americans. Indeed, much of the debate over the proposal now revolves
around the question of the proposal’s impact on patients’ health.”
Unlike in Study 1, where the frames were owned by particular sides in the debate (opponents
owning one side and supporters the other), this manipulation additionally highlights that the
emphasized frame is defining public debate. In other words, the emphasized frame has won.14
Furthermore, we changed the question wording for our main outcome measure. Specifically,
those in the Cost Frame condition were asked: “Based on whether you think the proposal will
mean higher or lower costs, to what extent do you favor or oppose this proposal?” while those in
the Impact Frame condition were asked: “Based on whether you think the proposal will have a
large or small impact on patients' health, to what extent do you favor or oppose this proposal?”
Response categories for both questions were the same as in Study 1. This manipulation of
question wording reinforces the framing manipulations by asking participants to explicitly weight
particular beliefs while, again, not supplying any new information.
Otherwise the protocol for Study 2 closely matched that for Study 1. Indeed, the remaining
questionnaire was identical. We again recruited participants from Amazon Mechanical Turk,
excluding those who had participated in Study 1. This time, however, we engaged in quota
sampling based on self-reported party identification. We used a Qualification Test to identify
workers who were self-described Democrats, Republicans, and Independents and then capped the
14
We also made small changes to the wording of our information treatments (see the Online Appendix).
19
number of participants from each of these groups.15 Given the liberal-Democratic leanings of
MTurk participants, we hoped this quota sampling procedure would balance our sample
ideologically (lest that affected the Study 1 results) and indeed it succeeded (see Table A1 in the
Online Appendix). While the demographics of our Study 2 sample closely mirror those for Study
1 in terms of race, age, and education, the Study 2 sample of 743 participants is notably more
conservative, Republican, and male, thus providing us with a more face-representative sample.
Results
As in Study 1, we begin by first examining whether our information manipulations at t1
changed participants’ beliefs about the cost and impact of the policy. We report these results in
Table 2. Consistent with Study 1, we see that those in the High Cost condition saw electronic
medical records as more costly than those in the Low Cost condition, but they differed only
modestly in their perceptions of the health impact of electronic medical records. Similarly, those
in the High Impact condition saw electronic medical records as more beneficial for patients’
health than those in the Low Impact condition, while these groups differed only modestly in their
perceptions of the policy’s costs. In other words, the information manipulations influenced
beliefs as expected.
Table 2. Beliefs about Cost and Impact by Information Condition (Study 2)
Information Condition Cost Beliefs
Impact Beliefs
High Cost
0.81 (0.02)
0.23 (0.03)
Low Cost
-0.70 (0.03)
0.41 (0.03)
High Impact
-0.23 (0.04)
0.89 (0.02)
Low Impact
0.03 (0.02)
0.14 (0.02)
We can then examine the mean level of policy support in each of our experimental
conditions. As in Study 1, we report these results visually (see Figure 3). As in Study 1, we find
15
We accomplished this using separate HITs for each group, though workers were naïve to why they qualified for
only one of the three HITs. This was achieved using MTurkR (Leeper, 2012).
20
that the information treatments have large and intuitive effects on opinions: Those exposed to
favorable information (High Impact or Low Cost) are more supportive than those who are told
the policy has a low impact on health or those in the control group. Participants in the High Cost
condition are particularly unfavorable. Information matters, yet Figure 3 also makes clear that—
despite our efforts to strengthen the framing manipulation in order to create favorable conditions
for the framing hypothesis—we once again find no support for framing. Emphasis alone did not
change opinions.
As in Study 1, we can verify if the framing manipulation “worked.” It did. In the Cost
Frame condition, 64% (2.50) of respondents correctly perceived the debate as emphasizing costs
and only 7% (1.35) perceived the debate as emphasizing impact. Similarly, in the Impact Frame
condition, 34% (2.46) of respondents correctly perceived the debate as emphasizing impact and
29% (2.35) of respondents perceived the debate to be emphasizing costs. While it may be that the
Impact Frame did not adequately steer participants’ attention to impact, the Cost Frame clearly
did so, and still, we did not find support for the framing hypothesis even in the Cost Frame
conditions.
Consistent with these results, we also find that the framing treatment did not substantially
affect the importance participants attached to issue-relevant beliefs. In the Cost Frame condition,
respondents rated the importance of costs at 0.71 (0.01) (on the 0 to 1 scale) and rated the
importance of health impact at 0.74 (0.01). Similarly, in the Impact Frame condition, respondents
rated the importance of health impact at 0.78 (0.01) and the importance of impact at 0.75 (0.01).
In sum, Study 2 aimed to build on the results from Study 1 while intentionally modifying
the design to favor the framing hypothesis. Thus, we modified the framing manipulations to make
the emphasized dimensions clearer and to more explicitly emphasize that debate about the policy
was universally seen as either about costs or impact on health. Despite these modifications, the
21
results of Study 2 closely mirror those of Study 1. Information (the change in belief content), not
framing (the changing importance of beliefs), led to opinion changes.
Study 3: Inducing Multiple, Competing Beliefs
Given that Study 2, with its strengthened framing manipulation, again failed to find support
for the framing hypothesis, we decided to pursue a third and final study. It might have been the
case that Studies 1 and 2 created unrealistic conditions for finding a framing effect. Specifically,
all participants were exposed to only one-sided information at t1. Thus, when participants who at
t1 were presented with High Cost information about electronic medical records, subsequently (at
t2) received the Impact Frame, it is possible that they had no relevant beliefs about impacts on
patients’ health that could be weighted by the frame. In other words, without having a mix of
beliefs available in memory about the dimension being emphasized, it may be that we
unintentionally neutered our framing manipulation. Indeed, the perception in the framing
literature is that frames are particularly influential when they play at different, opposing
considerations (Nelson and Kinder 1996: 1058; Sniderman and Theriault 2004: 138).
In response, for Study 3 we modified our information treatments. Specifically, rather than
exposing each participant to information about only one issue dimension, all respondents were
exposed to both dimensions such that every participant received information about the cost of
electronic medical records and information about the impact on patients’ health.
The design therefore consisted of five conditions at t1: (1) Low Cost/Low Impact, (2) High
Cost/Low Impact, (3) Low Cost/High Impact, and (4) High Cost/High Impact, and (5) the usual
control condition from Studies 1 and 2. Two of these conditions (1 and 4) should induce high
degrees of belief ambivalence and thus create ripe conditions for framing effects because on one
dimension the policy scores well while on the other it scores poorly. Weighting one dimension
versus the other should therefore produce a change in opinion. Similarly, in condition (2) the
22
policy should be disliked regardless of emphasis because it scores poorly on both dimensions and
in condition (3) the policy should be liked regardless of framing because it scores well on both
dimensions.
Otherwise, the protocol for Study 3 closely mirrored that of Studies 1 and 2. Aside from
changes to the t1 information manipulation, the remainder of the questionnaire (including the
framing manipulation) was unchanged. We again recruited participants from Amazon Mechanical
Turk, excluding those who had participated in either Study 1 or Study 2, and we again relied on
the quota sampling procedure used in Study 2 to create an ideologically balanced sample of 806
participants (for demographics, see the Online Appendix).
Results
Given that Study 3 used substantially different information manipulations than Studies 1
and 2, it is worth reading these results carefully. Our concern in the previous experiments was
that we only supplied participants with information about a single dimension of consideration
during the information phase, thus preventing them from being framed by the subsequent mere
emphasis of a particular dimension of the issue. It is therefore important that the information
treatments induced changes in beliefs about both cost and impact.
As is clear from Table 3, our information treatments indeed worked as expected. In the
High Cost/High Impact condition, participants report that electronic medical records are both
costly and impactful while those in the Low Cost/Low Impact condition see the proposal as
inexpensive but also as having little impact on health. Therefore, depending on the dimension by
which one considers the issue, participants in every condition should hold both favorable and
unfavorable beliefs, creating a situation ripe for finding support for the framing hypothesis. For
these two conditions, depending on which dimension is framed (i.e., which evaluative belief is
weighted), policy opinions should differ.
23
Table 3. Beliefs about Cost and Impact by Information Condition (Study 3)
Information Condition
Cost Beliefs
Impact Beliefs
High Cost/High Impact
0.76 (0.03)
0.89 (0.02)
Low Cost/High Impact
-0.71 (0.02)
0.77 (0.02)
High Cost/Low Impact
0.65 (0.03)
0.13 (0.02)
Low Cost/Low Impact
-0.47 (0.03)
0.10 (0.02)
In contrast, in the two remaining information conditions, policy beliefs are consistently
positive or negative. Thus, as expected, participants in the Low Cost/High Impact condition
believe the proposal is both very inexpensive and highly impactful whereas those in the High
Cost/Low Impact condition see the proposal as costly and ineffectual. In these information
conditions, we should expect little impact of framing because their evaluative beliefs are positive
or negative, respectively, regardless of which dimension is emphasized.
With the information hypothesis again supported, we can now turn to our results on policy
support which we report visually in Figure 4. First we note that the control group reports opinions
consistent with the control groups in our previous studies. Without information about the policy,
this group is neutral toward the policy regardless of frame. We are particularly interested in
whether framing mattered in the conditions where the information treatments induced conflicting
beliefs among participants to make them ripe for framing effects. Consider first the High
Cost/High Impact information condition. If framing matters, we would expect those in the Impact
Frame condition (black bar) to be more supportive than those in the Cost Frame condition (gray
bar), yet these groups are indistinguishable from one another.
Similarly, in the Low Cost/Low Impact information condition, we would again expect
participants’ opinions to vary depending on what dimension is framed. Framing participants in
this information condition to think about costs should make them more favorable (given they see
the policy as low cost) while inducing them to think about impact should make them less
24
favorable (given they see little impact on patients’ health). We find, however, that framing does
not influence opinions as policy support is not significantly different across the two framing
conditions.
Moving to the last two information conditions—High Cost/Low Impact and Low Cost/High
Impact—we would expect minimal framing effects because each evaluative belief points in the
same direction within the conditions. As expected, we find participants in the High Cost/Low
Impact information conditions to be the least supportive of the policy, regardless of framing.
Looking at the Low Cost/High Impact information conditions, we see that they as expected are
among the most positive toward the policy. Here we do find a difference in opinion across
framing conditions where policy support is higher in the Cost Frame (0.58) than in the Impact
Frame (0.46) condition. However this is only marginally significant (p<0.07) and, ironically, it
occurs at an instance where it should not be expected according to the framing hypothesis.
As we found in Studies 1 and 2, this lack of evidence for the framing hypothesis is unlikely
to be explained by a failure to grasp the changing emphasis in the debate. Thus, our manipulation
check showed that our frames were successfully received by participants. In the Cost Frame
condition, 61% (2.43) of respondents correctly perceived the debate as emphasizing costs and
only 9% (1.44) perceived the debate as emphasizing impact. Similarly, in the Impact Frame
condition, 35% (2.39) of respondents correctly perceived the debate as emphasizing impact
whereas 28% (2.24) of respondents perceived the debate to be emphasizing costs. As in our
previous studies, we recognize the Cost Frame appears to be more easily recognized than the
Impact Frame, but we yet again found no support for the framing hypothesis.
Consistent with the previous studies, we find substantially tiny framing effects on belief
importance. The alternative considerations were perceived as essentially equally important
regardless of frame. Specifically, in the Cost Frame condition, respondents rated the importance
25
of costs at 0.71 (0.01) and the importance of health impact at 0.77 (0.01). Similarly, in the Impact
Frame condition, respondents rated the importance of health impact at 0.74 (0.01) and the
importance of impact at 0.75 (0.01).
What does this mean? Our concern in Studies 1 and 2 was that we had unfairly
disadvantaged the framing hypothesis by manipulating participants’ beliefs about only one
dimension of the policy. Our response was to cross the two information dimensions (cost and
impact) in an effort to more fairly test the framing hypothesis. When we disentangle the impact of
information from the impact of framing in this way, the results suggest it is information, not
emphasis, that explains the opinion changes consistently observed in previous studies of
“framing.”
Discussion
A framing effect—the result of placing emphasis on a particular dimension of
consideration—is widely thought to be among politicians’ most important tools for influencing
public opinions. Yet we have argued that most extant framing research does not disentangle the
influence of emphasizing a particular dimension of an issue (the framing hypothesis) from the
provision of information and arguments (the information hypothesis). We approached this
problem with a novel experimental design that physically and temporally separated the provision
of information and the communication of frames. Across three studies, each carefully crafted to
provide an easier and easier opportunity for framing effects to emerge, we find no evidence for
framing effects. Information, not emphasis, shifts citizens’ opinions.
These results invite us to reconsider the meaning of the sizeable extant literature on framing
effects. We do not mean to question the validity or importance of the findings in this literature.
However, our results suggest that extant studies demonstrate the influence of information and
arguments, rather than emphasis alone. Opinions in previous experiments were likely moved by
26
the information and arguments contained in the treatments not the way the information was
framed. This means that framing can rightfully be thought of as a phenomenon separate from
argument-based persuasion or information effects, but we see little evidence here for the impact
of framing (true to its name) on public opinions. This finding has important political, theoretical,
and normative implications.
Politically, the limited empirical support for pure framing implies that politicians, political
parties, and other actors must do far more than frame (or reframe) issues to change public
attitudes. They must provide arguments and evidence in support of their positions. As noted by
Sniderman and Carmines (1997: 129), “the politics of public policy is [not] merely a matter of
marketing. It is a matter of argument. The public’s reaction to public policy hinges in part on
what is proposed by way of public opinion, and on what arguments are made both for and against
what is being proposed.” This more demanding political strategy should lead political scientists
to rethink the scope and size of elite influence, and might hold promising normative implications
for citizens forming opinions grounded in substantive arguments to a larger extant than
previously thought (see Druckman 2014).
Theoretically, our results challenge the meaning and relevance of the framing concept and
the viability of the expectancy-value model of attitudes that underlies it. The literature on media
effects already discusses how to distinguish between framing and other types of effects and
psychological processes (e.g., de Vreese and Lecheler 2012; Kinder 2003; Scheufele and Iyengar
2015). Notably, Chong and Druckman (2012: 310-311) suggest to “view framing, priming and
agenda setting as equivalent processes that involve alterations of the weight component of an
attitude … until definitive evidence reveals mediational or moderating differences, the concepts
should be treated as the same, meaning research on each should be merged and redundancy
avoided,” and they add: “Along similar lines, the extent to which persuasion and the other
27
concepts differ remains unclear” (p. 321). If framing, priming and agenda setting are related, or
perhaps equivalent, concepts, our findings should also raise concern for the distinctiveness of
these other concepts. Lenz (2009) already showed that the processes underlying “priming effects”
are more often learning followed by rationalization, and Togeby (2007) showed that in order for
priming to occur, an argument was needed to make clear why a given issue would be relevant for
evaluating government performance (also see Druckman, Kuklinski, and Sigelman 2009; Miller
and Krosnick 2000; Mutz 1998: 72-73).16
That evidence, in tandem with our findings, suggests that “direct” communication effects
might play a much larger role than the “indirect” influences of agenda setting, priming, and
framing has suggested (Kinder 2003). We need future studies that probe the potential and
meaning of framing further. It is clear that citizens are able to view an issue from different
perspectives, emphasizing different set of considerations (Chong 1993; Feldman and Zaller
1992). Yet it has been difficult to illuminate empirically what it takes for citizens to come to rely
on different dimensions and, furthermore, what those different frames in thought actually mean
for opinion dynamics. A growing body of evidence suggests that citizens can approach a
judgment or decision-making task from different perspectives: elite arguments shape how citizens
choose information (Druckman, Fein, and Leeper 2012) and how they evaluate political debates
(Druckman et al. 2010). Perhaps frames provide a lens for understanding new information, even
if they do little to reorganize beliefs already in memory.
Finally, democracy as a form of government is premised upon the representation of public
preferences by government. The consistent finding of a reverse pattern of influence whereby
16
Mutz (1998: 72-73) notes that evidence of agenda setting and priming is “sparse”: “evidence to date suggests that
what is happening in many cases may be a directional effect instead of, or perhaps in addition to, an effect on the
accessibility of certain information … much of the evidence brought to bear on these hypotheses suggests that media
are capable of altering more than simply the salience attached to a given issue or the standards people use to evaluate
political leaders.”
28
elites shape public preferences problematizes this democratic foundation. Our findings suggest
that elites cannot easily sway public preferences without communicating policy-relevant
information and develop persuasive arguments to support their preferred position. We find that
emphasis alone is an insufficient political strategy. Citizens are not so easily swayed.
29
Figure 1: Experimental Design for Separating Information and Framing Effects
30
Figure 2. Mean Opinion by Information and Framing Conditions (Study 1)
Note: Points are mean levels of policy support, by information and framing conditions. Gray
points represent Cost Frame conditions and black points represent Impact Frame conditions. Bars
represent one and two standard errors of the treatment group mean.
31
Figure 3. Mean Opinion by Information and Framing Conditions (Study 2)
Note: Points are mean levels of policy support, by information and framing conditions. Gray
points represent Cost Frame conditions and black points represent Impact Frame conditions. Bars
represent one and two standard errors of the treatment group mean.
32
Figure 4. Mean Opinion by Information and Framing Conditions (Study 3)
Note: Points are mean levels of policy support, by information and framing conditions. Gray
points represent Cost Frame conditions and black points represent Impact Frame conditions. Bars
represent one and two standard errors of the treatment group mean.
33
References
Aarøe, Lene. 2011. “Investigating Frame Strength: The Case of Episodic and Thematic Frames.”
Political Communication 28(2): 207-226.
Berinsky, Adam J., Gregory A. Huber, and Gabriel S. Lenz. 2012. “Evaluating Online Labor
Markets for Experimental Research: Amazon.com's Mechanical Turk.” Political Analysis
20(3): 351-368.
Berinsky, Adam J., and Donald R. Kinder. 2006. “Making Sense of Issues Through Media
Frames: Understanding the Kosovo Crisis.” Journal of Politics 68(3): 640–656.
Blumenthal, David, and Tavenner, Marilyn. 2010. "The 'Meaningful Use' Regulation for
Electronic Health Records.” New England Journal of Medicine 363: 501-504.
Boudreau, Cheryl. 2015. “Party versus Principle: How Competing Parties and Frames Affect the
Consistency between Citizens’ Values and Policy Views.” Typiscript. University of
California, Davis.
Brewer, Paul R. 2001. “Value Words and Lizard Brains: Do Citizens Deliberate About Appeals
to Their Core Values?” Political Psychology 22(1): 45-64.
Chong, Dennis. 1993. “How People Think, Reason, and Feel about Rights and Liberties.”
American Journal of Political Science 37(3): 867.
Chong, Dennis. 2000. Rational Lives: Norms and Values in Politics and Society. Chicago:
University of Chicago Press.
Chong, Dennis, and James N. Druckman. 2007a. “Framing Theory.” Annual Review of Political
Science 10(1): 103-126.
Chong, Dennis, and James N. Druckman. 2007b. “Framing Public Opinion in Competitive
Democracies.” American Political Science Review 101(4): 637-655.
Chong, Dennis, and James N. Druckman. 2012. “Dynamics in Mass Communication Effects
Research.” In The Sage Handbook of Political Communication, eds. Holli Semetko, and
Maggie Scammell. Thousand Oaks, CA: Sage Publications.
Dahl, Robert A. 1971. Polyarchy. Chicago: University of Chicago Press.
Dahl, Robert A. 1989. Democracy and Its Critics. New Haven, CT: Yale University Press.
de Vreese, Claes H. 2003. Framing Europe: Television News and European Integration.
Amsterdam: Aksant.
de Vreese, Claes H., and Sophie K. Lecheler. 2012. “News Framing Research: An Overview and
New Developments.” In The SAGE Handbook of Political Communication, eds. Holli
Semetko, and Margaret Scammell. SAGE Publications.
34
Disch, Lisa. 2011. “Toward a Mobilization Conception of Democratic Representation.” American
Political Science Review 105(1): 100-114.
Druckman, James N. 2001. “The Implications of Framing Effects for Citizen Competence.”
Political Behavior 23(3): 225-256.
Druckman, James N. 2014. “Pathologies of Studying Public Opinion, Political Communication,
and Democratic Responsiveness.” Political Communication 31(3): 467-492.
Druckman, James N., and Toby Bolsen. 2011. “Framing, Motivated Reasoning, and Opinions
About Emergent Technologies.” Journal of Communication 61(4): 659–688.
Druckman, James N., Cari Lynn Hennessya, Kristi St. Charlesa, and Jonathan Webber. 2010.
“Competing Rhetoric Over Time: Frames Versus Cues.” The Journal of Politics 72(1):
136-148.
Druckman, James N., James H. Kuklinski, and Lee Sigelman. 2009. “The Unmet Potential of
Interdisciplinary Research: Political Psychological Approaches to Voting and Public
Opinion.” Political Behavior 31(4): 485-510.
Druckman, James N., and Thomas J. Leeper. 2012. “Learning More from Political
Communication Experiments: Pretreatment and Its Effects.” American Journal of
Political Science 56(4): 875-896.
Druckman, James N., and Kjersten R. Nelson. 2003. “Framing and Deliberation: How Citizens'
Conversations Limit Elite Influence.” American Journal of Political Science 47(4): 729745.
Druckman, James N., Erik Peterson, and Rune Slothuus. 2013. “How Elite Partisan Polarization
Affects Public Opinion Formation.” American Political Science Review 107 (1): 57-79.
Feldman, Stanley, and John Zaller. 1992. “The Political Culture of Ambivalence: Ideological
Responses to the Welfare State.” American Journal of Political Science 36(1): 268-307.
Ford, Eric W., Menachemi, Nir, and Phillips, M. Thad. 2006. “Predicting the Adoption of
Electronic Health Records by Physicians: When Will Health Care Be Paperless?” Journal
of the American Medical Informatics Association 13: 106-112.
Fowler, Erika F., and Sarah E. Gollust. 2015. “The Content and Effect of Politicized Health
Controversies.” The ANNALS of the American Academy of Political and Social Science
658(1): 155-171.
Gamson, William A., and Andre Modigliani. 1987. “The Changing Culture of Affirmative
Action.” Research in Political Sociology 3: 137-177.
Hartman, Todd K., and Christopher R. Weber. 2009. “Who Said What? The Effects of Source
Cues in Issue Frames.” Political Behavior 31(4): 537-558.
35
Hopkins, Daniel J. 2014. “The Exaggerated Life of Death Panels: The Limits of Framing Effects
in the 2009‐2012 Health Care Debate.” Typescript. Georgetown University.
Huber, Gregory A., and Celia Paris. 2013. “Assessing the Programmatic Equivalence
Assumption in Question Wording Experiments: Understanding Why Americans Like
Assistance to the Poor More Than Welfare.” Public Opinion Quarterly 77: 385-397.
Jacoby, William G. 2000. “Opinion and Public Issue Framing Spending on Government.”
American Journal of Political Science 44(4): 750-767.
Jerit, Jennifer. 2009. “How Predictive Appeals Affect Policy Opinions.” American Journal of
Political Science 53(2): 411-426.
Kinder, Donald R. 2003. “Communication and Politics in the Age of Information.” In Oxford
Handboook of Political Psychology, eds. David O. Sears, Leonie Huddy, and Robert
Jervis. New York: Oxford University Press.
Lecheler, Sophie K., and Claes H. de Vreese. 2013. “What a Difference a Day Makes? The
Effects of Repetitive and Competitive News Framing Over Time.” Communication
Research 40(2): 147-175.
Leeper, Thomas J. 2012. “MTurkR: R Client for the MTurk Requester API.” Version 0.6.
Leeper, Thomas J. 2014. “The Informational Basis for Mass Polarization.” Public Opinion
Quarterly 78(1): 27-46.
Leeper, Thomas J., and Kevin J. Mullinix. N.d. “What If You Had Done Things Differently?
Testing the Generalizability of Framing Effects with Parallel Experiments.” Typescript.
Aarhus University.
Lenz, Gabriel S. 2009. “Learning and Opinion Change, Not Priming: Reconsidering the Priming
Hypothesis.” American Journal of Political Science 53(4): 821-837.
Miller, Joanne M., and Jon A. Krosnick. 2000. “News Media Impact on the Ingredients of
Presidential Evaluations: Politically Knowledgeable Citizens are Guided by a Trusted
Source.” American Journal of Political Science 44(2): 301-315.
Miller, Robert H., and Ida Sim. 2004. “Pursuit Of Quality: Physicians' Use Of Electronic Medical
Records: Barriers And Solutions.” Health Affairs 23: 2116-2126.
Mullinix, Kevin J., James N. Druckman, and Jeremy Freese. N.d. “When Convenience Samples
Yield Generalizable Estimates in Social Science Experiments.” Typescript. Northwestern
University.
Mutz, Diana C. 1998. Impersonal Influence. New York: Cambridge University Press.
Nelson, Thomas E., and Donald R. Kinder. 1996. “Issue Frames and Group-Centrism in
American Public Opinion.” The Journal of Politics 58(4): 1055.
36
Nelson, Thomas E., Rosalee A. Clawson, and Zoe M. Oxley. 1997. “Media Framing of a Civil
Liberties Conflict and Its Effect on Tolerance.” American Political Science Review 91(3):
567-583.
Nelson, Thomas E., Zoe M. Oxley, and Rosalee A. Clawson. 1997. “Toward a Psychology of
Framing Effects.” Political Behavior 19(3): 221-246.
Price, Vincent, and David Tewksbury. 1997. “News Values and Public Opinion: A Theoretical
Account of Media Priming and Framing.” In Progress in Communication Sciences.
Norwood, N.J.: Ablex Publishing Corporation, 173-212.
Schemer, Christian, Werner Wirth, and J• org Matthes. 2012. “Value Resonance and Value
Framing Effects on Voting Intentions in Direct-Democratic Campaigns.” American
Behavioral Scientist 56(3): 334-352.
Scheufele, Dietram A., and Shanto Iyengar. 2015. “The State of Framing Research: A Call for
New Directions.” In The Oxford Handbook of Political Communication, eds. Kate Kenski
and Kathleen Hall Jamieson. New York: Oxford University Press. Forthcoming.
Schuck, Andreas R.T., Rens Vliegenthart, and Claes H. de Vreese. 2015. “Who's Afraid of
Conflict? The Mobilizing Effect of Conflict Framing in Campaign News.” British Journal
of Political Science. Forthcoming.
Schuman, Howard, and Stan Presser. 1981. Questions and Answers in Attitude Surveys:
Experiments on Question Form, Wording, and Context. Thousand Oaks, CA: SAGE
Publications.
Slothuus, Rune. 2008. “More Than Weighting Cognitive Importance: A Dual-Process Model of
Issue Framing Effects.” Political Psychology 29(1): 1-28.
Slothuus, Rune. 2010. “When Can Political Parties Lead Public Opinion? Evidence from a
Natural Experiment.” Political Communication 27(2): 158-177.
Slothuus, Rune, and Claes H. de Vreese. 2010. “Political Parties, Motivated Reasoning, and Issue
Framing Effects.” The Journal of Politics 72(3): 630-645.
Sniderman, Paul M., and Sean M. Theriault. 2004. “The Structure of Political Argument and the
Logic of Issue Framing.” In Studies in Public Opinion: Attitudes, Nonattitudes,
Measurement Error, and Change, eds. William E. Saris, and Paul M. Sniderman.
Princeton, NJ: Princeton University Press, 133-165.
Togeby, Lise. 2007. “The Context of Priming.” Scandinavian Political Studies 30(3): 345-376.
Weaver, David H. 2007. “Thoughts on Agenda Setting, Framing, and Priming.” Journal of
Communication 57(1): 142-147.
Zaller, John. 1992. The Nature and Origins of Mass Opinion. New York: Cambridge University
Press.
37
SUPPORTING INFORMATION ONLINE APPENDIX
Appendix 1: Supplemental Statistical Results
Table A1. Study Demographics
% Female
% White
Mean Age
% With University Degree
% Democrat
% Republican
Mean Ideology
Sample size
Dates of Data Collection
Study 1
0.56 (0.25)
0.77 (0.18)
36.72 (12.84)
0.50 (0.25)
0.51 (0.25)
0.26 (0.19)
0.16 (0.57)
Study 3
0.44 (0.25)
0.80 (0.16)
33.27 (10.78)
0.48 (0.25)
0.41 (0.24)
0.34 (0.23)
0.10 (0.54)
750
743
806
Jun. 19-21, 2014 Oct. 16-Nov. 5, 2014 Jan 5-28, 2015
38
Study 2
0.40 (0.24)
0.79 (0.16)
31.60 (9.71)
0.52 (0.25)
0.40 (0.24)
0.34 (0.22)
0.11 (0.54)
Table A2. Mean Opinion and Framing Effect by Information Condition (Study 1)
Information
Control
High Cost
Low Cost
High Impact
Low Impact
F-test
Impact Frame
Cost Frame t-statistic (p-value)
0.13
0.31
-2.53 (p<0.01)
0.12
0.14
-0.19 (p<0.85)
0.39
0.37
0.29 (p<0.77)
0.48
0.51
-0.34 (p<0.74)
0.27
0.23
0.42 (p<0.67)
F(4,368) = 8.73 (p<0.00)
F(4,371) =
7.08 (p<0.00)
Table A3. Mean Opinion and Framing Effect by Information Condition (Study 2)
Information
Impact Frame
Cost Frame t-statistic (p-value)
Control
0.14
0.22
-1.04 (p<0.30)
High Cost
0.02
0.07
-0.64 (p<0.52)
Low Cost
0.47
0.51
-0.58 (p<0.57)
High Impact
0.50
0.47
0.53 (p<0.59)
Low Impact
0.36
0.32
0.71 (p<0.48)
F-test
F(4,366) = 15.74 (p<0.00)
F(4,364) =
12.23 (p<0.00)
Table A4. Mean Opinion and Framing Effect by Information Condition (Study 3)
Information
Control
High Cost/High Impact
Low Cost/High Impact
High Cost/Low Impact
Low Cost/Low Impact
F-test
Impact Frame
Cost Frame t-statistic (p-value)
0.14
0.19
-0.69 (p<0.49)
0.33
0.30
0.41 (p<0.68)
0.46
0.58
-1.81 (p<0.07)
0.03
0.06
-0.44 (p<0.66)
0.37
0.45
-1.34 (p<0.18)
F(4,397) = 12.14 F(4,398) =
(p<0.00)
16.76 (p<0.00)
Table A5. Belief Importance by Framing Condition (All Studies)
Health Beliefs
Study 1
Study 2
Study 3
Cost Beliefs
Study 1
Study 2
Study 3
Impact Frame
Cost Frame
t-statistic (p-value)
0.79
0.78
0.74
0.75
0.74
0.77
2.28 (p<0.02)
2.22 (p<0.03)
-1.49 (p<0.14)
0.76
0.75
0.75
0.72
0.71
0.71
2.32 (p<0.02)
2.24 (p<0.03)
2.36 (p<0.02)
39
Table A6. Proportion Perceiving Debate Frame by Framing Condition (Study 1)
Perceive Cost
Perceive Impact
Emphasized
Emphasized Difference in Proportions
Impact Frame
0.24
0.34 χ2 = 8.86 (p<0.00)
Cost Frame
0.63
0.06 χ2 = 270.71 (p<0.00)
Difference in
χ2 = 112.72 (p<0.00)
χ2 = 94.81 (p<0.00)
Proportions
Table A7. Proportion Perceiving Debate Frame by Framing Condition (Study 2)
Perceive Cost
Perceive Impact
Emphasized
Emphasized Difference in Proportions
Impact Frame
0.29
0.34 χ2 = 2.25 (p<0.13)
Cost Frame
0.64
0.07 χ2 = 256.84 (p<0.00)
Difference in
χ2 = 90.73 (p<0.00) χ2 = 79.95 (p<0.00)
Proportions
Table A8. Proportion Perceiving Debate Frame by Framing Condition (Study 3)
Perceive Cost
Perceive Impact
Emphasized
Emphasized Difference in Proportions
Impact Frame
0.28
0.35 χ2 = 5.15 (p<0.02)
Cost Frame
0.61
0.09 χ2 = 235.98 (p<0.00)
Difference in
χ2 = 87.56 (p<0.00) χ2 = 80.37 (p<0.00)
Proportions
40
Appendix 2: Study 1 Treatments and Questionnaire
Introductory Text
Today we ask you to participate in three short studies. First you will read some newspaper
excerpts and answer some questions about what you read. Then, you will complete another study
and we will ask you some background questions about yourself. Finally, we will ask you for your
opinions about some political issues.
Information Manipulation
<CONTROL GROUP>
The first study you will participate in is about the quality of journalism. We are interested in how
well journalists write about the news. You will read a few short excerpts from a news article and
then we will ask you some questions to see how well you understood what was written.
We are asking different people to read articles on different topics. You are being asked to read
excerpts from an article about music.
<Excerpt1>
Here is the first part of the article. Please read it carefully.
LOS ANGELES -- Most albums aren't recorded overnight; just ask Steely Dan or Bruce
Springsteen. However, No Doubt's Push and Shove, is one of the
longest-gestating pop albums in recent memory. According to bassist Tony Kanal, it's a triumph
that the album – the band's first since 2001's Rock Steady – was completed at all.
</Excerpt1>
<Excerpt2>
Here is the next part of the article. Please read it carefully.
Gwen Stefani hates it when people point out that No Doubt haven't released an album in 10
years. "A lot of stuff happened during that time period," she says. "Marriages, babies and, for
me, two records and two clothing lines. So if you really worked out the math, you'd be like,
'Wow, you guys are going fast.'"
</Excerpt2>
<Excerpt3>
Here is the next part of the article. Please read it carefully.
The winding road to Push and Shove began in 2007 while lead singer Gwen Stefani was touring
behind her second solo album, The Sweet Escape. In June, during the encore for one of her
shows in Irvine, California, the other members of No Doubt joined her onstage for a four-song
set that included their hits "Just a Girl" and "Hella Good." "The response was so amazing that we
said, 'We have to start working,'" explains Kanal.
</Excerpt3>
<Excerpt4>
Here is the last part of the article. Please read it carefully.
Not long after the show, the band convened to start writing new material, but Stefani was still
41
exhausted from touring and was expecting her second child. The band was also hit with
collective writers' block. Those songwriting sessions and further ones in 2008 yielded little, and
the band decided to give it a rest. "We know when it's right and when it isn't," Kanal says. "We
had to get that feeling again."
</Excerpt4>
<PAGEBREAK/>
What is the name of the band discussed in the article?
Bruce Springsteen and the E Street Band
Bad Religion
No Doubt
Red Hot Chili Peppers
Don’t Know
<PAGEBREAK/>
When did the band first start work on their current album?
1996
2001
2007
2014
Don’t know
</CONTROL GROUP>
<TREATMENT CONDITIONS>
The first study you will participate in is about the quality of journalism. We are interested in how
well journalists write about the news. You will read a few short excerpts from a news article and
then we will ask you some questions to see how well you understood what was written.
We are asking different people to read articles on different topics. You are being asked to read
excerpts from an article about electronic medical records.
<PAGEBREAK/>
<Exercept1>
Here is the first part of the article. Please read it carefully.
DENVER, Colo. – Leaders of the American Medical Association have been busy at work this
week in an annual conference held here in Denver. On Tuesday, the AMA’s policy advisory
committee began debate on a proposal that would change rules regarding how electronic medical
records are handled by the nation’s hospitals and clinics. The proposal would standardize medical
record technology. If implemented, the proposal would mean
<CostHigh>
significantly higher costs for doctors and clinics. A typical clinic could see expenses rise by as
much as $5000 per year under the proposed changes. Nationwide, administrative costs related to
medical records could rise by as much as $17 million over the next two years.
42
</CostHigh>
<CostLow>
significant cost savings for doctors and clinics. A typical clinic could see expenses reduced by as
much as $5000 per year under the proposed changes. Nationwide, administrative costs related to
medical records could be lowered by as much as $17 million over the next two years.
</CostLow>
<BenefitsHigh>
that doctors, clinics, and hospitals would be able to share records more easily and be able to
better serve patient needs and more quickly retrieve vital health information. This would improve
the quality of health care for most Americans.
</BenefitsHigh>
<BenefitsLow>
that doctors, clinics, and hospitals would use standardized software so electronic records are
interoperable – meaning computer systems would rely on compatible formats. This would have
little impact on the quality of health care for most Americans.
</BenefitsLow>
<PAGEBREAK/>
<Comprehension>
What organization was mentioned in the text?
American Hospital Association
American Medical Association
Association of Nurse Practitioners
Doctors Without Borders
Don’t know
</Comprehension>
</Excerpt1>
<Excerpt2>
Here is the next part of the article . Please read it carefully.
Electronic medical records have been the subject of debate within the medical field for more than
ten years. These systems would replace paper records kept by individual doctors and clinics with
sophisticated in-house computer systems. Numerous companies currently produce competing
software.
<CostHigh>
The push for electronic medical records is seen by many experts as an expensive alternative to
paper records. Startup costs for a new electronic medical records system can cost a five-physician
clinic $32,000 plus $8000-$9000 in annual maintenance costs.
</CostHigh>
<CostLow>
The push for electronic medical records is seen by many experts as an cost-saving alternative to
paper records. A new electronic medical records system can save a five-physician clinic $8000$9000 per year in operating costs.
</CostLow>
<BenefitsHigh>
43
The push for electronic medical records is seen by many experts as a clear way to improve the
quality of Americans’ health care. With these systems, doctors can more easily track patient
health over time, and more easily catch risk factors for chronic and life-threatening diseases.
</BenefitsHigh>
<BenefitsLow>
The push for electronic medical records is seen by many experts as a clear way to keep pace with
changing technology. With these systems, medical coding is streamlined, simplifying recordkeeping processes for clinic staff, even if it will have no impact on patients’ health.
</BenefitsLow>
<PAGEBREAK/>
<Comprehension>
According to the article, how long has the issue of electronic medical records been debated?
Less than 1 year
1-5 years
5-10 years
More than 10 years
Don’t know
</Comprehension>
</Excerpt2>
<Excerpt3>
Here is the next part of the article. Please read it carefully.
The Brookings Institution, an independent policy think tank based in Washington, DC, released a
report this morning describing an in-depth analysis of electronic medical records in the United
States. At a press conference, Brookings spokesman Will Richardson said that the report details
how proposed rule changes currently being debated by the nation’s leading hospitals, clinics, and
doctors would
<CostHigh>
lead to significantly higher costs for doctors and patients. Clinics have been slow to adopt
electronic medical records precisely because the costs are so high.
</CostHigh>
<CostLow>
lead to significantly lower costs for doctors and patients. Clinics have been slow to adopt
electronic medical records precisely because they have been waiting for these kinds of savings to
become available.
</CostLow>
<BenefitsHigh>
substantially improve the flexibility of the U.S. health care system, allowing patients and doctors
to quickly share vital health information. The changes would also mean simplified handling of
insurance claims, reducing hassle for American families.
</BenefitsHigh>
<BenefitsLow>
44
likely lead to modifications of existing clinical software and the future development of uniform
formatting requirements for electronic records. The changes would also mean modified rules for
software developers to comply with the new standards.
</BenefitsLow>
<PAGEBREAK/>
<Comprehension>
In what city is the Brookings Institution located?
New York City, New York
Washington, DC
Seattle, Washington
Denver, Colorado
Don’t know
</Comprehension>
</Excerpt3>
<Excerpt4>
Here is the last part of the article. Please read it carefully.
Back in Denver, the AMA conference continues through the end of the week where electronic
medical records seem to be the big topic of debate among conference attendees. Both
practitioners and technology developers are eager to see the policy advisory committee’s
recommendations, which are due by the end of the week.
<CostHigh>
One doctor, Jan Miller of Oakland, CA, remained concerned about her clinic’s bottom line.
“Electronic medical records are a substantial change in clinical practice and my practice cannot
afford such a system without passing costs on to patients.”
</CostHigh>
<CostLow>
One doctor, Jan Miller of Oakland, CA, was confident that electronic medical records would
reduce costs her in practice. “Electronic medical records are a substantial change in clinical
practice and my patients are asking for this system because they know it will save them money.”
</CostLow>
<BenefitsHigh>
One doctor, Jan Miller of Oakland, CA, was confident that electronic medical records would
improve her patients’ health. “Electronic medical records are a substantial change in clinical
practice and patients will see the benefits of these systems right away. Electronic records help me
make fewer errors and know my patients better.”
</BenefitsHigh>
<BenefitsLow>
One doctor, Jan Miller of Oakland, CA, downplayed the importance of this week’s debates.
“Electronic medical records mean little for my practice. It’s really same old, same old. I don’t see
how patients would be affected really.”
</BenefitsLow>
<PAGEBREAK/>
<Comprehension>
45
What was the profession of Jan Miller, the individual quoted in the article?
Software developer
Nurse
Doctor
Insurance agent
Don’t know
</Comprehension>
</Excerpt4>
Information Manipulation Check
The article you read discussed a proposal to standardize electronic medical records.
From what you read, will health care costs be more expensive or less expensive under the
proposed changes?
Much more expensive
Somewhat more expensive
A little more expensive
No change
A little less expensive
Somewhat less expensive
Much less expensive
From what you read, will health care quality be improved or harmed under the proposed
changes?
Improved a lot
Improved somewhat
Improved a little
No change
Harmed a little
Harmed somewhat
Harmed a lot
How easy or difficult was this article for you to understand?
Very easy
Moderately easy
Somewhat easy
Neither easy nor difficult
Somewhat difficult
Moderately difficult
Very difficult
How engaging did you find this article to be?
Extremely engaging
Moderately engaging
Somewhat engaging
Not very engaging
46
Not at all engaging
How similar is this article to news articles you typically see in newspapers or on the internet?
Extremely similar
Moderately similar
Somewhat similar
Not very similar
Not at all similar
Distractor Activity
Demographics
Are you male or female?
Male
Female
Which of the following do you consider to be your primary racial or ethnic group?
White or Caucasian
Black or African American
Asian American
Hispanic
Native American
Other
What is your age in years?
[Free Response]
What is the last grade or class that you completed in school?
None, or grade 1-8
Some high school
High school (Grade 12 or GED)
Technical, trade, or vocational school after high school
Some college
College graduate
Post-graduate training or professional schooling after college
Generally speaking, do you consider yourself a Democrat, Independent, or Republican?
Strong Democrat
Weak Democrat
Independent leans Democrat
Independent
Independent leans Republican
Weak Republican
47
Strong Republican
Framing Manipulation
The third study asks you for your opinion about a recent political issue.
Recently there has been some debate in Congress about a new proposal regarding electronic
medical records.
[RANDOMIZE INTO ONE OF TWO FRAME CONDITIONS]
<CostFrame>
Some support the proposal. Others are opposed because they say that we should judge the
proposal based on whether it is costly.
</CostFrame>
<BenefitsFrame>
Some support the proposal. Others are opposed because they say that we should judge the
proposal based on whether it will affect the health of average Americans.
</BenefitsFrame>
Outcome Measures
Given this information, to what extent do you favor or oppose this proposal?
Strongly favor
Moderately favor
Somewhat favor
Neither favor nor oppose
Somewhat oppose
Moderately oppose
Strongly oppose
How certain are you of your opinion on the proposal regarding electronic medical records?
Not at all certain
Somewhat certain
Moderately certain
Very certain
Extremely certain
How important to you personally is your opinion about the proposal regarding electronic medical
records?
Not at all important
Somewhat important
Moderately important
Very important
Extremely important
48
Framing Manipulation Checks
We are now going to list a few ideas that individuals have expressed when describing their
opinions about the proposal regarding electronic medical records. Some of these ideas may seem
important to you, while others may seem less important. Please tell us how important each of
these ideas is to you when thinking about your overall opinion of this proposal.
[GRID]
Improving technology
Costs
Errors in medical records
The health of average Americans
The opinions of doctors
Patient privacy
[GRID]
Not at all important
Somewhat important
Moderately important
Very important
Extremely important
Please think again about the proposal regarding electronic medical records.
Do opponents say we should judge the proposal according to…
Its costs
Its impact on health
Both
Neither
Don’t know
Appendix 3: Study 2 Treatments and Questionnaire
Today we ask you to participate in four short studies. First you will read some newspaper
excerpts and answer some questions about what you read. Second, you will complete another
study and we will ask you some background questions about yourself. Third, we will ask you for
your opinions about some political issues. Finally, we will ask you for your opinions about some
different groups in society.
Information Manipulation
[RANDOMIZE INTO ONE OF FIVE CONDITIONS]
<CONTROL GROUP>
[SAME AS STUDY 1]
49
</CONTROL GROUP>
<TREATMENT CONDITIONS>
The first study you will participate in is about the quality of journalism. We are interested in how
well journalists write about the news. You will read a few short excerpts from a news article and
then we will ask you some questions to see how well you understood what was written.
We are asking different people to read articles on different topics. You are being asked to read
excerpts from an article about electronic medical records.
<PAGEBREAK/>
<Exercept1>
Here is the first part of the article. Please read it carefully.
DENVER, Colo. – Leaders of the American Medical Association have been busy at work this
week in an annual conference held here in Denver. On Tuesday, the AMA's policy advisory
committee began debate on a proposal that would change rules regarding how electronic medical
records are handled by the nation's hospitals and clinics. The proposal would standardize medical
record technology. If implemented, the proposal would mean
<CostHigh>
significantly higher costs for doctors and clinics. A typical clinic could see expenses rise by as
much as $5000 per year under the proposed changes. Nationwide, administrative costs related to
medical records could rise by as much as $17 million over the next two years.
</CostHigh>
<CostLow>
significant cost savings for doctors and clinics. A typical clinic could see expenses reduced by as
much as $5000 per year under the proposed changes. Nationwide, administrative costs related to
medical records could be lowered by as much as $17 million over the next two years.
</CostLow>
<BenefitsHigh>
that doctors, clinics, and hospitals would be able to share records more easily and be able to
better serve patient needs and more quickly retrieve vital health information. This would improve
the quality of health care for most Americans.
</BenefitsHigh>
<BenefitsLow>
that doctors, clinics, and hospitals would use standardized software so electronic records are
interoperable – meaning computer systems would rely on compatible formats. This would have
little impact on the quality of health care for most Americans.
</BenefitsLow>
<PAGEBREAK/>
<Comprehension>
What organization was mentioned in the text?
American Hospital Association
50
American Medical Association
Association of Nurse Practitioners
Doctors Without Borders
Don't know
</Comprehension>
</Excerpt1>
<Excerpt2>
Here is the next part of the article . Please read it carefully.
Electronic medical records have been the subject of debate within the medical field for more than
ten years. These systems would replace paper records kept by individual doctors and clinics with
sophisticated in-house computer systems. Numerous companies currently produce competing
software.
<CostHigh>
The push for electronic medical records is seen by many experts as an expensive alternative to
paper records. Startup costs for a new electronic medical records system can cost a five-physician
clinic $32,000 plus $8000-$9000 in annual maintenance costs.
</CostHigh>
<CostLow>
The push for electronic medical records is seen by many experts as an cost-saving alternative to
paper records. A new electronic medical records system can save a five-physician clinic $8000$9000 per year in operating costs.
</CostLow>
<BenefitsHigh>
The push for electronic medical records is seen by many experts as a clear way to improve the
quality of Americans' health care. With these systems, doctors can more easily track patient
health over time, and more easily catch risk factors for chronic and life-threatening diseases.
</BenefitsHigh>
<BenefitsLow>
The push for electronic medical records is seen by many experts as a clear way to keep pace with
changing technology. With these systems, medical coding is streamlined, simplifying recordkeeping processes for clinic staff, even if it will have no impact on patients' health.
</BenefitsLow>
<PAGEBREAK/>
<Comprehension>
According to the article, how long has the issue of electronic medical records been debated?
Less than 1 year
1-5 years
5-10 years
More than 10 years
Don't know
</Comprehension>
</Excerpt2>
51
<Excerpt3>
Here is the next part of the article. Please read it carefully.
The Brookings Institution, an independent policy think tank based in Washington, DC, released a
report this morning describing an in-depth analysis of electronic medical records in the United
States. At a press conference, Brookings spokesman Will Richardson said that the report details
how proposed rule changes currently being debated by the nation's leading hospitals, clinics, and
doctors would
<CostHigh>
lead to significantly higher costs for doctors and patients. Clinics have been slow to adopt
electronic medical records precisely because the costs are so high.
</CostHigh>
<CostLow>
lead to significantly lower costs for doctors and patients. Clinics have been slow to adopt
electronic medical records precisely because they have been waiting for these kinds of savings to
become available.
</CostLow>
<BenefitsHigh>
substantially improve the flexibility of the U.S. health care system, allowing patients and doctors
to quickly share vital health information. The changes would also mean simplified handling of
insurance claims, reducing hassle for American families.
</BenefitsHigh>
<BenefitsLow>
likely lead to modifications of existing clinical software and the future development of uniform
formatting requirements for electronic records. The changes would also mean modified rules for
software developers to comply with the new standards.
</BenefitsLow>
<PAGEBREAK/>
<Comprehension>
In what city is the Brookings Institution located?
New York City, New York
Washington, DC
Seattle, Washington
Denver, Colorado
Don't know
</Comprehension>
</Excerpt3>
<Excerpt4>
Here is the last part of the article. Please read it carefully.
Back in Denver, the AMA conference continues through the end of the week where electronic
medical records seem to be the big topic of debate among conference attendees. Both
practitioners and technology developers are eager to see the policy advisory committee's
recommendations, which are due by the end of the week.
<CostHigh>
52
One doctor, Jan Miller of Oakland, CA, remained concerned about her clinic's bottom line.
“Electronic medical records are a substantial change in clinical practice and my practice cannot
afford such a system without passing costs on to patients.”
</CostHigh>
<CostLow>
One doctor, Jan Miller of Oakland, CA, was confident that electronic medical records would
reduce costs her in practice. “Electronic medical records are a substantial change in clinical
practice and my patients are asking for this system because they know it will save them money.”
</CostLow>
<BenefitsHigh>
One doctor, Jan Miller of Oakland, CA, was confident that electronic medical records would
improve her patients' health. “Electronic medical records are a substantial change in clinical
practice and patients will see the benefits of these systems right away. Electronic records help me
make fewer errors and know my patients better.”
</BenefitsHigh>
<BenefitsLow>
One doctor, Jan Miller of Oakland, CA, downplayed the importance of this week's debates.
“Electronic medical records mean little for my practice. It's really same old, same old. I don't see
how patients would be affected really.”
</BenefitsLow>
<PAGEBREAK/>
<Comprehension>
What was the profession of Jan Miller, the individual quoted in the article?
Software developer
Nurse
Doctor
Insurance agent
Don't know
</Comprehension>
</Excerpt4>
Information Manipulation Check
[SAME AS STUDY 1]
Distractor Activity
[SAME AS STUDY 1]
Demographics
[SAME AS STUDY 1]
Framing Manipulation
The third study asks you for your opinion about a recent political issue.
53
Recently there has been some debate in Congress about a new proposal regarding electronic
medical records.
[RANDOMIZE INTO ONE OF TWO FRAME CONDITIONS]
<CostFrame>
Some support the proposal. Others are opposed because they say that we should judge the
proposal based on whether it is costly. Indeed, much of the debate over the proposal now
revolves around the question of costs.
</CostFrame>
<BenefitsFrame>
Some support the proposal. Others are opposed because they say that we should judge the
proposal based on whether it will affect the health of average Americans. Indeed, much of the
debate over the proposal now revolves around the question of the proposal's impact on patients'
health.
</BenefitsFrame>
Outcome Measures
<CostFrame>
Based on whether you think the proposal will mean higher or lower costs, to what extent do you
favor or oppose this proposal?
Strongly favor
Moderately favor
Somewhat favor
Neither favor nor oppose
Somewhat oppose
Moderately oppose
Strongly oppose
</CostFrame>
<BenefitsFrame>
Based on whether you think the proposal will have a large or small impact on patients' health, to
what extent do you favor or oppose this proposal?
Strongly favor
Moderately favor
Somewhat favor
Neither favor nor oppose
Somewhat oppose
Moderately oppose
Strongly oppose
</BenefitsFrame>
How certain are you of your opinion on the proposal regarding electronic medical records?
Not at all certain
54
Somewhat certain
Moderately certain
Very certain
Extremely certain
How important to you personally is your opinion about the proposal regarding electronic medical
records?
Not at all important
Somewhat important
Moderately important
Very important
Extremely important
Framing Manipulation Check
We are now going to list a few ideas that individuals have expressed when describing their
opinions about the proposal regarding electronic medical records. Some of these ideas may seem
important to you, while others may seem less important. Please tell us how important each of
these ideas is to you when thinking about your overall opinion of this proposal.
[GRID]
Improving technology
Costs
Errors in medical records
The health of average Americans
The opinions of doctors
Patient privacy
[GRID]
Not at all important
Somewhat important
Moderately important
Very important
Extremely important
Do opponents say we should judge the proposal according to...
Its costs
Its impact on health
Both
Neither
Don't know
Appendix 4: Study 3 Treatments and Questionnaire
Today we ask you to participate in four short studies. First you will read some newspaper
excerpts and answer some questions about what you read. Second, you will complete another
study and we will ask you some background questions about yourself. Third, we will ask you for
55
your opinions about some political issues. Finally, we will ask you for your opinions about some
different groups in society.
Information Manipulation
[RANDOMIZE INTO ONE OF FIVE CONDITIONS]
<CONTROL GROUP>
[SAME AS STUDY 1]
</CONTROL GROUP>
<TREATMENT CONDITIONS>
The first study you will participate in is about the quality of journalism. We are interested in how
well journalists write about the news. You will read a few short excerpts from a news article and
then we will ask you some questions to see how well you understood what was written.
We are asking different people to read articles on different topics. You are being asked to read
excerpts from an article about electronic medical records.
<PAGEBREAK/>
<Exercept1>
Here is the first part of the article. Please read it carefully.
DENVER, Colo. – Leaders of the American Medical Association have been busy at work this
week in an annual conference held here in Denver. On Tuesday, the AMA's policy advisory
committee began debate on a proposal that would change rules regarding how electronic medical
records are handled by the nation's hospitals and clinics. The proposal would standardize medical
record technology. If implemented, the proposal would mean
<BenefitsHigh>
that doctors, clinics, and hospitals would be able to share records more easily and be able to
better serve patient needs and more quickly retrieve vital health information. This would improve
the quality of health care for most Americans.
</BenefitsHigh>
<BenefitsLow>
that doctors, clinics, and hospitals would use standardized software so electronic records are
interoperable – meaning computer systems would rely on compatible formats. This would have
little impact on the quality of health care for most Americans.
</BenefitsLow>
It would also mean
<CostHigh>
significantly higher costs for doctors and clinics. A typical clinic could see expenses rise by as
much as $5000 per year under the proposed changes. Nationwide, administrative costs related to
medical records could rise by as much as $17 million over the next two years.
</CostHigh>
56
<CostLow>
significant cost savings for doctors and clinics. A typical clinic could see expenses reduced by as
much as $5000 per year under the proposed changes. Nationwide, administrative costs related to
medical records could be lowered by as much as $17 million over the next two years.
</CostLow>
<PAGEBREAK/>
<Comprehension>
What organization was mentioned in the text?
American Hospital Association
American Medical Association
Association of Nurse Practitioners
Doctors Without Borders
Don't know
</Comprehension>
</Excerpt1>
<Excerpt2>
Here is the next part of the article. Please read it carefully.
Electronic medical records have been the subject of debate within the medical field for more than
ten years. These systems would replace paper records kept by individual doctors and clinics with
sophisticated in-house computer systems. Numerous companies currently produce competing
software.
<CostHigh>
The push for electronic medical records is seen by experts as an expensive alternative to paper
records. Startup costs for a new electronic medical records system can cost a five-physician clinic
$32,000 plus $8000-$9000 in annual maintenance costs.
</CostHigh>
<CostLow>
The push for electronic medical records is seen by experts as a cost-saving alternative to paper
records. A new electronic medical records system can save a five-physician clinic $8000-$9000
per year in operating costs.
</CostLow>
<BenefitsHigh>
Many also see it as a clear way to improve the quality of Americans' health care. With these
systems, doctors can more easily track patient health over time, and more easily catch risk factors
for chronic and life-threatening diseases.
</BenefitsHigh>
<BenefitsLow>
Many also see it as a clear way to keep pace with changing technology. With these systems,
medical coding is streamlined, simplifying record-keeping processes for clinic staff, even if it
will have no impact on patients' health.
</BenefitsLow>
57
<PAGEBREAK/>
<Comprehension>
According to the article, how long has the issue of electronic medical records been debated?
Less than 1 year
1-5 years
5-10 years
More than 10 years
Don't know
</Comprehension>
</Excerpt2>
<Excerpt3>
Here is the next part of the article. Please read it carefully.
The Brookings Institution, an independent policy think tank based in Washington, DC, released a
report this morning describing an in-depth analysis of electronic medical records in the United
States. At a press conference, Brookings spokesman Will Richardson said that the report details
how proposed rule changes currently being debated by the nation's leading hospitals, clinics, and
doctors would
<BenefitsHigh>
substantially improve the flexibility of the U.S. health care system, allowing patients and doctors
to quickly share vital health information. The changes would also mean simplified handling of
insurance claims, reducing hassle for American families.
</BenefitsHigh>
<BenefitsLow>
likely lead to modifications of existing clinical software and the future development of uniform
formatting requirements for electronic records. The changes would also mean modified rules for
software developers to comply with the new standards.
</BenefitsLow>
The report also expects the change to
<CostHigh>
lead to significantly higher costs for doctors and patients. Clinics have been slow to adopt
electronic medical records precisely because the costs are so high.
</CostHigh>
<CostLow>
lead to significantly lower costs for doctors and patients. Clinics have been slow to adopt
electronic medical records precisely because they have been waiting for these kinds of savings to
become available.
</CostLow>
<PAGEBREAK/>
<Comprehension>
In what city is the Brookings Institution located?
58
New York City, New York
Washington, DC
Seattle, Washington
Denver, Colorado
Don't know
</Comprehension>
</Excerpt3>
<Excerpt4>
Here is the last part of the article. Please read it carefully.
Back in Denver, the AMA conference continues through the end of the week where electronic
medical records seem to be the big topic of debate among conference attendees. Both
practitioners and technology developers are eager to see the policy advisory committee's
recommendations, which are due by the end of the week.
<CostHigh>
One doctor, Jan Miller of Oakland, CA, remained concerned about her clinic's bottom line.
"Electronic medical records are a substantial change in clinical practice and my practice cannot
afford such a system without passing costs on to patients."
</CostHigh>
<CostLow>
One doctor, Jan Miller of Oakland, CA, was confident that electronic medical records would
reduce costs her in practice. "Electronic medical records are a substantial change in clinical
practice and my patients are asking for this system because they know it will save them money."
</CostLow>
<BenefitsHigh>
But when asked about how patients' health would be impacted, Dr. Miller said that "patients will
see the benefits of these systems right away. Electronic records help me make fewer errors and
know my patients better."
</BenefitsHigh>
<BenefitsLow>
But when asked about how patients' health would be impacted, Dr. Miller said that "patients will
see limited benefits with these systems. It's really same old, same old. I don't see how patients
would be affected really."
</BenefitsLow>
<PAGEBREAK/>
<Comprehension>
What was the profession of Jan Miller, the individual quoted in the article?
Software developer
Nurse
Doctor
Insurance agent
Don't know
</Comprehension>
</Excerpt4>
59
<PAGEBREAK/>
Information Manipulation Checks
[SAME AS STUDIES 1 AND 2]
Distractor Activity
[SAME AS STUDIES 1 AND 2]
Demographics
[SAME AS STUDIES 1 AND 2]
Framing Manipulation
[SAME AS STUDY 2]
Framing Manipulation Check
[SAME AS STUDIES 1 AND 2]
60