Download Report

PCORI’s Process for Peer Review of Primary
Research and Public Release of Findings
Key Findings from Public Comments and
Proposed Final Revisions
Prepared for PCORI by American Institutes for Research
February 17, 2015
i
This report was prepared under PCORI RFQ # PCO-CVDLDSR2013.004
Acknowledgments
PCORI and the American Institutes for Research (AIR) wish to thank the members of the
Stakeholder Advisory Group for their guidance throughout the public comment process, as well
as the stakeholders who participated in PCORI’s live and online events about the proposed
process during the public comment period.
PCORI is appreciative of the efforts of Thomas Workman, PhD and Marla L. Clayman, PhD, MPH
and the team at AIR for their work on collecting and analyzing the public comments.
ii
Overview
On September 15, 2014, the Patient-Centered Outcomes Research Institute (PCORI) Board of Governors
approved for posting for public comment PCORI’s draft Proposed Process for Peer Review of Primary
Research and Public Release of Findings. On the same day, PCORI opened a 54-day public comment
period for the proposed process, developed in response to the section of PCORI’s authorizing legislation
that states in part that “The institute shall ensure that there is a process for peer review of primary
research” and provides a set of additional requirements for the release of research findings.
From September 15 to November 7, 2014, the American Institutes for Research® (AIR®), as contracted by
PCORI, collected public comments on the proposed process. A total of 63 unique submissions and 832
unique comments were received in addition to comments from the National Institutes of Health (NIH)
and the PCORI Methodology Committee.
Researchers, patients, patient or caregiver advocacy organizations, and clinicians made up the majority
of those making comments, accounting for 86% of the total submissions (32%, 11%, 21%, and 22%,
respectively). No other group contributed more than 10% of total comments submitted; additional
comments came from professional journals, the life sciences industry, and individual family members or
caregivers.
PCORI also received comments and recommendations from the PCORI Methodology Committee and the
NIH. These comments were processed separately as they were not part of the public-commentcollection process. Several of these comments, however, echoed sentiments expressed by the collected
public comments.
Although there was a substantial degree of agreement with the proposed process across all stakeholder
groups, five major themes emerged from an analysis of the comments following a systematic
assessment process, resulting in nine recommendations for revision that were reflected across the
comments.
Revisions to the draft process document were made based on these recommendations and the updated
draft presented to the PCORI Board of Governors on February 24, 2015 to consider its adoption.
iii
Contents
Overview ................................................................................................................................................ iii
Introduction ............................................................................................................................................ 1
Methods of Analysis………………………………………………………………………………………………………………………………..1
Stakeholder Advisory Group…………………………………………………………………………………………………………………….4
Major Findings......................................................................................................................................... 6
Quantity and Source of the Comments ................................................................................................ 6
Comments per Proposal Section .......................................................................................................... 8
Areas of General Agreement................................................................................................................ 8
Themes from the Comments of Disagreement or Concern ................................................................. 11
Suggestions for Revision and PCORI Responses………………………………………………………………………………….13
iv
Introduction
The Patient-Centered Outcomes Research Institute (PCORI) drafted its Proposed Process for Peer Review
of Primary Research and Public Release of Findings to meet its statutory mandate—which states in part
that “The institute shall ensure that there is a process for peer review of primary research.”
The dual requirements in PCORI’s authorizing law for peer review of primary research and making
research findings publicly available within a specific timeframe are not directly linked in the legislation
and pose challenges that must be resolved if PCORI is to comply with both. The proposed process
drafted by PCORI and revised in response to public comment is intended to balance the challenges of
ensuring the timely release of research results while assessing their scientific integrity in a way that
respects the interests and needs of patients, other healthcare stakeholders, our awardees, and the
general public, as well as recognizes the crucial role of peer-reviewed journals in disseminating new
evidence to the scientific and clinical communities.
The mandate for peer review underscores the importance of ensuring that primary research funded by
PCORI be critically appraised for scientific integrity as well as for adherence to PCORI’s Methodological
Standards before results are released. Such a peer-review process adds to the credibility, authority, and
trustworthiness of PCORI-funded research findings.
On September 14, 2014, the PCORI Board of Governors approved for posting for public comment the
draft proposed process document. On the same day, PCORI opened a 54-day public comment period for
the proposed process. Public comments were collected from September 15 to November 7, 2014.
PCORI hosted a live public forum on September 29, 2014 and a webinar on October 29, 2014 featuring
stakeholder feedback on the proposed process.
PCORI contracted with the American Institutes for Research® (AIR®) to create the web-based tool used to
collect public comment and provide the analysis of the comments on the proposed process.
A total of 63 unique submissions and 832 unique comments were received in addition to comments
from the National Institutes of Health (NIH) and the PCORI Methodology Committee.
Methods of Analysis
Submissions were pulled as they appeared from the online tool (pcori.airprojects.org/node/1/webformresults/download) and entered into a coding table. Repeated submissions (two submissions with the
same content) were combined into a single entry, and repeated comments within a single submission
were eliminated for a final cleaned copy of the comment table. Elements of the letterhead (when used),
PCORI or return address, salutation, or signature line were removed. This table was used for the purpose
of coding and analysis. All counts of submissions and comments reflect the combination of duplicate
files.
PCORI’s Process for Peer Review of Primary Research and Public Release of
Findings
1
General Review
Comments as a whole were reviewed in their entirety. Analysts also reviewed the autocoded results of
all Likert-scale responses and compared them to the comments provided by individual respondent and
by stakeholder type. Analysts were able to get a clear feel of the comments in general to guide the
creation of a code book to organize and process comments.
Coding
A Code Book and Coding Scheme were developed to help organize the comments and identify the
suggestions for revision found in the comments as a whole. A team of four qualitative researchers coded
each of the comments. A set of comments was coded by multiple staff members to test agreement
across the coding staff. This helps determine how reliable the coding of comments is across the entire
set. Agreement across coding staff exceeded 90%, so no changes or clarifications to the code book were
necessary.
All comments were coded by the coding team for 10 general classifications:
1. Unique Identification. A number was used to count individual comments by submission.
Comments were separated and numbered by the order in which they were received.
2. Stakeholder Self-Identification. Codes were created for each of the stakeholder categories
provided in the online collection tool. A second code was created in cases where a stakeholder
category was not selected to indicate if the comment did or did not reflect a
knowledge/familiarity of the research process; however, this code was not necessary and
therefore was not used.
3. Relevance to the Report. Comments were coded to classify their relevance to (1) the report as a
whole, (2) a specific section, or (3) the overall process. Comments that did not relate to the
report in any of these ways were also coded appropriately.1 Comments were also autocoded
with the question in which the response was submitted.
4. Level of Agreement. All comments that related to the report were coded, based on their stated
or interpreted agreement with the section of the Proposal. Review of the entire set of
comments found that an additional code was needed, as a number of comments stated general
agreement but also mentioned a small concern. These comments were coded as “Agree With
Caution.” Coders also identified comments where agreement or disagreement was not made
clear.
5. Affect. All comments were also coded for the affect, or emotional direction, of the comment:
positive, negative, or indifferent.
Additional codes were used to organize the comments expressing disagreement or concern so that a
clear set of revision recommendations could be identified. The following codes were used for this
purpose:
1
Comments considered unrelated to the report may still be related to PCORI in general or be of interest to its
work.
PCORI’s Process for Peer Review of Primary Research and Public Release of
Findings
2
6. Suggested Action/Direction. Comments were coded to classify the suggested action directly
stated or implied within the comment using the following distinctions:
a. Add something: The comment stated or implied that an additional factor or item be added
to a section of the process.
b. Remove something: The comment stated or implied that some factor or item be removed
from a section of the process.
c. Modify something: The comment stated or implied that a factor or item in a section of the
process needed to be revised or modified.
d. No action indicated. This code was used when no stated or implied action was provided in
the comment.
e. Action suggested is unrelated to the process. This code was used when a comment stated or
implied an action that was unrelated to any section of the process.
f.
Action is unclear. This code was used when an action was implied but written in such a way
that its direction could not be interpreted.
7. Feasibility. Initial review of the comments as a whole found that issues of feasibility regarding
the process were often implied. Therefore, coders identified comments where feasibility
appeared to be the central tenet of the disagreement. Comments relating to the feasibility of
the proposed process were coded with the feasibility code as:
a. Process as a whole is not feasible.
b. Specific section of process is not feasible.
c. Legal mandate is not feasible.
i. The feasibility concern focused on peer-review system capacity.
ii. The feasibility concern focused on PCORI staffing capacity.
iii. The feasibility concern focused on the capacity of academic journals.
iv. The feasibility concern focused on researcher capacity.
8. Language and Writing Style. Comments focused on the style of the proposed process, including
organization, format, length, or word choice/phrasing.
Two additional codes were used to identify comment content important to PCORI:
9. Compliment or Support. All comments that indicated support for PCORI or included
compliments were coded and marked as support for PCORI, the process as a whole, or a specific
section.
10. Inquiry. All specific requests for clarification were also coded.
Additional Analysis to Determine Themes
The complete set of separated and coded comments was entered into NVivo 10 software. Reports of
comments per code were reviewed to determine:

General word frequency (without codes)
PCORI’s Process for Peer Review of Primary Research and Public Release of
Findings
3

Comments by agreement and disagreement codes

Comments by direction and affect of disagreement (recommended action and feasibility)

Comments of appreciation or complimentary to the process
Coded lists of comments were then reviewed for additional analysis. A conceptual map was developed
to further clarify the nature of the comments and their relationship to stakeholders’ interests. Analysts
then began the process of developing specific themes on the basis of four criteria:
1. The prominence of the theme across comment submissions. A theme was considered prominent
if it was raised in three or more of the submissions, particularly if repeated multiple times in
comments throughout each submission.
2. The clarity of the comment toward providing a recommendation for revision. Analysts were
particularly mindful of the suggested action or intent of the comments, and they used thematic
analysis to focus the nature of the direction of revisions, especially as revisions related to
specific stakeholder interests.
3. Tensions between stakeholder perspectives. Tensions—differing perspectives or interests,
usually aligned with specific stakeholder groups that appear to be oppositional to one another—
are an important aspect of comment analysis. As is typical in the collection and analysis of public
comments, tensions between stakeholder groups often emerge and need to be highlighted so
that revision recommendations can be developed to bridge differing perspectives.
4. The “weight” of the comment’s source. “Weight” refers to the level of importance that a
comment has in the formation of revision recommendations. A comment is given additional
weight when it represents the collective voice of a larger constituency, such as an organization
or society. Although AIR believes that each comment should be considered, regardless of its
source, analysts also appreciate that comments coming from an individual or single corporation
must be weighted differently from those of a professional association representing multiple
(and in some cases, many) stakeholders. A full table of comments with sources identified has
been made available to PCORI should the Committee choose to consider source more explicitly
in determining revisions.
Stakeholder Advisory Group
Nine stakeholders representing different stakeholder groups were invited to serve as stakeholder
advisors for the project. Group members were chosen from PCORI contacts who had participated in
public events about the proposed process or who PCORI knew were planning to submit comments about
the process document. This group was assembled to ensure an additional level of detailed stakeholder
input into the analysis of the public comments.
On December 15, 2014, members of the advisory group reviewed the synthesis of the public comments
and major recommendation themes, participated in an adapted Affinity Diagramming exercise, and
offered both feedback and suggestions for their improvement. These suggestions have been
incorporated in this report. Members of the Stakeholder Advisory Group are listed in Table 1.
PCORI’s Process for Peer Review of Primary Research and Public Release of
Findings
4
Table 1. Participants in the PCORI Methodology Report Patient Panel
Participant Information
Brief Biography
Marc Boutin, JD
Mr. Boutin is the Executive Vice President and Chief Operating Officer of
the National Health Council—an organization which provides a united
voice for millions of people with chronic diseases and disabilities and their
family caregivers. He provides guidance to patient organizations on
various association issues, including corporate structure, government
relations, fundraising, and outreach, and he is a regular spokesperson
before the media, Congress, and policy makers on major issues of interest
to the patient community.
Rachel Hess, MD, MS
Dr. Hess is a professor of internal medicine at the University of Utah. Her
research focuses on understanding and improving patient-centered
outcomes, such as health-related quality of life. She also directs the
University of Utah’s Health System Innovation and Research Program.
Rebecca Burkholder, JD
Ms. Burkholder is Vice President of Health Policy at the National
Consumers League—a national, nonprofit organization that has
represented consumers and workers since 1899. Ms. Burkholder
coordinates the League’s work on various healthcare issues, including safe
use of medication, patient safety, doctor–patient communication, and
direct-to-consumer advertising. She is responsible for research on these
issues, producing consumer information, and advocating for system
changes.
John Schall, MPP
Mr. Schall is Chief Executive Officer of the Caregiver Action Network. He is
a public policy and communications professional with expertise in a wide
range of policy fields, including health care, labor, education, economic
development, taxation, and budget policy.
David Bates, MD, MSc
Dr. Bates is the Medical Director of Clinical and Quality Analysis at Partners
HealthCare. He also serves as Chief of the Division of General Internal
Medicine and Primary Care in the Department of Medicine at Brigham and
Women’s Hospital and is a professor of medicine at Harvard Medical
School. His primary informatics focus concerns the use of information
technology to measure and improve the quality of care within the field of
medication safety.
Suzanne Belinson, PhD,
MPH
Dr. Belinson is the Executive Director for Clinical Effectiveness at the
BlueCross BlueShield Association (BCBSA). She is part of the leadership
team in BCBSA’s Office of Clinical Affairs, which focuses on developing
emerging programs and services that enhance the clinical effectiveness for
the independent BCBS Plans. Before joining the BCBSA, Dr. Belinson was at
Northwestern University, where she focused on community-based cervical
cancer prevention strategies for the United States and new screening and
diagnostic test development for the developing world.
PCORI’s Process for Peer Review of Primary Research and Public Release of
Findings
5
Participant Information
Brief Biography
Jennifer Graff, PharmD
Dr. Graff is the National Pharmaceutical Council’s (NPC) Vice President of
Comparative Effectiveness Research (CER). Dr. Graff works to advance
NPC’s CER and evidence-based medicine policy research initiatives
assessing the methods, interpretation, and application of CER.
Christine Laine, MD, MPH, Dr. Laine is Editor in Chief of Annals of Internal Medicine and a Senior Vice
FACP
President at the American College of Physicians. She is a practicing
physician in Philadelphia and is board-certified in internal medicine. She is
active in the field of medical journalism and holds leadership positions in
the International Committee of Medical Journal Editors, the Council of
Science Editors, and the ethics committee of the World Association of
Medical Editors. She has been instrumental in the development of
editorial policy on issues such as authorship and conflicts of interest in
medical research.
Chandra Branham, JD
Ms. Branham is Vice President of Payment and Health Care Delivery at the
Advanced Medical Technology Association (AdvaMed). She leads
AdvaMed’s policy work in a number of areas, including diagnostics
payment, coverage issues, and comparative effectiveness.
Major Findings
Quantity and Source of the Comments
A total of 63 individuals or organizations submitted comments online at the close of the open comment
period (midnight ET, November 7, 2014). Comments were provided in a formal letter by 25
organizations. PCORI also received formal comments and suggestions from the NIH and the PCORI
Methodology Committee; these were not included in the public comment count or statistical analyses
and were analyzed separately.
Collectively, the 63 unique submissions comprised a total of 832 unique comments. Comments were
identified by stakeholder category as indicated by the individual or organization submitting comments.
Figure 1 provides a chart of the percentages of total comments by self-identified stakeholder category.
PCORI’s Process for Peer Review of Primary Research and Public Release of
Findings
6
Figure 1. Number and Percentage of Submissions by Stakeholder Category 2
Submissions by Stakeholder Category (n=63)
Family member or
Caregiver, 1, 2%
Journal/Publisher,
3, 5%
Life Sciences
Industry,
5, 8%
Researcher, 20, 32%
Clinician/Clinician
Society, 14, 21%
Patient/Caregiver
Advocacy
Organization, 13,
21%
Patient, 7, 11%
Legend:
The total number of submissions
for each stakeholder group is
followed by the percentage.
The following identifiers were not selected by any of the individuals’ or organizations’ submissions:

Payer/Insurer

Training Institution

Hospital/Health System

Purchaser/Business

Organizational Provider

Policy Maker
2
Stakeholder identification as it is referenced throughout this report is self-identified. However, several
classifications were changed to better reflect the perspective of the submission. In several submissions, the name
of the individual or organization provided within the comment indicated a contradicting classification. Specifically,
several submissions representing an individual’s perspective were self-identified as coming from a patientadvocacy organization or a training institution. These submissions were carefully scrutinized and corrected. AIR
wishes to honor the self-identification process but also recognizes the importance of actual representation of
stakeholder categorization when providing weights for thematic analysis.
PCORI’s Process for Peer Review of Primary Research and Public Release of
Findings
7
Comments per Proposal Section
Comments were coded to identify the section of the process. Coding staff assessed the autocoding
function and made adjustments via manual coding if the comment was not truly relevant to the section
of the process as it was entered into the intake form. Table 2 reflects the accurate accounting of
comments received, per section of the Proposal.
The proposed peer-review process received the highest number of comments (119).
Table 2. Comment Totals by Section
Section
Submissions
Comments
Related to Process as a Whole
35
68
Related to a Specific Section
60
466
Registration at ClinicalTrials.gov, RoPR, or PROSPERO
36
65
Production of a Draft Final Report, Scientific Abstract, and StandAlone Results Table
33
76
Proposed Peer-Review Process
45
119
Providing a Lay-Friendly Summary
40
72
Submission of Results to ClinicalTrials.gov
21
25
Posting of Full Final Reports on PCORI’s Website
37
76
Notification to PCORI of Submission to Journals and Publication
24
33
Related to the Process Overall
47
253
Timeline of Process
31
59
How Well the Proposal Meets PCORI’s Obligations
19
26
How Well the Proposal Provides a Clear Understanding of the
Process
14
14
Feasibility of the Proposal
18
25
Sequencing of the Proposal
26
36
Patient Centeredness
7
27
Research Prioritization
7
16
Translation Table and Registries
1
2
Dissemination of Research
15
48
Areas of General Agreement
The majority of the submissions responded in agreement to the statements provided at the end of the
comment collection form, with an average of 57% of responses indicating the first and second highest
indicators of agreement. Responses to each Likert-scale assessment are provided in Figures 2 through 4.
PCORI’s Process for Peer Review of Primary Research and Public Release of
Findings
8
Figure 2. Responses to Rating Scale #1 (Meets Obligations)
How Well the Proposal Meets PCORI's Obligations Under its
Authorizing Legislation
No Response, 15,
24%
Very Well, 20, 32%
Very Poorly, 1, 1%
Somewhat
Poorly, 3, 5%
Neither Well nor
Poorly, 9, 14% Somewhat Well,
15, 24%
Legend:
The total number of submissions for each
stakeholder group is followed by the percentage.
Figure 3. Responses to Rating Scale #2 (Provides Clarity)
How Well the Proposal Provides a Clear Understanding of the Peer
Review and Public Release Process
No Response, 17,
27%
Very Clear, 25, 40%
Very Unclear, 0,
0%
Somewhat
Unclear, 4, 6%
Neither Clear nor
Unclear, 5, 8%
Somewhat Clear,
12, 19%
Legend:
The total number of submissions for each
stakeholder group is followed by the percentage.
PCORI’s Process for Peer Review of Primary Research and Public Release of
Findings
9
Figure 4. Agreement with Sequencing
Do You Agree with the Sequencing of the Peer Review and Public Posting
of Results Process
No Response,
16, 25%
Strongly Agree,
22, 35%
Strongly Disagree,
0, 0%
Somewhat
Disagree,
5, 8%
Somewhat Agree,
15, 24%
Neither Agree nor
Disagree, 5, 8%
Legend:
The total number of submissions
for each stakeholder group is
followed by the percentage.
In addition, the majority of actual comments collected demonstrate a strong sense across stakeholder
groups of general agreement with the proposed process, as Figure 5 shows.
Figure 5. Indications of Agreement per Comment
Indications of Agreement Per Comment
Indicated
Disagreement, 57,
12%
Indicated
Agreement with
Caution, 189, 39%
Indicated
Agreement, 239,
49%
Legend:
The total number of submissions
for each stakeholder group is
followed by the percentage.
PCORI’s Process for Peer Review of Primary Research and Public Release of
Findings
10
Themes
Emerging Themes/Overarching Issues
Themes from the public comments were developed to better understand the comments that expressed
disagreement or concern about one or more sections of the process.
In general, the comments were respectful, expressing recognition and appreciation for PCORI’s efforts
to make research publicly available. Some differences did emerge, however, between members of the
research community and those who identified as patients/caregiver advocacy organizations. These
differences, reflected in the conceptual map of the comment themes, are shown in Figure 6.
Emerging Themes Surrounding Caution or Disagreement
Five unique themes emerged from the analysis of the comments identified as indicating caution or
disagreement. As described above, the themes were derived by considering the source, frequency, and
intensity.
1. Unclear details concerning the conduct of peer review
2. Feasibility and threats to accuracy
3. Excessive burdens on researchers
4. Dissemination of the research findings
5. Lay-abstract authorship, readability, and utility
These five themes are explored in detail with PCORI’s responses in Table 3. These themes, however, do
not represent the entirety of all comments, which spanned a wide range of individual reactions across a
variety of subjects. Differences between comments from researchers and comments from
patients/caregivers/advocacy organizations were found with respect to composition of reviewer lists,
writing of lay abstracts, and dissemination of findings.
Table 3. Themes from the comments of Disagreement or Concern
Aggregate Theme
Unclear details concerning
the conduct of peer review
Components of the Theme
•
•
•
•
•
The peer-reviewer selection process and concerns about what
determines “content” expertise
The ways in which the peer-review process will be
standardized across multiple reports
The ways in which peer reviews will be quality controlled
Role of patients, caregivers, or their advocates in the process
The inclusion of a Methodological Review in the process
PCORI’s Process for Peer Review of Primary Research and Public Release of
Findings
11
Feasibility and threats to
accurate interpretation of
results
•
•
The resolution of conflicts between peer reviewers, the
review panel and the authors, or between the PCORI peer
review and a publisher’s peer review
Potential misinterpretation or inconsistent reporting of study
findings between different abstracts and publications
Excessive burdens the
process may place on
researchers
•
•
Potential for multiple rounds of review and revisions
Potential for additional work requested after the end of the
funding period
Dissemination of the
research findings
•
•
Lay abstract authorship,
readability, and utility
•
•
Public release process is limited for widespread dissemination
Unclear connection of process to broader PCORI
Dissemination Plan
Proposed reading level of 8th grade too high
Role of patients, caregivers, or their advocates in the
authorship process
Length of abstract may be insufficient
•
PCORI’s Response to Public Comments: Summary and Conclusion
Suggestions for revision were organized by theme and reviewed to determine a list of nine specific areas
for revision of the proposed process. Table 4 provides a summary of the most significant revision
suggestions, with PCORI's responses and proposed revisions to the draft language.
Table 4. Suggestions for Revision and PCORI’s Response
What We Heard
Incorporate patients and
other stakeholders into the
peer review, lay abstract,
and dissemination
processes.
PCORI’s Response
PCORI is committed to including
patients and other stakeholders
at every stage of the research
and dissemination processes.
We have modified the
document accordingly.
Draft Revision Language
Reviewers for a particular draft final
report will include the
Methodologist and/or
biostatistician, External subject
matter experts, Patient/caregiver
and other stakeholder reviewers: At
least one patient and/or caregiver
with experience with the disease or
condition relevant to the study will
be invited to serve as a reviewer,
commenting on the study’s
relevance and usefulness. The goal
will be to determine if
patient/caregiver perspectives,
values, and preferences were
adequately considered in the
report’s summary of the study’s
results and to confirm that the
results are useful to them in making
PCORI’s Process for Peer Review of Primary Research and Public Release of
Findings
12
decisions about care options. As
appropriate, a representative of
another stakeholder group, such as
payers, employers, the life sciences
industry, or policy makers, also may
be invited to comment as part of the
review process.
Creating and posting the 500-word
(lay) abstract for patients,
consumers, and the general public
addresses…
a. PCORI will select a qualified and
experienced contractor to
develop all of the lay abstracts
in a manner and format that is
consistent.
b. The contractor will be required
to conduct cognitive and focus
testing, ensure that the
abstracts are written at an
appropriate reading level, and
include patients to ensure that
the findings are
comprehensible.
Establish a clear process
PCORI recognizes that the
There may be times when there is a
for the resolution of
conclusions in the peermaterial difference of opinion
potential disagreements or reviewed final report may differ between the Awardee’s Principal
differences in
from what appears in journal
Investigator and PCORI about
interpretation between
publications, for a variety of
reviewer comments or proposed
required changes to a final reasons. PCORI will work closely revisions … the final report as
report by the PCORI peer
with investigators to update
revised by agreement between
review and those required material posted on PCORI.org to PCORI and the Awardee will be
by a journal review.
note any updates in findings
posted on PCORI.org, along with the
based on publications. To the
anonymized comments of the peer
Make the entire review
fullest extent possible, PCORI’s
reviewers and the Awardee’s
process transparent so that process will be coordinated
response.
reviewer comments and
with a journal’s review process.
researcher responses are
available to the public.
PCORI agrees that for purposes
of transparency, anonymized
reviewer comments, and
Awardees’ responses, will be
posted as part of PCORI’s
review process.
PCORI’s Process for Peer Review of Primary Research and Public Release of
Findings
13
Create a standardized
approach to the selection
of peer reviewers and the
peer-review process as a
whole. Include details in
the process about how
PCORI will select the peer
reviewers or use a vendor
to provide peer review of
the reports.
Clarify or revise the
process to indicate the
relationship between this
process and PCORI’s
broader plan and efforts to
disseminate research
findings.
PCORI agrees that more detail
about the review process is
useful and warranted.
PCORI shall engage a qualified
contractor, who will be closely
managed by PCORI staff, to
administer the peer review of draft
final reports.
Reviewers for a particular draft final
report will include the
Methodologist and/or
biostatistician, External subject
matter experts, Patient/caregiver
and other stakeholder reviewers.
PCORI agrees that it would be
helpful to clarify how the peer
review and public release
process fits within PCORI’s
broader plans for disseminating
the results of its funded studies
and has added such language.
Currently a draft dissemination
and implementation framework
and toolkit is posted on
PCORI.org; additional updates
and planning documents will
posted as they are available.
The Principal Investigator shall
recommend up to two subject
matter experts for PCORI to
consider inviting to participate in
the review of the final report.
Clinical, scientific, and technical
experts from drug and device
manufacturers may be among those
chosen as methodologists or
content experts.
This draft document outlines
PCORI’s proposed process for
fulfilling its statutory mandate to
develop and implement a process
for peer-reviewing its primary
research and making research
findings publicly available in a form
and format useful to patients,
clinicians, and others. This process is
envisioned as one element within a
far more extensive and integrated
effort to disseminate the results of
PCORI-funded research to
stakeholders across the healthcare
community.
We are planning and will implement
this broader dissemination and
implementation strategy in close
collaboration with the Agency for
Healthcare Research and Quality
(AHRQ), as outlined in our
authorizing legislation, as well as
through the community of
healthcare stakeholders, both
PCORI’s Process for Peer Review of Primary Research and Public Release of
Findings
14
Clarify the purpose and
approach to the
methodological review of
the report.
PCORI agrees and has added
such language.
Reduce the required
reading level of the lay
abstract to below 8th
grade.
PCORI recognizes the need to
disseminate information that is
accessible by those with limited
health literacy and has modified
its requirement for readability
accordingly.
Assure that the abstract
and tables are not prior
publication.
PCORI recognizes the
importance of these concerns
and has taken several steps to
address them. These include
clarifying that the International
Committee of Medical Journal
Editors considers the 500-word
abstract and results tables
referenced in the peer-review
process document not to be
prior publication and that PCORI
will work closely with Awardees
to see that PCORI’s process
does not interfere with their
fulfilling any other legal or
regulatory reporting
requirements or impede their
ability to publish papers
resulting from their funded
projects.
Harmonize the processes
of producing the final
report, reporting on
ClinicalTrials.gov and
publication in the peerreviewed literature.
Minimize the burden to
Awardees.
individuals and organizations, with
whom we and our funded
investigators have been engaged
since early in PCORI’s existence.
The methodological review is
intended to provide final
confirmation of the validity of the
ongoing PCORI staff review of each
project for adherence to PCORI’s
Methodology Standards, a process
undertaken throughout the life of a
particular study. If the review finds
that the study has methodological
flaws not previously identified and
addressed, the Awardee will be
asked to revise the conclusions or
other aspects of the research report
to reflect that fact.
Following formal acceptance of the
final report, PCORI will create a
standardized summary of the
study’s results for patients and
general public with readability at the
6th grade level, which will be
reviewed and approved by the
Awardee.
PCORI’s plan to include within its
proposed process an approach
developed by the National Institutes
of Health to implement the Food
and Drug Administration
Amendments Act (FDAAA) through
the National Library of Medicine’s
clinical trials registry
(ClinicalTrials.gov), one that journal
editors generally accept as not
constituting prior publication.
Registration is a documented
milestone in the contract between
the Awardee and PCORI.
The date when results tables are
submitted is a recorded milestone in
the Awardees contract with PCORI.
PCORI’s Process for Peer Review of Primary Research and Public Release of
Findings
15
Add the requirement of a
formal evaluation of the
process in one year.
PCORI agrees with this
suggestion and has added such
language to the process
document.
The Awardee Institution must
submit a draft final report to PCORI
on a date established and recorded
as a milestone in the contract with
PCORI. The date may not exceed 13
months from the primary
completion date.
This process will undergo a formal
review one year after adoption, with
additional review and revision as
appropriate in future years to assess
how well it is functioning and
consistent with PCORI’s authorizing
law.
PCORI’s Process for Peer Review of Primary Research and Public Release of
Findings
16