1. science
2. medicine
3. psychology and psychiatry

Accreditation Council for Graduate Medical Education
How to Engage and Benefit from the
Review Committee for Internal Medicine (RC-IM)
James A. Arrighi, MD, Chair, RC-IM
Jerry Vasilias, PhD, Executive Director, RC-IM
© 2013 Accreditation Council for Graduate Medical Education (ACGME)
Other possible titles…
Communicating with the RC-IM, or
Understanding the NAS: Working with the RCIM, or
What’s Expected of Me in the NAS:
Me and the RC-IM
© 2013 Accreditation Council for Graduate Medical Education (ACGME)
What will NAS mean for me
as a Program Director?
© 2013 Accreditation Council for Graduate Medical Education (ACGME)
It should lead to “less burden”
• No PIFs!!!
• Streamlined process
• Scheduled self-study visits from ACGME every 10 years
• Focused site visits when “issues” are identified
• “Internal Reviews” no longer required
• Streamlined ADS Annual Update
• Many data elements used in NAS are already in place in ADS
• Some ADS annual simplified
•
•
•
•
Removed 33 questions
14 questions simplified
Faculty CV removed (except for PD)
11 MCQ or Y/N questions added
© 2013 Accreditation Council for Graduate Medical Education (ACGME)
It should allow you to “innovate”
NAS = Innovation without
permission.
© 2013 Accreditation Council for Graduate Medical Education (ACGME)
How will programs “innovate?”
• Program Requirements (PRs) classified:
• Outcome
• Core
• Detail
• Programs in good standing:
• May freely innovate in detail standards
© 2013 Accreditation Council for Graduate Medical Education (ACGME)
Categorization of Program
Requirements (Example of IM)
Core
Detail
Outcome
Common Program Requirements
Total # %
89
45%
66
34%
42
21%
Majority of Common PRs -- “core”
© 2013 Accreditation Council for Graduate Medical Education (ACGME)
Core
Detail
Outcome
IM Program Requirements
Total #
56
83
24
%
34%
51%
15%
Majority of IM PRs -- “detail”
Examples of Program Requirements
“Core”
•
•
•
•
PD support from institution
Inpatient caps
Faculty qualifications (e.g. certification)
Overall resources needed “for resident education”
• Specific resources, e.g. angiography, are detail
• Continuity clinic experience inclusive of
“chronic disease management, preventive health, patient
counseling, and common acute ambulatory problems.”
• Major duty hours rules
© 2013 Accreditation Council for Graduate Medical Education (ACGME)
Examples of Program Requirements
“Detail”
•
•
•
•
•
•
Simulation
Minimum 1/3 ambulatory, 1/3 inpatient
Critical care min (3 mos) and max (6 mos)
130-session clinic rule
Specific conference structure
Specific aspects of evaluation structure
• Semiannual evals remain core
• 5 year rule for PD’s
© 2013 Accreditation Council for Graduate Medical Education (ACGME)
Evaluation Program Requirements in NAS
An Example
The program director must provide a summative evaluation for each
resident upon completion of the program. (Core)
This evaluation must:
V.A.2.b).(1) become part of the resident’s permanent record
maintained by the institution, and must be accessible for review
by the resident in accordance with institutional policy; (Detail)
V.A.2.b).(2) document the resident’s performance during the final
period of education; and, (Detail)
V.A.2.b).(3) verify that the resident has demonstrated sufficient
competence to enter practice without direct supervision. (Detail)
© 2013 Accreditation Council for Graduate Medical Education (ACGME)
Examples of Program Requirements
“Outcome”
• Sections listed under the 6 competencies
• 80%/80% board take/pass rule
• PR’s related to principles of professionalism
• Safety, recognition of fatigue, commitment to LLL,
honesty of reporting, etc.
• Effective hand overs
© 2013 Accreditation Council for Graduate Medical Education (ACGME)
Annual Data Review Elements
The following are the “primary” annual data
elements:
1)
2)
3)
4)
5)
6)
7)
8)
Program Attrition
Program Changes
Scholarly Activity
Board Pass Rate
Clinical Experience Data
Fellow Survey
Faculty Survey
Milestones
© 2013 Accreditation Council for Graduate Medical Education (ACGME)
Where did the NAS annual data
elements come from?
© 2013 Accreditation Council for Graduate Medical Education (ACGME)
Where did the NAS annual data
elements come from?
History of prior
accreditation decisions
Data analysis & modeling
Analysis to determine what combination of data
elements may predict a “problem” program.
Adequate sensitivity
Minimize false negative and positives
Importance of trends
Understand that this is a…
New data elements will likely be introduced in future.
© 2013 Accreditation Council for Graduate Medical Education (ACGME)
What will happen at my program?
What will my year look like?
© 2013 Accreditation Council for Graduate Medical Education (ACGME)
NAS: Annual Data Submission
Year 1
Jul Aug Sep Oct Nov Dec Jan Feb Mar Apr May Jun
Jul Aug Sep Oct Nov Dec Jan Feb Mar Apr May Jun
© 2013 Accreditation Council for Graduate Medical Education (ACGME)
NAS: Annual Data Submission
Year 1
Jul Aug Sep Oct Nov Dec Jan Feb Mar Apr May Jun
ADS Update
Yr 1
Jul Aug Sep Oct Nov Dec Jan Feb Mar Apr May Jun
© 2013 Accreditation Council for Graduate Medical Education (ACGME)
Annual Data Review Elements
1)
2)
3)
4)
5)
6)
7)
8)
Program Attrition
Program Changes
Scholarly Activity
Board Pass Rate
Clinical Experience
Resident Survey
Faculty Survey
Milestones
© 2013 Accreditation Council for Graduate Medical Education (ACGME)
• Collected as part of annual
ADS update
• ADS streamlined this year:
33 fewer questions & more
multiple choice or Y/N
• Initially time intensive/
challenging, but gets easier
Annual Data Review Element #1:
Program Attrition
• General Definition: Composite variable that measures
degree of personnel and trainee change w/in program.
• How measured: Has the program experienced any of the
following:
• Changes in PD?
• Decrease in core faculty?
• Residents withdraw/transfer/dismissed?
• Change in Chair?
• DIO Change?
© 2013 Accreditation Council for Graduate Medical Education (ACGME)
Annual Data Review Element # 2:
Program Changes
• General Definition: Composite variable that measures
the degree of structural changes to the program.
• How measured: Has the program experienced any of the
following:
• Participating sites added or removed?
• Resident complement changes?
• Block diagram changes?
• Major structural change?
© 2013 Accreditation Council for Graduate Medical Education (ACGME)
Annual Data Review Element #3:
Scholarly Activity: Faculty – Core IM
Number of
abstracts,
posters, and
Pub Med Ids (assigned presentations
by PubMed) for articles given at
published between
international,
7/1/2011 and
national, or
6/30/2012.
regional
List up to 4.
meetings
between
7/1/2011 and
6/30/2012
Faculty PMID PMID PMID PMID
Member
1
2
3
4
John Smith
12433
32411
Number of other
presentations given
(grand rounds, invited Number of
professorships),
chapters or
materials developed textbooks
(such as computer- published
based modules), or between
work presented in
7/1/2011
non-peer review
and
publications between 6/30/2012
7/1/2011 and
6/30/2012
Number of
grants for
which faculty
member had
a leadership
role (PI, CoPI, or site
director)
between
7/1/2011 and
6/30/2012
Between 7/1/2011 and
6/30/2012, held responsibility
for seminars, conference
Had an active
series, or course coordination
leadership role
(such as serving (such as arrangement of
on committees or presentations and speakers,
governing boards) organization of materials,
in national medical assessment of participants'
performance) for any didactic
organizations or
served as reviewer training within the sponsoring
or editorial board institution or program. This
includes training modules for
member for a
medical students, residents,
peer-reviewed
fellows and other health
journal between
professionals. This does not
7/1/2011 and
include single presentations
6/30/2012
such as individual lectures or
conferences.
Conference
Presentations
Other Presentations
Chapters /
Textbooks
Grant
Leadership
Leadership or PeerReview Role
Teaching Formal Courses
3
1
1
3
Y
N
RC-IM Expectation/Threshold: Within the last academic year, at least 50% of
the program’s “core” faculty need to have done at least one type of scholarly
activity from the list of possible activities in the table above.
© 2013 Accreditation Council for Graduate Medical Education (ACGME)
Annual Data Review Element #3:
Scholarly Activity: Residents
Number of abstracts,
posters, and
Pub Med Ids (assigned by
presentations given at
PubMed) for articles
international, national,
published between 7/1/2011
or regional meetings
and 6/30/2012. List up to 3.
between 7/1/2011 and
6/30/2012
Resident
PMID 1
June Smith
12433
PMID 2 PMID 3
Number of chapters
or textbooks
published between
7/1/2011 and
6/30/2012
Participated in funded or
non-funded basic science
or clinical outcomes
research project between
7/1/2011 and 6/30/2012
Lecture, or presentation (such
as grand rounds or case
presentations) of at least 30
minute duration within the
sponsoring institution or
program between 7/1/2011 and
6/30/2012
Conference
Presentations
Chapters /
Textbooks
Participated in research
Teaching / Presentations
1
0
N
Y
RC-IM Expectation/Threshold: At least 50% of the program’s recent graduates
need to have done at least one type of scholarly activity from the list of possible
activities in the table above.
• Broad definition
• What recent graduates did during entirety of training.
© 2013 Accreditation Council for Graduate Medical Education (ACGME)
Annual Data Review Element #4:
Board Pass Rates
80% take, 80% pass rule over a 3 year timeframe
© 2013 Accreditation Council for Graduate Medical Education (ACGME)
NAS: Annual Data Submission
Year 1
Jul Aug Sep Oct Nov Dec Jan Feb Mar Apr May Jun
Resident Survey
ADS Update
Yr 1
Yr 1
Jul Aug Sep Oct Nov Dec Jan Feb Mar Apr May Jun
© 2013 Accreditation Council for Graduate Medical Education (ACGME)
Annual Data Review Elements
1)
2)
3)
4)
5)
6)
7)
8)
Program Attrition
Program Changes
Scholarly Activity
Board Pass Rate
Resident Survey
Clinical Experience
Faculty Survey
Milestones
© 2013 Accreditation Council for Graduate Medical Education (ACGME)
Annual Data Review Element #5:
ACGME Resident Survey
© 2013 Accreditation Council for Graduate Medical Education (ACGME)
Annual Data Review Element #5:
IM Resident Survey – simpler, shorter
• Significantly streamlined the IM survey: 64 of 92 items
survey were removed b/c they were associated with “detail”
PRs or were redundant with items on the ACGME survey
• Items retained:
•
•
•
•
•
Adequacy of on-call facilities
Availability of support personnel
Adequacy of conference rooms & other facilities used for teaching
Patient cap questions
Questions related to clinical experience (next slide)
• No changes to survey for next year
• 28 items long for PGY3s, and
• 14 items long for PGY1 & 2s
© 2013 Accreditation Council for Graduate Medical Education (ACGME)
Annual Data Review Element #6:
Clinical Experience Data (Core)
• Composite variable on residents’ perceptions of clinical
preparedness based on the specialty specific section of the
resident survey.
• How measured: 3rd year residents’ responses to RS
•
•
•
•
Adequacy of clinical & didactic experience in IM, subs, EM, & Neuro
Variety of clinical problems/stages of disease?
Experience w patients of both genders and a broad age range?
Does continuity experience allow development of a continuous
therapeutic relationship with panel of patients
• Are you able to manage patients in the prevention, counseling,
detection, diagnosis and treatment of diseases appropriate of a
general internist?
© 2013 Accreditation Council for Graduate Medical Education (ACGME)
Annual Data Review Element #6:
Clinical Experience Data (Subs)
•
•
•
•
Proxy for case/procedure logs
Broad + Brief – 9 total questions
Will appear after ACGME survey
Assesses fellows’ perceptions of clinical preparedness
•
•
•
•
experience w variety of clinical problems/stages of disease (PR II.D.5.a))
experience w patients of both genders/ages (PR II.D.5.b))
Adequacy of continuity experience (PR IV.A.3.e))
Do you believe you will be able to competently perform all of the medical/
diagnostic procedures of a subspecialists in this area (PR IV.A.2.a).(2)
• Do you believe you will be able to provide patient care that is
compassionate, appropriate and effective for the treatment of health
problems and promotion of health (PR IV.A.2.a).(1)
• To be implemented in 2014
© 2013 Accreditation Council for Graduate Medical Education (ACGME)
NAS: Annual Data Submission
Year 1
Jul Aug Sep Oct Nov Dec Jan Feb Mar Apr May Jun
Faculty Survey
Yr 1
Fellow Survey
Yr 1
ADS Update
Yr 1
Jul Aug Sep Oct Nov Dec Jan Feb Mar Apr May Jun
© 2013 Accreditation Council for Graduate Medical Education (ACGME)
Annual Data Review Elements
1)
2)
3)
4)
5)
6)
7)
8)
Program Attrition
Program Changes
Scholarly Activity
Board Pass Rate
Clinical Experience
Fellow Survey
Faculty Survey
Milestones
© 2013 Accreditation Council for Graduate Medical Education (ACGME)
Annual Data Review Element #7:
Faculty Survey
•
•
Will be administered to whoever is
listed in the faculty roster – physician
faculty only
Content areas align with Resident
Survey
•
•
•
•
•
Faculty supervision & teaching
Educational Content
Resources
Patient Safety
Teamwork
Annual Data Review Elements
1)
2)
3)
4)
5)
6)
7)
8)
Program Attrition
Program Changes
Scholarly Activity
Board Pass Rate
Clinical Experience
Fellow Survey
Faculty Survey
Milestones
© 2013 Accreditation Council for Graduate Medical Education (ACGME)
Annual Data Review Element #8:
ACGME Reporting Milestones
“A key element of the NAS is the
measurement and reporting of
outcomes through educational
milestones…”
1 Nasca, T.J., Philibert, I., Brigham, T.P., Flynn, T.C.
The Next GME Accreditation System: Rationale and Benefits.
New England Journal of Medicine. Published Electronically, February 22,
2012. In Print, March 15, 2012.
DOI:10.1056/nejmsr1200117 www.nejm.org .
NEJM. 2012.366;11:1051-1056.
© 2013 Accreditation Council for Graduate Medical Education (ACGME)
“Programs in the NAS will
submit composite milestone
data on their residents every 6
months, synchronized with
residents’ semiannual
evaluations.”
Annual Data Element # 8:
Reporting Milestones (IM Residency)
ABIM
Competencies (6)
AAIM
ACGME
Sub-Competencies (22)
Reporting Milestones (5
per sub-competency)
NAS Milestones
© 2013 Accreditation Council for Graduate Medical Education (ACGME)
Reporting Milestones
• De-identified, aggregate (program) data will
gradually be used as one element of accreditation
decisions
• Individual reports by trainee will be provided to PD
• Perfection is not the expectation
• In response to concerns, the first reporting period
for IM will be delayed from 12/2013 to 6/2014
• Semiannual reporting remains a foundation of
NAS
© 2013 Accreditation Council for Graduate Medical Education (ACGME)
Reporting Milestones: Test Run
• For those interested…
• Test-Run = November 1-December 31, 2013
• Data entered will not be used for accreditation
matters or trending reports and will be purged
• In academic year 2014, IM programs will be
reporting twice annually
• First window: November 1 – December 31
• Second Window: May 1 – June 15
© 2013 Accreditation Council for Graduate Medical Education (ACGME)
What are the “Reporting Milestones”?
• The reporting milestones are not assessment tools
• They are descriptors of behavior along a
continuum of performance.
• Existing tools will need to be used and new tools will
need to be developed to assess resident Milestone
achievement
© 2013 Accreditation Council for Graduate Medical Education (ACGME)
Example of Reporting Milestone (Core IM)
Sub-Competency
Developmental Progression or
Set of Milestones
Milestone
© 2013 Accreditation Council for Graduate Medical Education (ACGME)
Assessment  Evaluation 
Reporting
Direct
Obs
C C C
Semiannual Evaluation
Rotation
evals
Other
formative
assessments
Assessment Machinery
© 2013 Accreditation Council for Graduate Medical Education (ACGME)
ACGME
and
ABIM Reporting
Milestones
Sidebar on Terms
• “Curricular” milestones
• Developed by subspecialty societies
• Granular, specific, practical
• May be used to develop curricula, evaluations
• “Reporting” milestones
• Reported to ACGME and (eventually) to ABIM
• Developed by community, but approved by
ACGME & ABIM
• Broad, generalizable
• Q 6 months (linked to semiannual eval)
© 2013 Accreditation Council for Graduate Medical Education (ACGME)
One more sidebar…EPAs
Entrustable Professional Activities (EPAs)
•
•
are important tasks of the physician for which it is desired that
competency-based decisions be made regarding the level of
supervision needed.
For EPAs it is desired that residents attain the competency needed to
perform the task without supervision by the time they graduate.
• Examples from IM:
• Manage care of pts w chronic disease across multiple settings
• Lead and work within interprofessional care teams
Two page “primer” on EPAs: March issue of JGME, pages 157-158
•
The ACGME does not require EPAs
•
© 2013 Accreditation Council for Graduate Medical Education (ACGME)
What specific elements of the
system are ACGME?
Curricular
Milestones
EPA’s
Specific
curriculum
Evaluation
System
Reporting
Milestones
Partly
ACGME
ACGME
(and ABIM)
Not
ACGME
© 2013 Accreditation Council for Graduate Medical Education (ACGME)
NAS: Annual Data Submission
Year 1
Jul Aug Sep Oct Nov Dec Jan Feb Mar Apr May Jun
Milestones
Yr 1
Yr 1
Faculty Survey
Yr 1
Resident Survey
Yr 1
ADS Update
Yr 1
Jul Aug Sep Oct Nov Dec Jan Feb Mar Apr May Jun
© 2013 Accreditation Council for Graduate Medical Education (ACGME)
What happens after I submit my data?
© 2013 Accreditation Council for Graduate Medical Education (ACGME)
What happens after data are submitted?
• Each program is reviewed & given feedback
annually
• NAS is continuous accreditation model
• Review of annual data w following supplemental info:
• Reports of self-study visits every ten years
• Progress reports (when requested)
• Reports of site visits (as necessary)
– Full or Focused Site Visit
© 2013 Accreditation Council for Graduate Medical Education (ACGME)
What happens after data are reviewed?
• “Cycle Lengths” will not be given – that’s OAS, not NAS
• Citations may be given or removed
• Areas for Improvement may be given
• Areas for Improvement are different from citations
•
•
•
•
Will not be reviewed annually by RC
Are not necessarily linked to a PR
Programs do not need to provide response in ADS
RC will monitor whether addressed using annual data
• Status Options:
Continued Accreditation
Accreditation with Warning
Probationary Accreditation *
Withdrawal of Accreditation *
* Status conferred only after a site visit.
© 2013 Accreditation Council for Graduate Medical Education (ACGME)
© 2013 Accreditation Council for Graduate Medical Education (ACGME)
Conceptual Model of NAS
Accreditation
w/ Warning
Initial
Accreditation
Accredited Programs
with Major Concerns
New
Programs
Program
Requirements
Accredited Programs without
Major Concerns
Probationary
Accreditation
2-4%
Core Process Core Process
Detailed Process Detailed Process
Outcomes Outcomes
Continued
Accreditation
15%
Core Process
Detailed Process
Outcomes
75%
Core Process
Detailed Process
Outcomes
Withhold Accreditation
Withdrawal of Accreditation
6-8%
© 2013 Accreditation Council for Graduate Medical Education (ACGME)
Final thoughts…
© 2013 Accreditation Council for Graduate Medical Education (ACGME)
Q: So, in the NAS, “detail” PRs go away?
• Not really…
• If you want to comply with them, you can, but
• If you want to innovate/experiment – do
something different, w/ “detail” PRs, you can do
so,
• RC is interested in seeing that the “core” or
“outcome” PRs associated w/ the “detail” PRs are
being achieved.
- continuity clinic
© 2013 Accreditation Council for Graduate Medical Education (ACGME)
Q: Now that I have a Self-Study visit does that
mean I will not be reviewed except for every 10
years?
• Although the Self-Study will take place in the
future, the RC will be reviewing the NAS data
annually.
• RC can request a site visits whenever multiple
data element(s) show outliers/extreme responses.
• One flagged data element will not trigger a site
visit
• These visits will be focused or full visits and will
not require a PIF.
© 2013 Accreditation Council for Graduate Medical Education (ACGME)
Q: What’s a “focused site visit?”
• Assesses selected aspects of a program and may
be used:
• to address potential problems identified during
review of annually submitted data;
• to diagnose factors underlying deterioration in a
program’s performance
• to evaluate a complaint
• Minimal notification given
• Minimal document preparation
• Team of site visitors
• Specific program area(s) specified by RC
© 2013 Accreditation Council for Graduate Medical Education (ACGME)
© 2013 Accreditation Council for Graduate Medical Education (ACGME)
Q: When will a “full site visit” occur?
•
•
•
•
New application for a core program
At the end of the initial accreditation period
RRC identifies broad issues / concerns
Other serious conditions or situations identified by
the RRC
© 2013 Accreditation Council for Graduate Medical Education (ACGME)
© 2013 Accreditation Council for Graduate Medical Education (ACGME)
Q: Any further info on “Self Study
Visits?”
•
•
•
•
•
•
•
Different from focused or full
Not fully developed
For IM: to start May 2015
Will require minimal document prep
Will be conducted by team of site visitors
Will review core and subs together
Will review annual program evaluation
• Responses to citations
• Faculty development
• Focus will be on continuous improvement w/i prog
• Verify compliance w/ “core” PRs
© 2013 Accreditation Council for Graduate Medical Education (ACGME)
© 2013 Accreditation Council for Graduate Medical Education (ACGME)
Q: Is CLER part of the NAS?
• At this point, Clinical Learning Environment
Review (CLER) is not part of NAS; data from
CLER visits will not be used for accreditation
• CLER focuses on institutional environment –
not individual programs
• PD role limited to:
• Facilitating peer selection of residents
• Participation in group interview
• Ongoing & active involvement of residents in
quality and safety initiatives w/in institution
© 2013 Accreditation Council for Graduate Medical Education (ACGME)
Q: What role can PD have on CCC?
• PD can be a member in whatever capacity
• PRs do not preclude or limit PD’s participation
• CCC must be composed of at least 3 faculty
• non-physician members may be included
© 2013 Accreditation Council for Graduate Medical Education (ACGME)
Thank you.
“You can’t teach an old dogma
new tricks.”
Dorothy Parker
© 2013 Accreditation Council for Graduate Medical Education (ACGME)