1. business and industrial
2. company
3. annual report

Please note that no one has reviewed this documentation for accuracy or
completeness. If you find mistakes, please report them to Anne Rigg,
[email protected]. Thanks for your help in improving this documentation.
What is New Innovations (NI)?
New Innovations is a web-based software application used by residents/fellows to log duty hours,
by trainees and faculty to evaluate each other and the program, by program administrators to
monitor the program, by GME for email lists and monitoring of duty hours and other aspects of
the program.
Depending on your role and the size of your residency you might never use NI or you might use it
daily. Even if you never use it, your program will use the information in it so it's useful for you to
know what data is available.
There are about a dozen "modules" in NI. There are 4 modules a program or its residents must
use: personnel data, duty hours, evaluations, and block schedule.
Personnel Data
The Personnel data module contains information about residents/fellows, coordinators, program
directors, and probably faculty and, usually, allows them to log into NI. Finance uses the
information for Medicare reimbursement. GME uses the names and email addresses to contact
all residents/fellows at the request of hospital administrators, to survey non-accredited fellows, to
contact coordinators.
Accredited Trainee Personnel Data
Each program is required to have personnel records of their trainees in NI. The GME Office asks
each program for this information shortly after the NRMP match in March of each year. If your
program goes through the NRMP match, we will get your program’s data from the consultant who
works jointly for the MGH Multicultural Affairs Office and the BWH Office for Multicultural Faculty
Careers. If not, we will send you an Excel spreadsheet that contains 50+ fields, about 15 of which
are for required data and the remaining for optional data, and the GME Office will create the
resident and fellow accounts.
The required information includes things like the resident or fellows' name, birth date, social
security number, etc. - information that is required by the Finance department to get Medicare
reimbursement for part of the cost of training residents and fellows. The other information such as
pager number, advisor, test scores, home address, etc. is there for your benefit. The GME Office
asks that you return the data within two or three weeks. Once the data is input into NI, the GME
Office needs to update it with trainees Partners email addresses and Partners network/login
accounts.
After we do so, the program is expected to notify the residents and fellows of their username and
passwords and instruct them to change their passwords. The GME Office has created a NI
custom report that can be used to find out the usernames.
When the GME Office creates the incoming residents and fellows’ accounts, the username
always defaults to the first initial of the first name combined with the last name UNLESS that
name is already in use. If it is, NI will try creating an account using the first two letters of the first
name combined with the last name or the first three letters of the first name, etc. Residents and
10/7/11
Page 1 of 24
fellows can change their username though the GME Office only tells them that they can change it
to be the same as their Partners username.
You can not find out a person’s current password in NI but when NI creates a password or resets
it, the password always defaults to the first initial of the first name combined with the last name.
For example, if someone’s name is John O’Connor, his password will be jo’connor. If there is a
space or hyphen in the last name, it is included in the password as well as in the username.
Once the GME Office creates the accounts, you may want to load the residents or fellows’ CVs
and ECFMG certificates in NI. Finance is required to have copies of them and will get them from
NI if they are stored there.
Anytime there is a change to your residents or fellows, for example, someone leaves the
program, you should notify the GME Office so that NI can be updated. If a resident or fellow goes
on leave and his/her program end date or advancement date changes, you should notify the GME
Office.
NI also stores information about rotators to Partners hospitals (partners is required by Medicare
to do so) and about fellows in non-accredited programs, and about faculty.
Two fields of data we need include the status type and post-graduate year (PGY). The status
type for residents in accredited years of training starts with YIP which stands for “year in
program”, YIP1 for residents in their first year in the program, YIP2 for those in their second year,
up through YIP7. For non-accredited or research residents or fellows we also have “nonaccredited” or “research”. Anyone with a status type of “YIP” something is required to log duty
hours. The status type is used by NI to determine which set of duty hour rules apply to the
resident or fellow (those for interns, those for intermediate residents, and those for
residents/fellows in the “final year(s)” of training. It is also used by New Innovations in the
Evaluations module so that residents/fellows at the same year of training can be compared to
each other for evaluation purposes. The post-graduate year, or PGY, is for the year of training the
resident or fellow is in. For residents, the YIP and PGY are generally the same. For programs in
which the resident must have prior training before starting the program, e.g., Anesthesiology or
Urology and all fellowships, they are different. For example, if someone does a 5-year surgical
residency and then starts a Thoracic Surgery fellowship, his/her status type in Thoracic Surgery
starts with YIP1 but he/she’s probably a PGY6. A resident is always a PGY1 in their first year of
training so to calculate the PGY, count the number of years of training the resident has completed
and add 1. Note that the PGY only includes clinical training. If a resident did 1 year of research
between his residency and fellowship, that year does not count in calculating the PGY because
research is not clinical training. Only when a research year is part of the definition of an
accredited training program does it count toward the PGY.
Advancement
At the end of each academic year, a resident/fellow “advances” to the next year of training. NI
automatically “advances” a resident by increasing their YIP status and their PGY year by 1 on the
“advancement date” which is generally the anniversary of their program start date.
Faculty and Non-accredited Trainee Personnel Data
When the GME Office creates an account for a faculty member or anyone else who already has a
Partners username, the NI username is configured to be the same as the Partners username, all
lowercase. While the Partners username is not case-sensitive, the NI username is.
10/7/11
Page 2 of 24
Your program is not required to store information about faculty but if your program uses NI for
faculty evaluations of trainees, the faculty member needs to have an account in NI.
For faculty and coordinators and fellows in non-accredited programs, all we store is name,
department/division, email address, employer, credentials, and Partners login (if they have one).
We can also store information about nurses and other healthcare professionals if the program
wants to include them as evaluators of residents or fellows.
Logging In to NI
Once an account is created, the user can log on one of three ways:
(1) from the Start menu of a Partners Windows computer, select "Partners Applications" and then
select "New Innovations". This will log the user in automatically using your Partners username. If
the user does not have a Partners user name, obviously, this doesn’t apply. This works through
what is called the “Single Sign On” in NI. When a person clicks on “New Innovations”, a small
software application sends the person’s Partners username to NI. The software application knows
the person’s Partners username because the person logged onto the computer using it. NI looks
through the list of Single Sign On accounts listed in NI and identifies the person who has that
Single Sign On and logs that person into NI automatically.
(2) most emails from New Innovations contain a "click here" link which will log you in
automatically. The first time a person uses such a link and after changing his or her password,
he/she will need to authenticate him/herself to New Innovations.
(3) use Internet Explorer to navigate to http://www.newinnov.com/login/login.aspx?hospital=partners and supply his/her institution (Partners), his/her
New Innovations user name and his/her NI password. Both your username and your password
are case-sensitive; the institution is not.
10/7/11
Page 3 of 24
Departments/Divisions and Programs
Partners has about 100+accredited training programs and about 120+ non-accredited programs
which are all listed in NI. As a coordinator, you will have privileges in the department/division to
which your program belongs. To obtain privileges in another department/division, please ask the
program director or coordinator for that program to email me granting you privileges in their
“program” or “division”. Programs belong to departments/divisions. Faculty and staff have
accounts in departments/divisions. Trainees accounts are in department/divisions but they are in
programs. In addition to departments/divisions containing programs, we also have in NI, several
other departments/divisions to contain faculty accounts. The naming conventions for
departments/divisions and programs is slightly different. Department/divisions always start with
the hospital followed by the department and then the division. Programs start with the sponsoring
institution (PHS for combined programs) followed by the program name. For example, MGH
Orthopaedic Surgery is a combined, or integrated program whose residents rotate to MGH, BWH,
and BIDMC (Beth Israel Deaconess Medical Center). The accounts for all residents and for
faculty at MGH are in the division “MGH – ORTHO – Orthopaedic Surgery (PHS). The accounts
for faculty at BWH are in “BWH – ORTHO – Orthopaedic Surgery” and the accounts for faculty at
10/7/11
Page 4 of 24
BIDMC are in “X - Other Faculty - BIDMC”. The program itself is called “PHS – Orthopaedic
Surgery”. Finding the correct department/division or program is usually easy because NI presents
you with either the department/division list or the program list EXCEPT in the case of custom
reports where you can choose either. If you want to find all the residents in a particular program,
be sure to use the “program” field for filtering. If you’re looking for faculty, be sure to use the
“department/division” field for filtering.
Security and Privileges
Coordinators have the highest level access allowed to those outside of Finance and GME. The
main difference between the level of access you have and those in the GME Office have is that
only GME staff can access data for all programs, view the duty hour evaluations, create rotations,
and create new accounts. Coordinators have higher privilege levels, by default, than most
program directors. Generally, program directors have “faculty” level privileges but when a
program director requests level 5 privileges, he/she is always given them.
As a coordinator, you have access to highly confidential information about a trainee, including
home address, birth date, home phone number, the last 4 digits of the SSN, evaluations by
faculty. Please be sure to use a very secure password for your NI account and do not write the
password down. Your password should be at least 8 characters long and contain at least one
numeral and one alphabetic character. It should not be a word in any dictionary, English or
foreign language.
Work Role
The work role is used by NI for a variety of minor purposes, for example, to determine whether or
not all residents/fellows are being evaluated and to run custom reports (administrator work role
required). Residents and fellows both are referred to as “residents” by NI and have a “resident”
work role. Faculty members have a “faculty” work role. Coordinators have an “administrator” work
role. Program directors have a “faculty” and often an “administrator” work role.
Personnel Data screen shots
The “Demographics” tab for a faculty member looks something like the image below:
10/7/11
Page 5 of 24
The person’s name (last name, first name) is listed on the top left side of the page. The person’s
status is listed below the photo placeholder and the person’s NI username is listed below that.
Each of the tabs and the links in blue provide additional information. CVs and ECFMG certificates
are added on the Files and Notes tab.
You can search for anyone in NI by using the search box in the upper right. Key in the person’s
last name followed by a comma, a space and the first name. Partial names are ok. For example,
if you don’t know if a person’s name is spelled “goldstein” or “goldstien”, key in “goldst” and NI will
find all names that start with goldst. If you’re searching for someone who isn’t in your program, NI
will simply tell you what program the person is in but won’t let you access his/her record.
Block schedule and academic year
Rotation
Residents and fellows spend different parts of the year learning different clinical skills. For
example, a Pediatrics resident might spend a month in the Pediatric Intensive Care Unit or two
weeks on an Emergency Medicine rotation. These "assignments" are referred to as "rotations"
and have a specific curriculum associated with them. The group of rotations that a resident/fellow
is on in the course of an academic year is referred to as the block schedule. Residencies typically
have rotations that are from 1 - 4 weeks in duration. Fellowships most likely will have rotations
that are four to 13 weeks in duration.
10/7/11
Page 6 of 24
Finance uses NI schedules Medicare reimbursement. The reimbursement goes to the hospital
where the trainee’s work took place. So if your residents are in a program that is sponsored by
MGH but sends residents to BWH and BIDMC, Finance needs to know at which hospital each
resident's rotations took place.
Only level 6 administrators (the GME Office and Finance) can create rotations. When a rotation is
created, please indicate the location where the training takes place.
Academic Year
The academic year is a description of the blocks or intervals within the year that dictate when
residents/fellows start/end a rotation. The academic year is typically July 1 to June 30 but may be
slightly different due to intern orientation. For example, some programs start their academic year
as early as June 20. Some programs will have multiple academic years - e.g., interns might start
on June 20 and be on a two-week rotation schedule but the other residents may start on July 1
and be on a 4-week schedule.
The program coordinator sets up the academic year(s) before or at the beginning of each
academic year (July 1).
Block Schedule
The block schedule is not a detailed daily schedule (that's the Assignment schedule).
The block schedules for residents and fellows must be in NI. Most ACGME duty hour rules are
rotation-based. NI can not determine if a resident or fellow has complied with the duty hour rules
without knowing the date on which his/her rotation started and ended. A resident/fellow can be on
more than one rotation at a time but one must be designated as a “primary” rotation. In checking
the duty hour rules, NI check them against the “primary” rotation.
The block schedules for faculty should be in NI if your program is going to use NI for evaluations
of trainees and faculty. It’s necessary to have the block schedule for faculty in NI because NI
uses the schedule to determine which attendings supervised which residents/fellows.
The block schedule needs to be complete - that is, it should cover every day of the academic
year, even if the trainee is on vacation or leave. While residents and fellows are on different
rotations during the academic year, faculty are generally on the same rotation - because of their
particular expertise - for the entire year. It's not unusual for faculty to be put on the same rotation
for multiple years, e.g., 5 years.
Entering the Block Schedule
The block schedule can be entered manually through NI, imported through AmIOn, or you can
use NI's MedScheduler tool to build your schedule on your computer and upload to NI. There's a
tool in NI that tells you what your schedule gaps are. No one should have schedule gaps even if
they're on vacation. The block schedule is the foundation for NI. If the block schedule is wrong,
you'll have problems with duty hour violations or lack of data and you'll have problems matching
evaluators and subjects for evaluations.
Viewing the Block Schedule
10/7/11
Page 7 of 24
You can view the block schedule for your entire program and anyone on your rotations by clicking
on “Main”, “Scheduling Block”. Go about halfway down the page and under “View Schedules”
click on the second link.
To see a detailed schedule for one person, click on “Main” and “Scheduling Block” and then
halfway down the page under “Create/Edit Schedules” click on “Populate List”. The list will include
everyone in your program plus everyone who is on one of your rotations. It will not include faculty
from other department/divisions but, as a level 5 administrator, you can add them to the schedule.
You cannot do so for residents/fellows. They have to be scheduled by someone with level 5
privileges in their program. Once you’ve selected the person from the drop-down list, click on
“Create/Edit Block Schedule Rotations for:” to the left of the person’s name.
Duty hours
The program director is required by the ACGME to monitor duty hours.
Each resident/fellow has two responsibilities:
1. log hours 4 times/year (August and every third month after August: November, February,
2. fill out evaluation (questionnaire) about duty hours they worked the previous month. We
call this questionnaire an evaluation to help remind residents/fellows that they find it in
the "Evaluations" module in NI, not in the "Duty Hours" module. This evaluation is sent
out once each month, on the first of each month and is due the 21st of each month. It is
10/7/11
Page 8 of 24
sent by the GME Office. It usually contains 16 questions about duty hours; the first
several are required and are about whether or not they broke any duty hour rule. If they
didn't break any rule, they can skip the remaining questions. The next 7 or 8 questions
are about the reasons why they broke one or more rule. These questions are optional.
Once a year - usually in October, we send a questionnaire or evaluation that contains all
the ACGME survey questions plus questions about duty hours. This questionnaire
contains 40-45 questions but should take only a few minutes to complete.
Programs are required to have an 80% response rate to the questionnaire and have a 90%
compliance rate on the duty hour rules. Programs that are under these response and compliance
rates for two consecutive months are contacted by the GME office. At the end of the month the
results for each program are sent to the program director and coordinator for each program within
a department and to the chair of that department. For example, for MGH Anesthesiology, the
results for each residency and fellowship in Anesthesiology is sent to all program directors and
coordinators in Anesthesiology as well as to the chair of Anesthesiology. The comments that are
included as the last question on the questionnaire are not distributed. They go only to Dr. Co.
In addition, the ACGME sends an annual survey to residents and fellows of all accredited
programs that have at least 3 or 4 trainees and asks them questions about the quality of the
program that they participate in. This survey is sent out between January and June of each year
and programs are required to have a 90% response rate.
Logging hours
This is done through the Duty Hours module. Residents choose one of several duty types
(shift/work hours, call, conference/didactics, pager call-not called in, etc.) for each hour they work.
It's important to log using the correct duty type because the rules are different for different duty
types. Vacation/leave is logged slightly differently (the resident simply indicates the first and last
day of the vacation. Days off are not logged. The simple fact that no hours were logged indicate
to NI that the resident/fellow had a day off. Residents can log their hours ahead of time but,
except for vacation/leave, if they log hours ahead of time, have to confirm the hours after they
have actually worked them.
10/7/11
Page 9 of 24
Logging reminders
Residents/fellows are sent an email reminder every 3 days if they are at least 3 days behind on
their logging. The reminder is part of the "daily email reminder" from NI. This is a configuration
setting for each program that can be modified. Generally, the program coordinator receives an
email listing the residents/fellows you received the reminder.
Duty Hour Violations
If their logged hours cause a duty hour violation, the hours that caused the violation are outlined
in red on the logging screen and detailed information indicating the rule that was violated is
provided. They can edit their hours if they made a logging error. It's not a problem to log a
violation; it is a problem if residents/fellows don't log their hours completely, accurately, and
honestly. Your residents and fellows are likely to violate the rules and you can reasonably want to
know why. Please be very careful when asking a resident or fellow why he/she violated a rule to
never give him/her the impression that he/she should change the logged hours to get rid of the
violation. Instead, use the information to figure out how to adjust the schedule to minimize
violations.
10/7/11
Page 10 of 24
Justification
Some violations are "justifiable". There are two rules (24+4 and short break rule) that
residents/fellows in the final year(s) of their education can violate with justification. Valid
justifications are spelled out by the ACGME in the Common Program Requirements
(http://www.acgme-2010standards.org/pdf/Common_Program_Requirements_07012011.pdf) and
the Duty Hour FAQs (http://www.acgme.org/acWebsite/dutyHours/dh-faqs2011.pdf). Which years
are considered "final year(s)" vary by program and are defined by the RRC in the SpecialtySpecific Duty Hour Definitions (http://www.acgme-2010standards.org/pdf/Specialtyspecific_DH_Definitions.pdf). If a violation is justifiable, the resident/fellow will be prompted
through NI to comment on the reason he/she broke the rule. Those comments are then sent to
the program director who can then approve the justification, deny it, or ask for more information. If
a justification is approved, the exception no longer appears as a violation.
10/7/11
Page 11 of 24
Vacation Logging
There is one violation that is often the result of incorrect logging. If a resident takes a week
vacation and would normally have worked only 6 days during that period, he/she should log only
6 days vacation and 1 day off. However, if the resident would normally have worked all 7 days if
not for vacation, then he/she should log all 7 days as vacation days.
Duty Hour Dashboard
The home page of the duty hour module is a dashboard that contains live data. It provides a
useful overview of the status of your program's compliance with the duty hour rules. It also allows
you to view the logged hours of anyone in your program.
Duty Hour Compliance Report
NI developed the Compliance Report from a report originally created by Partners. It is the report
regularly run for the Partners education committees. Many coordinators run it regularly and save
the data to have available for internal reviews and site visits. When running the Compliance
Report, select all YIP1 through YIP7 status types that correspond to your program, and, under
Additional Options, select "Breaks - The number of valid breaks compared to the number of
breaks" for calculating Short Break compliance and choose "Work Periods - The number of valid
shifts compared to the number of shifts" for calculating Max Consecutive Hours compliance. All
other default settings should be fine.
Run the report: Go to "Main", "Duty Hours", "Reports" and clicking on "Compliance". On the
configuration page, select the dates of 8/1/11 to 8/31/11. In the list after "Select Departments",
make sure that your department is selected. Under "Rotations & Statuses" select "Filter by
10/7/11
Page 12 of 24
Status" and choose "YIP 1" to "YIP7". If you are the coordinator for a program like Anesthesiology
where the residents have non-traditional start and graduation dates, also include "Alumni". Under
"Additional Options" check the radio button "Breaks - The number of valid breaks compared to the
number of breaks" and "Work Periods - The number of valid shifts compared to the number of
shifts". Click "View Report".
Check the number of trainees for which NI can find data: In the Compliance report under
"Complete Data" the fraction after "People" shows, as the numerator, the number of people who
have logged all their data while the denominator shows the number of people for which NI can
find data. The denominator should be the number of residents you had at any time in your
program in August. If the denominator is lower than your count of residents/fellows, there is a
problem with your block schedule. I will be comparing this count with the count I find in the
Personnel Data module.
Fix problems with your block schedule: In the Compliance Report, to the left of your program
name is an arrow. Clicking on the arrow will show you the list of trainees for whom NI has data. NI
looks only at data for trainees who are in the block schedule and on a primary rotation. If a trainee
in your program is not listed in the report, it's for one of three reasons: (a) he is not on a rotation
for part of the month of August; (b) the rotation(s) he is on for the month of August are not listed
as "primary"; (c ) the block schedule for August lists his program incorrectly. To determine if the
program is listed correctly, you must look at the block schedule; that portion of the schedule that
you can see in the Personnel Data Schedule tab is not sufficient. Fix the problems with the block
schedule and rerun the Duty Hour Compliance report.
Make sure that at least 80% of your residents/fellows have logged their hours (though it's far
better to have a 100% response rate): On the Duty Hour Compliance report under "Complete
Data" look at the fraction and percent after "Weeks". The percent should be equal to or greater
than 80%. If it is not, you may want to ask your residents/fellows to log their duty hours for August
by Thursday night. If you added the block schedule after a resident logged his duty hours, you
must "confirm" the hours for NI to include them. To confirm them, look at the resident's logged
hours (Main, Duty Hours, My Duty Hours, View Hours. Key in the dates for 8/1/11 to 8/31/11. In
the top right corner of the screen, use the "Find" link to select "YIP1" through "YIP7" and "Alumni"
) to get a list of residents/fellows in your program. Select each person whose data is incomplete,
view their hours, and under the "Last Checked" column look for a date. If it says "Not Checked",
go to the far left of the last row and click on "Edit". On the "Edit" screen, simply click "Save" to
confirm the hours. You may need to confirm the hours by selecting "Edit" and then "Save" on
multiple rows, one for the last day of each rotation the trainee was on.
Please note that even when NI says that a program has a 100% logging rate, this only means
that in 100% of the weeks that residents were on the schedule, they logged at least one day
during the week. So 100% really might mean only 15%. So don't rely on this report as your sole
source of information.
Evaluations
Just as most employees are by your managers, residents/fellows are evaluated by their faculty.
A questionnaire used to evaluate a resident or fellow is often referred to as an evaluation.
Evaluations can be used not just to evaluate faculty, trainees, and the program but can be used
to evaluate rotations and conferences. With the duty hours evaluation, we call it an "evaluation"
rather than a "survey" or a "questionnaire" to help residents and fellows figure out where in NI to
find the evaluation, that is, in the Evaluations module.
10/7/11
Page 13 of 24
When faculty or trainees have an evaluation to complete, they are almost always notified via
email by NI (whether or not they are notified is a configuration option set by the program
coordinator) and a notice is put in the “Notifications” section of their home page.
After clicking on the link “complete them”, the user will see a page similar to that below. The red
text indicates that the evaluations are past due.
Evaluation Session
The configuration options that determine when a questionnaire or evaluation is sent out, who is
evaluating who or what, how frequently the evaluation is sent out, what kind of anonymity is used,
etc. is called an "evaluation session". If you are responsible for NI for your program, I suggest
you set up a meeting with me the first time you set up an evaluation session. It's the most
complicated area of NI.
Each program should have at least three evaluation sessions - if they are using NI for
evaluations: faculty evaluation of trainee, trainee evaluation of faculty, and trainee evaluation of
the program. The ACGME requires that these evaluations take place. The ACGME does not
require by what means a program creates and collects the evaluations. They don't even have to
10/7/11
Page 14 of 24
be in writing but, of course, most people want them in writing. The ACGME requires that residents
and fellows are evaluated at the end of each rotation - it's not sufficient to evaluate them once a
year.
Program Evaluation
A Program Evaluation is a special type of evaluation session. It differs from other evaluation
sessions in three ways: (1) the questionnaire used by the session must be saved as a “program
evaluation”; (2) the anonymity setting is always “total”; and (3) instead of matching based on a
rotation that someone is on, people are matched based on the “program” they are in.
Questionnaires
Most programs have questionnaires that they have used in the past. In addition, the GME Office
has distributed to each program, on behalf of the Partners Education Committee (PEC)
Subcommittee on Evaluation and Feedback, several questionnaires that can be used to evaluate
trainees and faculty. The subcommittee is currently working on several additional questionnaires.
Each question in a questionnaire has three parts: the question (e.g., please rate the fellow on his
medical knowledge), the grade scale (exceptional, above average, average, poor, substandard)
and the category. A category is both an organizing method as well as the way to associate a
question with one or more core competencies. The ACGME requires that all residents and fellows
be trained in each of 6 core competencies: Patient Care, Medical Knowledge, Practice-Based
Learning and Improvement, Interpersonal and Communication Skills, Professionalism, and
System-Based Practice. NI provides reports that allow you to summarize a trainee’s ratings in
each core competency. Questions can be included in the rating averages only when core
competencies are associated with them.
Once a questionnaire has been used in an evaluation, it can no longer be modified. However, it
can be copied and the copy can be modified. While modifying questionnaires is not difficult, you
may benefit from personal instruction the first time you try to modify a questionnaire so please
feel free to contact the GME Office for assistance.
Anonymity
There are 4 types of anonymity configured in an evaluation session.
None – the subject of the evaluation knows the identity of the evaluator. This is the type of
anonymity recommended for faculty evaluations of trainees.
Subject – identifying information about the evaluator such as his/her name and the dates of the
rotation are hidden from the subject. This is the type of anonymity suggested for trainee
evaluation of faculty
Elective – the evaluator decides whether or not the subject can know the identity of the evaluator
Total – the name and identifying information about the evaluator is never stored in NI so it’s
impossible to determine the evaluator’s identity. This is the type of anonymity recommended for
evaluation of the program.
10/7/11
Page 15 of 24
Evaluators don’t know the type of anonymity configured for an evaluation session so if
confidentiality is a concern, you can include information about anonymity in the instructions on the
questionnaire.
Completed Evaluations
Depending on the type of anonymity configured for the evaluation session, residents and faculty
may or may not have access to the evaluations written about them. Some program directors
prefer to keep all evaluations on hold until he/she can review them with the subjects of the
evaluations. To view evaluations, go to “Main”, Evaluations”, “View” and select “Completed
Evaluations”. On the page that opens, select the academic year, the sessions for which you want
to see the evaluations, the name of the evaluator and the subject. Click “List ALL Evaluations”.
Evaluations can then be view on line, printed to PDF, or returned to the evaluator (a rare
occurrence). Evaluations can also be edited but, of course, this is not recommended except when
the “Edit” function is used to save an evaluation as “draft” which is the method used to return an
evaluation to an evaluator at his/her request.
Delinquent Evaluations
Also under the “View” menu, you can get a list of delinquent evaluations and send customized
email messages to the evaluators. Unlike most emails from NI which are bundled into one
message a day, these email messages are sent immediately in addition to the daily email
notification.
Evaluation Tools
10/7/11
Page 16 of 24
Occasionally a faculty member will fill out an evaluation for the wrong person, in other words,
he/she will fill out an evaluation for Trainee A when the evaluation is really for Trainee B. These
evaluations can be returned to the evaluator. Simply “edit” the evaluation but don’t change
anything. Simply save it as “draft”.
Evaluations can be placed on hold automatically and then either released automatically or
manually.
Evaluation Reports
Note that there are serious limitations using the standard evaluation reports in NI. NI Help Desk
staff recommend that you create a custom evaluation report. If you choose to use one of the other
reports it’s advisable to both carefully read the documentation about the report and test the
validity of the data by calculating a sample of the results by hand. Note that none of the reports
list the number of evaluations used in calculating the average.
Competency Report
Competency Report Notes:
The Competency Report includes all evaluations that have been completed for an individual
including those by other departments that might use a different grade scale. If the evaluations
included in the averages have differing grade scales, the results are so misleading that the report
should not be used. For example, if Resident A is rated average by his evaluators using a
questionnaire with a 5-point scale, he’ll have an average rating of 3. If Resident B is rated
average by his evaluators using a questionnaire with a 9-point scale, the report will show that his
average rating is 5. Before you use this report you need to be sure that everyone is graded using
the same scale.
Competency Report Example
Trainee’s name and date range of evaluations
Professionalism
{* Peer Avg: 3.08} {My Avg: 3.00 } {My Min: 3.00 } {My Max: 3.00 } {Std Dev: 0.00}
Medical Knowledge
{* Peer Avg: 2.79} {My Avg: 2.50 } {My Min: 2.00 } {My Max: 3.00 } {Std Dev: 0.71}
Patient Care
{* Peer Avg: 2.85} {My Avg: 2.50 } {My Min: 2.00 } {My Max: 3.00 } {Std Dev: 0.58}
Practice-based Learning
{* Peer Avg: 2.89} {My Avg: 2.50 } {My Min: 2.00 } {My Max: 3.00 } {Std Dev: 0.71}
Interpersonal & Communication Skills
10/7/11
Page 17 of 24
{* Peer Avg: 3.07} {My Avg: 3.00 } {My Min: 3.00 } {My Max: 3.00 } {Std Dev: 0.00}
Systems-Based Practice
{* Peer Avg: 2.95} {My Avg: 2.50 } {My Min: 2.00 } {My Max: 3.00 } {Std Dev: 0.71}
Evaluation Summary Report
Evaluation Summary Report Notes
The Evaluation Summary Report includes all evaluations that have been completed for an
individual including those by other departments that might use a different numeric grade scale. If
the evaluations included in the averages have differing grade scales, the results are so
misleading that the report should not be used. For example, if Resident A is rated average by his
evaluators using a questionnaire with a 5-point scale, he’ll have an average rating of 3. If
Resident B is rated average by his evaluators using a questionnaire with a 9-point scale, the
report will show that his average rating is 5. Before you use this report you need to be sure that
everyone is graded using the same numeric scale.
Evaluation Summary Report Example
Name of trainee
Name of questionnaire
Dates of evaluation sessions
List of rotations and dates
Medical Knowledge - (1-7)
AVG:3.0 (average score for this individual)
3,3,3,3,3,3 (individual scores that this trainee received)
Patient Care - Cognitive - (1-7)
AVG:3.2
3,3,4,3,3,3
Patient Care - Technical - (1-7)
AVG:3.2
3,3,3,4,3,3
Peer Average Report
Peer Average Report Notes
10/7/11
Page 18 of 24
The report allows you to select the questionnaires whose results are included in the report. For
accurate data, you need to make sure that the questionnaires you include in a report all use the
same numeric grade scale and that all questions within a core competency use the same grade
scale.
One graph for all residents/fellows.
Difficult to see the highest ranking people. Most effective is small numbers of trainees.
People whose scores are identical cover each other’s graphs
Peer Average Report Example
Peer Comparison Graph
Peer Comparison Graph Notes
The report allows you to select the questionnaires whose results are included in the report. For
accurate data, you need to make sure that the questionnaires you include in a report all use the
same numeric grade scale and that all questions within a core competency use the same grade
scale.
One graph for each person comparing him/her to the average of all trainees.
Peer Comparison Graph Example
10/7/11
Page 19 of 24
Person Statistics Report
Person Statistics Example
Ranking Report
Ranking Report Notes
Assumes that every grade scale used in the questionnaire lists the choices from low to high, left
to right. For example, if the grade scale is “Poor – Average – Good – Excellent”, you will get valid
results. If the grade scale used throughout the report is “Excellent – Good – Average – Poor”,
10/7/11
Page 20 of 24
then the results will be the exact opposite of what the scores in the report suggest. Use this report
with caution.
Ranking Report Example
Other Modules in New Innovations
Conferences
Conferences are teaching sessions for residents or fellows at which outside experts may present
or residents and fellows may present information. The conference module allows the
administrator to schedule the date, time, and location of the conferences and automatically email
residents and faculty a few days before each conference. The administrator can also take
attendance and can automate the sending of evaluation forms or questionnaires about the
conference to all attendees.
Continuity Clinics
The continuity clinics module allows residents and fellows to record information about the patients
they see during continuity clinics. Each program configures the type of data it wants captured.
Custom Reports
You can use any of dozens of data sources and fields to create a custom report providing the
data you need. The GME Office creates a “non-responders” report every month that you can use
10/7/11
Page 21 of 24
to determine which of your residents or fellows have not yet responded to the duty hour
evaluation due on the 21st of each month.
Curriculum
The Curriculum module is where you store curricula for your rotations. You can require that
residents/fellows confirm (read) the curriculum for each rotation they are on and can produce
reports tracking their compliance rate.
Log Books
Log books is a general purpose logging module that can be used to track patients that a resident
sees or his/her reaction to conferences or a journal about his/her training experiences.
Portfolio Review
The ACGME recommends that residents and fellows’ performance reviews consider all aspects
of a trainee’s education. A portfolio review allows an administrator to gather together on one web
page, a summary of all information that NI contains about a resident or fellows. For example, the
report might contain the average scores per core competency that a resident or fellow received
on evaluation forms, the % of required conferences that a resident attended, a list of the rotations
that the resident was on, a list of scholarly activities that the resident engaged in, the % of
evaluations that a resident completed, the average number of hours per week that a resident
worked. Portfolio Reviews provides 18 “widgets” which pull data from various modules in NI and
which are customizable by the administrator.
10/7/11
Page 22 of 24
10/7/11
Page 23 of 24
Procedure Logger
Some programs require that their residents log procedures (e.g., tracheotomy, central line
placement) so that the program can determine if a resident is certified to perform the procedure
independently. When logging a procedure, the resident indicates his/her role in the procedure
(observer, assistant, etc.) and indicates the name of the attending who supervised the procedure.
The attending must then “confirm” the procedure. The program administrator configures the
number of times a resident must perform the procedure independently with confirmation to
become credentialed in the procedure.
The residents of some programs log their procedures on the ACGME web site. These procedures
can be downloaded to NI.
Program Dashboard
A one-page display of the 11 major modules in NI that show how a program is performing in all
areas: duty hours (rate of compliance with rules), scheduling (is it complete), personnel data, (is it
complete), etc. The Program Dashboard is accessed through the Administration module.
Help
NI offers webinars and a lot of documentation about NI. There is a "Help" button in the top right
corner of every page. The documentation includes 20-30 minute webinars.
10/7/11
Page 24 of 24