What is
Usability tests
Birgitte Berg
1
Agenda
1
Introduction
2
Definition of an usability test
3
Usability test – formative/summative
4
How is an usability test done
5
Standards – FDA and IEC
6
Conclusion/Summary
7
Q&A
2
Definition of a usability test
• User testing (face to face interview) conducted with
representative end users to obtain direct information
and observations about:
– How people will use the system,
– Whether they encounter problems,
– And how they deal with any problems…
• Procedure to assess usability and to determine whether
usability goals have been achieved
• Use of risk analysis to identify critical and essential tasks
for testing
(Definition from HE75)
14 September 2012
Formative test
• Formative Usability Testing
Definition (HE75):
Usability testing that is performed early with simulations
and the earliest working prototypes and that explores
whether usability objectives are attainable, but without
strict acceptance criteria
• How used: “Early and often” during design to improve
the interface and remove usability and use error risk
problems when the design still can be modified
14 September 2012
Formative test
•
•
•
•
One way mirror
Interviewer
Subject
Moderator (data
locker)
• Video
Usability test – how is it done?
•
Participant is
– welcomed and trained in the device
– asked to perform some handling and differentation tasks (”in use scenerios”)
– observed while performing tasks and ”use errors” are carefully registered
•
Use error
– Pattern of use failure that indicates a failure mode that is likely to occur with use
and this has a reasonable possibility of predictability of occurence.
• Note: Use errors can be addressed and minimized by the device designer and
proactively identified through the use of techniques such as usability testing and hazard
analysis.
(Definition from HE75)
14 September 2012
Overview of types of usability testing and
considerations from the HE 75 guidance document
Type
Description
When in design
cycle
Minimum
sample size
(per group)
Considerations
Formative usability testing
Exploratory
High-level test of users
performing tasks
Conceptual
design
5 to 8 more
Based on simulations of early
concepts that could be very lowfidelity paper prototypes or foamcore models. Useful to employ the “think
aloud” protocol.
Assessment
Representative users
performing real tasks (1:1
or two working together)
Iteratively
throughout the
design cycle
5 to 8
Used to gauge whether usability
objectives are obtainable. Early
designs or computer simulations.
Comparison
(contrast)
Comparison of two or
more design alternatives
During design
5 to 8
Used to decide if one Ul concept or prototype
is better than another
Comparison
(competitive)
Tests against
competitors´ Uls
During design
5 to 8
Used to learn about best Ul
features of competitors. May be done early
during conceptual
design or at end of design cycle.
End of design
cycle
At least 15 to
20
Validates the design with usability
objectives as acceptance criteria.
Should include training,
documentation and labelling, and riskmitigation controls related to use-related
hazards.
Summative usability testing
Validation
Real users and real tasks
are tested with final design
14 September 2012
Formative Test - Real users eg.
Diabetes care
•
•
•
•
•
•
5 children
5 adults/elderly
5 caregivers
5 physicians/nurses
5 inpatient nurses
5 pharmacists
• Total of 20-30 users
• Each interview takes about 1hour
14 September 2012
Impairments
• If participants with real impairment can’t be
recruited, impairments can be simulated in
terms of
– Colour blindness (Variantor glasses)
– Blurred vision
– Hearing problems
– Manual dexterities
14 September 2012
Worst case conditions:
Distractions
• Distractions will be presented for some
participants during the test in terms of:
– Telephone call (conversation with participant
regarding selling a news paper)
– Participant requested to leave the room
– Informal conversation during handling
– Dimmed light
14 September 2012
Formative test – overall risk
management process
• Formative Tests are used to identify early use
related hazards in initial hazards in initial
designs of the user interface
• May test only certain aspects of the interaction
that have been designed and prototyped
• Are constantly looking for potential ways to
eliminate use error hazards and improve overall
design
14 September 2012
What formative Usability Test is
NOT
• Focus group
– These are not tests of device use or risks in device
interaction, they provide a different kind of information
– This approach is useful measuring opinions,
attitudes and speculation about a product including
a device
• Customer Preference surveys
• Clinical device Studies
• Anecdotal information from
– Early equipment trials
– Training session
14 September 2012
Design Tools: standards (Best
Practice) Reviews
• Use HE-75 or other user interface design standards as a
guideline for considerations of best practices for
– Display design, use of colour, brightness, contrast
etc.
– Software layout, menu structure etc.
– Control design: buttons, dials etc.
– Labelling: quick reference, user manuals
– Auditory alarms: frequency, sequence, loudness etc.
– Other aspects of UI………
14 September 2012
Interface Design Tools: Rapid
Prototyping
• Used early in design stage (can be low
tech paper based) or can be simulated on
a screen
• Quick turnaround for formative usability
testing of prototypes and discovering use
errors potential induced by deign features
• Rapid iterative design
• Requires minimum development efforts
14 September 2012
Common Formative Testing
Mistakes
•
•
•
•
•
•
•
•
NOT pre-testing protocol
Leading or biasing the user
Helping to get the tasks completed
Talking too much
Rushing the subject
Making the subject feel inferior
Not making sessions friendly and interactive
Not keeping track of timing and having to rush to finish
all tasks in the protocol
14 September 2012
Formative Evaluation Summary
• Formative evaluations and testing should be an iterative
process in the design of medical device interfaces
• A variety of techniques may be used to modify the
design to maximise user efficiency and reduce use
errors
• Prototypes and working models can be valuable ways of
evaluating interface options in the hands of small
number of users
• Document your formative evaluations! This is vital to the
overall pre-market human factors submission to FDA
14 September 2012
Definition of a usability test
• Summative Usability Testing
Usability testing performed in the late stages of design
Note : Summative usability tests include verification and validation,
and it is a recommended best practice to have a formal acceptance
criteria (e.g. usability objectives for human performance and
satisfaction ratings)
• How used? Testing interaction with the device in a simulated or
actual environment. The “proof” of safe use for FDA submissions
(Definition from HE75)
14 September 2012
Summative test (Validation test)
- purpose
• This is the culmination of Human Factor Usability efforts
– It is assumed that modifications of the User Interface
resulting from formative evaluation are implemented
including modifications to labelling and training
• If preliminary analyses, formative evaluations and design
modifications and development was done well, the
validation testing should find good results
14 September 2012
Summative test (Validation test)
- purpose
• The test is the primary evidence to the
Authorities
– To support the conclusion that the final user
interface design is reasonably safe for intended
users, uses and use environments
14 September 2012
HF/Usability Validation
• Should be “generalizable” to expected
actual use
– Validation testing is a mini representation of
actual use
– Representative users (e.g. diabetes patients)
– Use environments (home settings, hospitals
etc)
– Systematic observation of task performance
and
– Subjective impressions of test participants
14 September 2012
Overview of types of usability testing and
considerations from the HE 75 guidance document
Type
Description
When in design
cycle
Minimum
sample size
(per group)
Considerations
Formative usability testing
Exploratory
High-level test of users
performing tasks
Conceptual
design
5 to 8 more
Based on simulations of early
concepts that could be very lowfidelity paper prototypes or foamcore models. Useful to employ the “think
aloud” protocol.
Assessment
Representative users
performing real tasks (1:1
or two working together)
Iteratively
throughout the
design cycle
5 to 8
Used to gauge whether usability
objectives are obtainable. Early
designs or computer simulations.
Comparison
(contrast)
Comparison of two or
more design alternatives
During design
5 to 8
Used to decide if one Ul concept or prototype
is better than another
Comparison
(competitive)
Tests against
competitors´ Uls
During design
5 to 8
Used to learn about best Ul
features of competitors. May be done early
during conceptual
design or at end of design cycle.
End of design
cycle
At least 15 to
20
Validates the design with usability
objectives as acceptance criteria.
Should include training,
documentation and labelling, and riskmitigation controls related to use-related
hazards.
Summative usability testing
Validation
Real users and real tasks
are tested with final design
14 September 2012
Summative Test - Real users
eg. Diabetes care
•
•
•
•
•
•
15 children
15 adults/elderly
15 caregivers
15 physicians/nurses
15 inpatient nurses
15 pharmacists
• Total of 100 – 120 users
• Each interview takes about 1½ hour
14 September 2012
Conclusion and decisions after usability test
Analysis
1
2
Final conclusive report
Usability test
-> CRO report
Device failed: Repeat circle
Device passed: End
14 September 2012
3
Regulatory framework &
guidance
•
•
•
•
FDA draft guidance: Applying Human Factors Usability and Engineering to optimize
Medical device Design (FDA 2011)
ANSI/AAMI HE75:2009, Human factors engineering – Design of medical devices,
developed by Association for the Advancement of Medical Instrumentation
IEC 62366:2007, Medical devices -- Application of usability engineering to medical
devices, published by the International Electro-technical Commission (IEC).
Medical Device Safety – Incorporating Human Factors Engineering into Risk
Management, July 18, 2000, available on the U.S. Food and Drug Administration’s
website
14 September 2012
Courses
• AAMI (Association for the advancement of
Medical Instrumentation) http://www.aami.org/meetings/hf/index.html course in March 2013 in Baltimore
• DS - Brugervenlighed for medicinsk udstyr Introduktion til standarden DS/EN
62366:2008, 11. oktober 2012.
14 September 2012
Klinisk afprøvning /usability
Lovgivning
• Rådet Direktiv 93/42/EEC om medicinsk udstyr med ændringer
(Rådets Direktiv 2007/47/EC)
• Ministeriet for Sundhed og Forebyggelses bekendtgørelse nr. 1263
af 15. december 2008 om medicinsk udstyr
• DS/EN ISO 14155:2011: Klinisk undersøgelse af medicinsk udstyr til
brug på mennesker
• DS/EN 62366:2008: Medicinsk udstyr – Indbyggelse af
anvendelighed i medicinsk udstyr
• ANSI/AAMI HE75: 2009, Human Factors Engineering – Design of
Medical Devices.
www.medicinskudstyr.dk / www.ds.dk / www.fda.gov
MTIC
Klinisk afprøvning af medicinsk udstyr
28. August 2012
Rikke Arendt Christiansen
Helene Quie
Qmed Consulting ApS • Niels Juelsgade 11 • 4600 Køge • Denmark • TEL.: +45 5464 1010
FAX: +45 56 653 842 Mail: [email protected]
Klinisk afprøvning /usability
Lovgivning
• Rådet Direktiv 93/42/EEC om medicinsk udstyr med ændringer (Rådets
Direktiv 2007/47/EC)
• Ministeriet for Sundhed og Forebyggelses bekendtgørelse nr. 1263 af 15.
december 2008 om medicinsk udstyr
• DS/EN ISO 14155:2011: Klinisk undersøgelse af medicinsk udstyr til brug på
mennesker
• DS/EN 62366:2008: Medicinsk udstyr – Indbyggelse af anvendelighed i
medicinsk udstyr
• ANSI/AAMI HE75: 2009, Human Factors Engineering – Design of Medical
Devices.
www.medicinskudstyr.dk / www.ds.dk / www.fda.gov
Qmed Consulting ApS • Niels Juelsgade 11 • 4600 Køge • Denmark • TEL.: +45 5464 1010
FAX: +45 56 653 842 Mail: [email protected]
Definitioner
MDD definition – Annex I
Udstyret skal konstrueres og fremstilles på en sådan måde, at det ikke
forværrer patientens kliniske tilstand eller bringer vedkommendes
sikkerhed i fare og heller ikke er til fare for brugerens og en eventuel
tredjemands sikkerhed
Det indebærer:
• at risici i forbindelse med forkert brug som følge af udstyrets
ergonomiske karakteristika og de omgivelser, hvori det skal
anvendes (design for patientsikkerhed), begrænses i videst mulig
omfang, og
• at der tages hensyn til brugernes teknologiske viden, erfaring og
uddannelse, og hvor det er hensigtsmæssigt, de lægelige og fysiske
omstændigheder (design for lægfolk, erhvervsfolk, handicappede
eller andre bruger).
Qmed Consulting ApS • Niels Juelsgade 11 • 4600 Køge • Denmark • TEL.: +45 5464 1010
FAX: +45 56 653 842 Mail: [email protected]
Definitioner
MDD definition – Annex I
Risikoen for enhver bivirkning og uønsket følgevirkning skal stå i
et acceptabelt forhold til den angivne ydeevne.
Påvisningen af overensstemmelse med de væsentlige krav skal
omfatte en klinisk evaluering i overensstemmelse med bilag X.
Qmed Consulting ApS • Niels Juelsgade 11 • 4600 Køge • Denmark • TEL.: +45 5464 1010
FAX: +45 56 653 842 Mail: [email protected]
Definitioner
MDD definition - klinisk data
Sikkerhedsdata eller data om ydeevne, der stammer fra brugen af medicinsk
udstyr. Kliniske data indsamles ved:
a) klinisk(e) afprøvning(er) af det medicinske udstyr,
b) klinisk(e) afprøvning(er) eller andre undersøgelser, der er
offentliggjort i den videnskabelige litteratur, af lignende
medicinsk udstyr, hvis ækvivalens med det pågældende
medicinske udstyr kan godtgøres, eller
c) offentliggjorte eller ikke-offentliggjorte rapporter om anden klinisk
erfaring med enten det pågældende medicinske udstyr eller et
lignende medicinsk udstyr, hvis ækvivalens med det pågældende
medicinske udstyr kan godtgøres.
Qmed Consulting ApS • Niels Juelsgade 11 • 4600 Køge • Denmark • TEL.: +45 5464 1010
FAX: +45 56 653 842 Mail: [email protected]
Definitioner
MDD definition - klinisk evaluering
Bekræftelsen af, at et medicinsk udstyr under normale anvendelsesforhold
opfylder kravene med hensyn til karakteristika og ydeevne, samt vurderingen
af bivirkninger og acceptabiliteten af forholdet mellem fordele og risici
En klinisk evaluering skal følge en defineret og metodologisk forsvarlig
fremgangsmåde baseret på kritisk evaluering af:
1. Relevant, tilgængelig videnskabelig litteratur, eller
2. Resultaterne af alle gennemførte kliniske afprøvninger, eller
3. De kombinerede kliniske data i punkt 1 og 2.
Qmed Consulting ApS • Niels Juelsgade 11 • 4600 Køge • Denmark • TEL.: +45 5464 1010
FAX: +45 56 653 842 Mail: [email protected]
Definitioner
ISO 14155 definition - klinisk afprøvning
Ethvert forsøg på mennesker, der har til formål at afdække eller efterprøve
sikkerheden og/eller ydeevnen af medicinsk udstyr.
DS/EN 62366 definition – brugervenlighed
Karakteristika ved brugerfladen, som dokumenterer at denne er nem at bruge,
er designet rigtigt, og fungerer som tiltænkt/beskrevet til brugeres tilfredshed
Qmed Consulting ApS • Niels Juelsgade 11 • 4600 Køge • Denmark • TEL.: +45 5464 1010
FAX: +45 56 653 842 Mail: [email protected]
Usability vs klinisk afprøvning
Eksempler
• Plejeseng med ny robotteknologi til styring af seng, der
muliggør selvbetjening af seng for personer med stærk
nedsat bevægelsesevne
• Sensorer til måling af muskelaktivitet i forbindelse med
genoptræning
• Software til beregning af patienters sundhedstilstand efter
indtastning af vitale parametre
Qmed Consulting ApS • Niels Juelsgade 11 • 4600 Køge • Denmark • TEL.: +45 5464 1010
FAX: +45 56 653 842 Mail: [email protected]
Klinisk afprøvning af medicinsk udstyr
- processen
Qmed Consulting ApS • Niels Juelsgade 11 • 4600 Køge • Denmark • TEL.: +45 5464 1010
FAX: +45 56 653 842 Mail: [email protected]
Klinisk afprøvning
Planlægning
• Deltagere i planlægning og gennemførsel af en klinisk afprøvning skal have
dokumenteret træning/ekspertise inden for området
• Begrundelse for gennemførsel og design af en klinisk afprøvning bør have
baggrund i en gennemgang af relevant litteratur, pre-klinisk dokumentation
og kliniske risici identificeret
• Er produktet anvendelig til det formål og de brugere/patienter, som det er
tiltænkt til.
• Resultaterne skal være klinisk relevante, opnået vha. videnskabelige
principper og understøtte formålet med afprøvningen
Qmed Consulting ApS • Niels Juelsgade 11 • 4600 Køge • Denmark • TEL.: +45 5464 1010
FAX: +45 56 653 842 Mail: [email protected]
Klinisk afprøvning
Dokumentation
•
•
•
•
•
•
•
•
•
•
Klinisk afprøvningsplan (CIP)
Investigator’s Brochure (IB)
Case Report Forms (CRF)
Patientinformation (PIC) og informeret samtykke (ICF)
Tilladelser – (etisk komite, sundhedsstyrelsen)
Kontrakter
Monitoreringsplan
Statistisk analyse plan (SAP)
Kvalitetssystem til kliniske afprøvninger
Budget
Qmed Consulting ApS • Niels Juelsgade 11 • 4600 Køge • Denmark • TEL.: +45 5464 1010
FAX: +45 56 653 842 Mail: [email protected]
Klinisk afprøvning
CIP
•
•
•
•
•
•
•
•
•
•
•
ID af dokument (fx version), involverede parter (Sponsor, Investigatorer)
Synopsis
Beskrivelse af udstyret
Design af afprøvning
Formål og hypoteser
Beskrivelse af patientpopulation (fx in- og eksklusionskriterier)
Procedurer
Monitorering
Statistik
Håndtering af hændelser/bivirkninger
Kvalitetssikring (fx datahåndtering, tidlig afslutning, afvigelser)
Qmed Consulting ApS • Niels Juelsgade 11 • 4600 Køge • Denmark • TEL.: +45 5464 1010
FAX: +45 56 653 842 Mail: [email protected]
Klinisk afprøvning
IB
•
•
•
•
•
•
•
•
•
Resume af litteratur
Generel beskrivelse af udstyret
Udstyrets virkemåde, herunder brugsanvisning
Forventet klinisk ydeevne
Materialer
Præ-kliniske test – resume
Resume af tidligere klinisk erfaring
Liste over anvendte standarder
Resultat af risikoanalyse
Qmed Consulting ApS • Niels Juelsgade 11 • 4600 Køge • Denmark • TEL.: +45 5464 1010
FAX: +45 56 653 842 Mail: [email protected]
Klinisk afprøvning
Andre dokumenter
• CV - investigatorer, projektsygeplejersker
• Identifikation af institution, der skal deltage i afprøvningen – fx Aarhus
Universitetshospital, Klinisk Gastroenterologisk Afdeling
• Etisk komite – afgørelse, korrespondance
• Kompetente myndigheder – afgørelse, korrespondance
• Aftaler – Sponsor/investigator og andre fx laboratorier, CRO
• Forsikring (hvis relevant)
• Patientinformation/Informeret samtykke
• Case report forms
• Formularer til indrapportering af hændelser
Qmed Consulting ApS • Niels Juelsgade 11 • 4600 Køge • Denmark • TEL.: +45 5464 1010
FAX: +45 56 653 842 Mail: [email protected]
Klinisk afprøvning
Før afprøvningen går i gang
•
•
•
•
•
•
•
”Holdsammensætning”
Færdiggørelse af al dokumentation
Fremstilling af produkter til afprøvning
Kontrakter
Monitoreringsplan
Træning/uddannelse
Tilladelser – etisk komite, kompetente myndigheder, datatilsynet
Qmed Consulting ApS • Niels Juelsgade 11 • 4600 Køge • Denmark • TEL.: +45 5464 1010
FAX: +45 56 653 842 Mail: [email protected]
Klinisk afprøvning
Under afprøvningen
•
•
•
•
•
Produkthåndtering – lagerstyring
Monitorering – kildedataverificering, device accountability, hændelser, CRF
Indrapportering af hændelser, afvigelser
Overvågning af afprøvningen – fx sikkerhedskomite
Årlige sikkerhedsrapporter
Qmed Consulting ApS • Niels Juelsgade 11 • 4600 Køge • Denmark • TEL.: +45 5464 1010
FAX: +45 56 653 842 Mail: [email protected]
Klinisk afprøvning
Efter afprøvningen
•
•
•
•
•
Returnering af produkter
Sikre at dokumentation på site er i orden
Statistisk analyse
Endelig rapport (indsendes til myndigheder og etisk komite)
Opbevaring/arkivering af data
Qmed Consulting ApS • Niels Juelsgade 11 • 4600 Køge • Denmark • TEL.: +45 5464 1010
FAX: +45 56 653 842 Mail: [email protected]
Klinisk afprøvning
Hvad skal kliniske data så bruges til?
•
•
•
•
•
•
Dokumentation til endelig CE - mærkning
Input til nye mulige/nødvendige afprøvninger
Forlængelse af eksisterende studie
Publikationsplan/strategi
Feedback til Marketing/salg
Planlægning af post-market surveillance, PMS (markedsovervågning)
Qmed Consulting ApS • Niels Juelsgade 11 • 4600 Køge • Denmark • TEL.: +45 5464 1010
FAX: +45 56 653 842 Mail: [email protected]
Spørgsmål?
Qmed Consulting ApS • Niels Juelsgade 11 • 4600 Køge • Denmark • TEL.: +45 5464 1010
FAX: +45 56 653 842 Mail: [email protected]