What is Usability tests Birgitte Berg 1 Agenda 1 Introduction 2 Definition of an usability test 3 Usability test – formative/summative 4 How is an usability test done 5 Standards – FDA and IEC 6 Conclusion/Summary 7 Q&A 2 Definition of a usability test • User testing (face to face interview) conducted with representative end users to obtain direct information and observations about: – How people will use the system, – Whether they encounter problems, – And how they deal with any problems… • Procedure to assess usability and to determine whether usability goals have been achieved • Use of risk analysis to identify critical and essential tasks for testing (Definition from HE75) 14 September 2012 Formative test • Formative Usability Testing Definition (HE75): Usability testing that is performed early with simulations and the earliest working prototypes and that explores whether usability objectives are attainable, but without strict acceptance criteria • How used: “Early and often” during design to improve the interface and remove usability and use error risk problems when the design still can be modified 14 September 2012 Formative test • • • • One way mirror Interviewer Subject Moderator (data locker) • Video Usability test – how is it done? • Participant is – welcomed and trained in the device – asked to perform some handling and differentation tasks (”in use scenerios”) – observed while performing tasks and ”use errors” are carefully registered • Use error – Pattern of use failure that indicates a failure mode that is likely to occur with use and this has a reasonable possibility of predictability of occurence. • Note: Use errors can be addressed and minimized by the device designer and proactively identified through the use of techniques such as usability testing and hazard analysis. (Definition from HE75) 14 September 2012 Overview of types of usability testing and considerations from the HE 75 guidance document Type Description When in design cycle Minimum sample size (per group) Considerations Formative usability testing Exploratory High-level test of users performing tasks Conceptual design 5 to 8 more Based on simulations of early concepts that could be very lowfidelity paper prototypes or foamcore models. Useful to employ the “think aloud” protocol. Assessment Representative users performing real tasks (1:1 or two working together) Iteratively throughout the design cycle 5 to 8 Used to gauge whether usability objectives are obtainable. Early designs or computer simulations. Comparison (contrast) Comparison of two or more design alternatives During design 5 to 8 Used to decide if one Ul concept or prototype is better than another Comparison (competitive) Tests against competitors´ Uls During design 5 to 8 Used to learn about best Ul features of competitors. May be done early during conceptual design or at end of design cycle. End of design cycle At least 15 to 20 Validates the design with usability objectives as acceptance criteria. Should include training, documentation and labelling, and riskmitigation controls related to use-related hazards. Summative usability testing Validation Real users and real tasks are tested with final design 14 September 2012 Formative Test - Real users eg. Diabetes care • • • • • • 5 children 5 adults/elderly 5 caregivers 5 physicians/nurses 5 inpatient nurses 5 pharmacists • Total of 20-30 users • Each interview takes about 1hour 14 September 2012 Impairments • If participants with real impairment can’t be recruited, impairments can be simulated in terms of – Colour blindness (Variantor glasses) – Blurred vision – Hearing problems – Manual dexterities 14 September 2012 Worst case conditions: Distractions • Distractions will be presented for some participants during the test in terms of: – Telephone call (conversation with participant regarding selling a news paper) – Participant requested to leave the room – Informal conversation during handling – Dimmed light 14 September 2012 Formative test – overall risk management process • Formative Tests are used to identify early use related hazards in initial hazards in initial designs of the user interface • May test only certain aspects of the interaction that have been designed and prototyped • Are constantly looking for potential ways to eliminate use error hazards and improve overall design 14 September 2012 What formative Usability Test is NOT • Focus group – These are not tests of device use or risks in device interaction, they provide a different kind of information – This approach is useful measuring opinions, attitudes and speculation about a product including a device • Customer Preference surveys • Clinical device Studies • Anecdotal information from – Early equipment trials – Training session 14 September 2012 Design Tools: standards (Best Practice) Reviews • Use HE-75 or other user interface design standards as a guideline for considerations of best practices for – Display design, use of colour, brightness, contrast etc. – Software layout, menu structure etc. – Control design: buttons, dials etc. – Labelling: quick reference, user manuals – Auditory alarms: frequency, sequence, loudness etc. – Other aspects of UI……… 14 September 2012 Interface Design Tools: Rapid Prototyping • Used early in design stage (can be low tech paper based) or can be simulated on a screen • Quick turnaround for formative usability testing of prototypes and discovering use errors potential induced by deign features • Rapid iterative design • Requires minimum development efforts 14 September 2012 Common Formative Testing Mistakes • • • • • • • • NOT pre-testing protocol Leading or biasing the user Helping to get the tasks completed Talking too much Rushing the subject Making the subject feel inferior Not making sessions friendly and interactive Not keeping track of timing and having to rush to finish all tasks in the protocol 14 September 2012 Formative Evaluation Summary • Formative evaluations and testing should be an iterative process in the design of medical device interfaces • A variety of techniques may be used to modify the design to maximise user efficiency and reduce use errors • Prototypes and working models can be valuable ways of evaluating interface options in the hands of small number of users • Document your formative evaluations! This is vital to the overall pre-market human factors submission to FDA 14 September 2012 Definition of a usability test • Summative Usability Testing Usability testing performed in the late stages of design Note : Summative usability tests include verification and validation, and it is a recommended best practice to have a formal acceptance criteria (e.g. usability objectives for human performance and satisfaction ratings) • How used? Testing interaction with the device in a simulated or actual environment. The “proof” of safe use for FDA submissions (Definition from HE75) 14 September 2012 Summative test (Validation test) - purpose • This is the culmination of Human Factor Usability efforts – It is assumed that modifications of the User Interface resulting from formative evaluation are implemented including modifications to labelling and training • If preliminary analyses, formative evaluations and design modifications and development was done well, the validation testing should find good results 14 September 2012 Summative test (Validation test) - purpose • The test is the primary evidence to the Authorities – To support the conclusion that the final user interface design is reasonably safe for intended users, uses and use environments 14 September 2012 HF/Usability Validation • Should be “generalizable” to expected actual use – Validation testing is a mini representation of actual use – Representative users (e.g. diabetes patients) – Use environments (home settings, hospitals etc) – Systematic observation of task performance and – Subjective impressions of test participants 14 September 2012 Overview of types of usability testing and considerations from the HE 75 guidance document Type Description When in design cycle Minimum sample size (per group) Considerations Formative usability testing Exploratory High-level test of users performing tasks Conceptual design 5 to 8 more Based on simulations of early concepts that could be very lowfidelity paper prototypes or foamcore models. Useful to employ the “think aloud” protocol. Assessment Representative users performing real tasks (1:1 or two working together) Iteratively throughout the design cycle 5 to 8 Used to gauge whether usability objectives are obtainable. Early designs or computer simulations. Comparison (contrast) Comparison of two or more design alternatives During design 5 to 8 Used to decide if one Ul concept or prototype is better than another Comparison (competitive) Tests against competitors´ Uls During design 5 to 8 Used to learn about best Ul features of competitors. May be done early during conceptual design or at end of design cycle. End of design cycle At least 15 to 20 Validates the design with usability objectives as acceptance criteria. Should include training, documentation and labelling, and riskmitigation controls related to use-related hazards. Summative usability testing Validation Real users and real tasks are tested with final design 14 September 2012 Summative Test - Real users eg. Diabetes care • • • • • • 15 children 15 adults/elderly 15 caregivers 15 physicians/nurses 15 inpatient nurses 15 pharmacists • Total of 100 – 120 users • Each interview takes about 1½ hour 14 September 2012 Conclusion and decisions after usability test Analysis 1 2 Final conclusive report Usability test -> CRO report Device failed: Repeat circle Device passed: End 14 September 2012 3 Regulatory framework & guidance • • • • FDA draft guidance: Applying Human Factors Usability and Engineering to optimize Medical device Design (FDA 2011) ANSI/AAMI HE75:2009, Human factors engineering – Design of medical devices, developed by Association for the Advancement of Medical Instrumentation IEC 62366:2007, Medical devices -- Application of usability engineering to medical devices, published by the International Electro-technical Commission (IEC). Medical Device Safety – Incorporating Human Factors Engineering into Risk Management, July 18, 2000, available on the U.S. Food and Drug Administration’s website 14 September 2012 Courses • AAMI (Association for the advancement of Medical Instrumentation) http://www.aami.org/meetings/hf/index.html course in March 2013 in Baltimore • DS - Brugervenlighed for medicinsk udstyr Introduktion til standarden DS/EN 62366:2008, 11. oktober 2012. 14 September 2012 Klinisk afprøvning /usability Lovgivning • Rådet Direktiv 93/42/EEC om medicinsk udstyr med ændringer (Rådets Direktiv 2007/47/EC) • Ministeriet for Sundhed og Forebyggelses bekendtgørelse nr. 1263 af 15. december 2008 om medicinsk udstyr • DS/EN ISO 14155:2011: Klinisk undersøgelse af medicinsk udstyr til brug på mennesker • DS/EN 62366:2008: Medicinsk udstyr – Indbyggelse af anvendelighed i medicinsk udstyr • ANSI/AAMI HE75: 2009, Human Factors Engineering – Design of Medical Devices. www.medicinskudstyr.dk / www.ds.dk / www.fda.gov MTIC Klinisk afprøvning af medicinsk udstyr 28. August 2012 Rikke Arendt Christiansen Helene Quie Qmed Consulting ApS • Niels Juelsgade 11 • 4600 Køge • Denmark • TEL.: +45 5464 1010 FAX: +45 56 653 842 Mail: [email protected] Klinisk afprøvning /usability Lovgivning • Rådet Direktiv 93/42/EEC om medicinsk udstyr med ændringer (Rådets Direktiv 2007/47/EC) • Ministeriet for Sundhed og Forebyggelses bekendtgørelse nr. 1263 af 15. december 2008 om medicinsk udstyr • DS/EN ISO 14155:2011: Klinisk undersøgelse af medicinsk udstyr til brug på mennesker • DS/EN 62366:2008: Medicinsk udstyr – Indbyggelse af anvendelighed i medicinsk udstyr • ANSI/AAMI HE75: 2009, Human Factors Engineering – Design of Medical Devices. www.medicinskudstyr.dk / www.ds.dk / www.fda.gov Qmed Consulting ApS • Niels Juelsgade 11 • 4600 Køge • Denmark • TEL.: +45 5464 1010 FAX: +45 56 653 842 Mail: [email protected] Definitioner MDD definition – Annex I Udstyret skal konstrueres og fremstilles på en sådan måde, at det ikke forværrer patientens kliniske tilstand eller bringer vedkommendes sikkerhed i fare og heller ikke er til fare for brugerens og en eventuel tredjemands sikkerhed Det indebærer: • at risici i forbindelse med forkert brug som følge af udstyrets ergonomiske karakteristika og de omgivelser, hvori det skal anvendes (design for patientsikkerhed), begrænses i videst mulig omfang, og • at der tages hensyn til brugernes teknologiske viden, erfaring og uddannelse, og hvor det er hensigtsmæssigt, de lægelige og fysiske omstændigheder (design for lægfolk, erhvervsfolk, handicappede eller andre bruger). Qmed Consulting ApS • Niels Juelsgade 11 • 4600 Køge • Denmark • TEL.: +45 5464 1010 FAX: +45 56 653 842 Mail: [email protected] Definitioner MDD definition – Annex I Risikoen for enhver bivirkning og uønsket følgevirkning skal stå i et acceptabelt forhold til den angivne ydeevne. Påvisningen af overensstemmelse med de væsentlige krav skal omfatte en klinisk evaluering i overensstemmelse med bilag X. Qmed Consulting ApS • Niels Juelsgade 11 • 4600 Køge • Denmark • TEL.: +45 5464 1010 FAX: +45 56 653 842 Mail: [email protected] Definitioner MDD definition - klinisk data Sikkerhedsdata eller data om ydeevne, der stammer fra brugen af medicinsk udstyr. Kliniske data indsamles ved: a) klinisk(e) afprøvning(er) af det medicinske udstyr, b) klinisk(e) afprøvning(er) eller andre undersøgelser, der er offentliggjort i den videnskabelige litteratur, af lignende medicinsk udstyr, hvis ækvivalens med det pågældende medicinske udstyr kan godtgøres, eller c) offentliggjorte eller ikke-offentliggjorte rapporter om anden klinisk erfaring med enten det pågældende medicinske udstyr eller et lignende medicinsk udstyr, hvis ækvivalens med det pågældende medicinske udstyr kan godtgøres. Qmed Consulting ApS • Niels Juelsgade 11 • 4600 Køge • Denmark • TEL.: +45 5464 1010 FAX: +45 56 653 842 Mail: [email protected] Definitioner MDD definition - klinisk evaluering Bekræftelsen af, at et medicinsk udstyr under normale anvendelsesforhold opfylder kravene med hensyn til karakteristika og ydeevne, samt vurderingen af bivirkninger og acceptabiliteten af forholdet mellem fordele og risici En klinisk evaluering skal følge en defineret og metodologisk forsvarlig fremgangsmåde baseret på kritisk evaluering af: 1. Relevant, tilgængelig videnskabelig litteratur, eller 2. Resultaterne af alle gennemførte kliniske afprøvninger, eller 3. De kombinerede kliniske data i punkt 1 og 2. Qmed Consulting ApS • Niels Juelsgade 11 • 4600 Køge • Denmark • TEL.: +45 5464 1010 FAX: +45 56 653 842 Mail: [email protected] Definitioner ISO 14155 definition - klinisk afprøvning Ethvert forsøg på mennesker, der har til formål at afdække eller efterprøve sikkerheden og/eller ydeevnen af medicinsk udstyr. DS/EN 62366 definition – brugervenlighed Karakteristika ved brugerfladen, som dokumenterer at denne er nem at bruge, er designet rigtigt, og fungerer som tiltænkt/beskrevet til brugeres tilfredshed Qmed Consulting ApS • Niels Juelsgade 11 • 4600 Køge • Denmark • TEL.: +45 5464 1010 FAX: +45 56 653 842 Mail: [email protected] Usability vs klinisk afprøvning Eksempler • Plejeseng med ny robotteknologi til styring af seng, der muliggør selvbetjening af seng for personer med stærk nedsat bevægelsesevne • Sensorer til måling af muskelaktivitet i forbindelse med genoptræning • Software til beregning af patienters sundhedstilstand efter indtastning af vitale parametre Qmed Consulting ApS • Niels Juelsgade 11 • 4600 Køge • Denmark • TEL.: +45 5464 1010 FAX: +45 56 653 842 Mail: [email protected] Klinisk afprøvning af medicinsk udstyr - processen Qmed Consulting ApS • Niels Juelsgade 11 • 4600 Køge • Denmark • TEL.: +45 5464 1010 FAX: +45 56 653 842 Mail: [email protected] Klinisk afprøvning Planlægning • Deltagere i planlægning og gennemførsel af en klinisk afprøvning skal have dokumenteret træning/ekspertise inden for området • Begrundelse for gennemførsel og design af en klinisk afprøvning bør have baggrund i en gennemgang af relevant litteratur, pre-klinisk dokumentation og kliniske risici identificeret • Er produktet anvendelig til det formål og de brugere/patienter, som det er tiltænkt til. • Resultaterne skal være klinisk relevante, opnået vha. videnskabelige principper og understøtte formålet med afprøvningen Qmed Consulting ApS • Niels Juelsgade 11 • 4600 Køge • Denmark • TEL.: +45 5464 1010 FAX: +45 56 653 842 Mail: [email protected] Klinisk afprøvning Dokumentation • • • • • • • • • • Klinisk afprøvningsplan (CIP) Investigator’s Brochure (IB) Case Report Forms (CRF) Patientinformation (PIC) og informeret samtykke (ICF) Tilladelser – (etisk komite, sundhedsstyrelsen) Kontrakter Monitoreringsplan Statistisk analyse plan (SAP) Kvalitetssystem til kliniske afprøvninger Budget Qmed Consulting ApS • Niels Juelsgade 11 • 4600 Køge • Denmark • TEL.: +45 5464 1010 FAX: +45 56 653 842 Mail: [email protected] Klinisk afprøvning CIP • • • • • • • • • • • ID af dokument (fx version), involverede parter (Sponsor, Investigatorer) Synopsis Beskrivelse af udstyret Design af afprøvning Formål og hypoteser Beskrivelse af patientpopulation (fx in- og eksklusionskriterier) Procedurer Monitorering Statistik Håndtering af hændelser/bivirkninger Kvalitetssikring (fx datahåndtering, tidlig afslutning, afvigelser) Qmed Consulting ApS • Niels Juelsgade 11 • 4600 Køge • Denmark • TEL.: +45 5464 1010 FAX: +45 56 653 842 Mail: [email protected] Klinisk afprøvning IB • • • • • • • • • Resume af litteratur Generel beskrivelse af udstyret Udstyrets virkemåde, herunder brugsanvisning Forventet klinisk ydeevne Materialer Præ-kliniske test – resume Resume af tidligere klinisk erfaring Liste over anvendte standarder Resultat af risikoanalyse Qmed Consulting ApS • Niels Juelsgade 11 • 4600 Køge • Denmark • TEL.: +45 5464 1010 FAX: +45 56 653 842 Mail: [email protected] Klinisk afprøvning Andre dokumenter • CV - investigatorer, projektsygeplejersker • Identifikation af institution, der skal deltage i afprøvningen – fx Aarhus Universitetshospital, Klinisk Gastroenterologisk Afdeling • Etisk komite – afgørelse, korrespondance • Kompetente myndigheder – afgørelse, korrespondance • Aftaler – Sponsor/investigator og andre fx laboratorier, CRO • Forsikring (hvis relevant) • Patientinformation/Informeret samtykke • Case report forms • Formularer til indrapportering af hændelser Qmed Consulting ApS • Niels Juelsgade 11 • 4600 Køge • Denmark • TEL.: +45 5464 1010 FAX: +45 56 653 842 Mail: [email protected] Klinisk afprøvning Før afprøvningen går i gang • • • • • • • ”Holdsammensætning” Færdiggørelse af al dokumentation Fremstilling af produkter til afprøvning Kontrakter Monitoreringsplan Træning/uddannelse Tilladelser – etisk komite, kompetente myndigheder, datatilsynet Qmed Consulting ApS • Niels Juelsgade 11 • 4600 Køge • Denmark • TEL.: +45 5464 1010 FAX: +45 56 653 842 Mail: [email protected] Klinisk afprøvning Under afprøvningen • • • • • Produkthåndtering – lagerstyring Monitorering – kildedataverificering, device accountability, hændelser, CRF Indrapportering af hændelser, afvigelser Overvågning af afprøvningen – fx sikkerhedskomite Årlige sikkerhedsrapporter Qmed Consulting ApS • Niels Juelsgade 11 • 4600 Køge • Denmark • TEL.: +45 5464 1010 FAX: +45 56 653 842 Mail: [email protected] Klinisk afprøvning Efter afprøvningen • • • • • Returnering af produkter Sikre at dokumentation på site er i orden Statistisk analyse Endelig rapport (indsendes til myndigheder og etisk komite) Opbevaring/arkivering af data Qmed Consulting ApS • Niels Juelsgade 11 • 4600 Køge • Denmark • TEL.: +45 5464 1010 FAX: +45 56 653 842 Mail: [email protected] Klinisk afprøvning Hvad skal kliniske data så bruges til? • • • • • • Dokumentation til endelig CE - mærkning Input til nye mulige/nødvendige afprøvninger Forlængelse af eksisterende studie Publikationsplan/strategi Feedback til Marketing/salg Planlægning af post-market surveillance, PMS (markedsovervågning) Qmed Consulting ApS • Niels Juelsgade 11 • 4600 Køge • Denmark • TEL.: +45 5464 1010 FAX: +45 56 653 842 Mail: [email protected] Spørgsmål? Qmed Consulting ApS • Niels Juelsgade 11 • 4600 Køge • Denmark • TEL.: +45 5464 1010 FAX: +45 56 653 842 Mail: [email protected]
© Copyright 2024