1. health and fitness
2. disease

FACULTY OF MEDICINE AND
HEALTH SCIENCES
Academic Year 2011 - 2012
HOW TO PRUNE GUIDELINES FOR DECISION
SUPPORT SYSTEMS
Maarten DE SMET
Promotor: Prof. Dr. Robert Vander Stichele
Co-promotor: Dr. Tine De Backer
Dissertation presented in the 2nd Master year
in the programme of
Master of Medicine in Medicine
FACULTY OF MEDICINE AND
HEALTH SCIENCES
Academic Year 2011 - 2012
HOW TO PRUNE GUIDELINES FOR DECISION SUPPORT
SYSTEMS
Maarten DE SMET
Promotor: Prof. Dr. Robert Vander Stichele
Co-promotor: Dr. Tine De Backer
Dissertation presented in the 2nd Master year
in the programme of
Master of Medicine in Medicine
I
Preface
These past two years have been a fascinating experience for me. I never expected myself to be able to combine
informatics with medicine when I enlisted myself for this discipline. Hence my surprise when this research
project showed up in the list of thesis topics. As a dedicated Linux user, I was excited to step into this web of
freely shared information technology in the spirit of the open source philosophy. The open source idea originated
in the automobile industry at the start of the 20th century when Henry Ford defeated the Selden patent, a patent
on single type of engine used by the Association of Licensed Automobile Manufacturers to monopolise the auto
industry, in court. Afterwards, a new association – the Automobile Manufacturers Association – was formed that
represented a different economical ethics. All the US car manufacturers focused on establishing a code of fair
competition by instituting a cross-licensing agreement. From then on, all manufacturers would still develop new
technologies and apply for patents, but these patents would be shared with other manufacturers free of charge.
This philosophy of distributing details on development and implementation of new technologies was later
adopted by researchers in the 1960s and gave birth to the internet in 1969. Furthermore, software manufacturers
(e.g. IBM) would also exchange pieces of source code of their operating systems and other programs in the
1960s. In the early days of the World Wide Web, source code would also be exchanged via networks that
connected different users (e.g. Internet Relay Chat). That way, Linux was widely distributed for the first time. In
the late nineties, the term “open source” started to receive wider attention by promoting it at different events
(e.g. Freeware Summit of Open Source Summit). The great strength of open source lies in joining different
persons with different capacities and of different skill in an original project in order to improve the current
product or to create new projects. The strength of this unifying strategy is illustrated by the development of the
internet, the technological evolution of the automobile industry after installing the Automobile Manufacturers
Association and, more recently, the human genome project. I am honored to be part of such a philosophy. That is
also why – in the past two years – I seized the opportunity to be part of other research projects, mostly in
cardiovascular and molecular research. This project has opened a new world for me. Therefore, I would like to
thank Robert Vander Stichele and Tine De Backer for their continuous support, their expertise and their
friendship. I would also like to thank them for giving me the opportunity to get more out of this project than was
expected. Furthermore, I would like to thank Thierry Christiaens, Peter Gheeraert and Marc De Buyzere for their
medical expertise. Other thanks go out to Fondazione Bruno Kessler, Guidelines International Network, Koen
Van Herck, Annemie Heselmans, Jan Van Houcke, Michiel Van Parys and Manu Rabaey.
Maarten De Smet
May 2nd 2012
II
Table of Contents
Preface ..................................................................................................................................................... II
Table of Contents................................................................................................................................... III
Definitions .............................................................................................................................................. IV
Abstract .................................................................................................................................................... 1
Summary in Dutch .................................................................................................................................. 3
1.
Introduction ..................................................................................................................................... 6
2.
Methodology .................................................................................................................................... 8
3.
2.1.
Guideline Adherence .................................................................................................................. 8
2.2.
Computerised Clinical Decision Support Systems ..................................................................... 8
2.3.
Specific Practice Recommendations and GRADE ..................................................................... 9
2.4.
Delphi Procedure ..................................................................................................................... 15
Results ............................................................................................................................................ 18
3.1.
Guideline adherence and Quality of Care ................................................................................ 18
3.1.1.
Guideline Adherence ......................................................................................................... 18
3.1.2.
Identifying Guideline Implementation Barriers ................................................................ 25
3.1.3.
Characteristics of Effective Clinical Practice Guidelines ................................................. 31
3.1.4.
Consequences of Increased Guideline Adherence ............................................................ 32
3.2.
Computerised Clinical Decision Support Systems ................................................................... 35
3.3.
Specific Practice Recommendations ........................................................................................ 46
3.4.
Delphi Procedure ..................................................................................................................... 46
4.
Discussion....................................................................................................................................... 50
5.
Conclusion...................................................................................................................................... 52
6.
References ...................................................................................................................................... 53
Annex 1: AGREE II ................................................................................................................................. i
Annex 2: Antiplatelet therapy in cardiovascular prevention .............................................................. ii
III
Definitions
Guideline formalisation: the entire process of transforming original narrative guideline documents into a file
that can be operationalised by computer-based decision support systems, a computer-interpretable guideline.
Guideline specification: one of the steps in the formalisation process after the guideline is marked up. It is a
multidisciplinary process that involves replacing vague or abstract information with clear, operational
knowledge.
Mark-up: to provide a label, also called knowledge elements in knowledge engineering, for different parts of a
text. Mark-up can be performed structurally or semantically.
Knowledge elements: labels or tags that specify the semantic nature of marked up pieces of text, knowledge
elements are assigned to chunks or little pieces (subelements) of text during the mark-up process by using one of
the mark-up tools.
The files on a computer can be roughly divided into human-readable documents and computer-executable files.
Computer-executable files are generally not human-understandable. Because knowledge elements can provide
insights for humans into the contents of computer-executable files, they can bridge the gap between humanreadable documents and computer-executable files. Knowledge elements are closely associated with XML.
Examples of knowledge elements from the Guideline Elements Model: <identity>, <developer>, <purpose>,
<intendedaudience>,
<methodofdevelopment>,
<targetpopulation>,
<knowledgecomponents>,
<testing>,
<implementationplan>.
Examples of subelements for <purpose>from the Guideline Elements Model: <mainfocus>, <rationale>,
<objective>, <availableoption>, <healthoutcome>, <exception>
Document type definition (DTD): a document type definition is a scheme technology that defines the elements
that can and may be introduced in a certain type of document. DTDs are closely associated with XML, XHTML,
HGML and HTML. Because of these specifications, documents can be easily validated. For example, in
semantic mark-up a DTD may determine the tags which may occur in the XML-document.
IV
Abstract
Background: Clinical practice guidelines contain important diagnostic and therapeutic information
extracted from trials, reviews and meta-analyses. However, guideline compliance is unsatisfactory. Their
lengthy textual form and other implementation barriers hinder easy access to clinical information at the
point of care. In other words, a guideline implementation gap is present. Computerised decision support
systems may improve the implementation of guidelines by providing patient-specific information at the
point-of-care. The most effective computer-based implementation strategies integrate decision support
modules into existing clinical information systems, e.g. the electronic health record.
Aim: To collect recent studies that assessed compliance with guidelines, addressed implementation
barriers, described effective clinical practice guidelines or consequences of increased guideline adherence.
To collect existing methodologies for guideline formalisation into computer algorithms for electronic
decision support and rework them into a generic framework. To apply these methods to a previously
published evidence base for cardiovascular prevention.
Methods: A literature search was conducted in Pubmed and Web of Science for relevant publications.
The retrieved methods were applied to the field of pharmacological cardiovascular prevention by
developing a number of specific practice recommendations on the use of antiplatelet therapy in primary
cardiovascular prevention and by using a Delphi procedure to clarify any controversial issues. The specific
practice recommendations that were extracted from the Belgian consensus Conference Report on
cardiovascular preventions were subsequently assessed according to quality of evidence and strength of
recommendation. Only clinical situations for cardiovascular prevention without clear evidence, meaning
that the subjects are currently being discussed, were admitted to the Delphi procedure. The consensus
group consisted of two cardiologists, one cardiovascular researcher and two general practitioners. The
Delphi procedure ran for two rounds.
Results: Guideline adherence is generally unsatisfactory. Furthermore, guideline adherence varies greatly
between different medical conditions, different recommendations, different guidelines, different
geographical locations, different working environments, different types of recommendations and different
medical doctors. Barriers for following clinical practice guidelines can be dichotomised into internal and
external barriers. External barriers are behavior-related, while internal barriers are knowledge-related and
attitude related. Increasing guideline adherence is interesting because it leads to more quality of care, more
efficient care and more patient satisfaction. Guideline implementation strategies should aim at improving
these three aspects. 62 relevant publications were retrieved discussing different methodologies for
building computer-interpretable guidelines. Generally, two approaches for building computer-interpretable
1
guidelines exist; namely the document-centric and the model-centric approach. Both approaches have their
merits in specific medical domains. Document-centric approaches were discussed in detail because they
are well suited for the multifaceted topic of cardiovascular prevention. Document-centric approaches use a
multistep methodology for converting clinical practice guidelines into operational formats. These steps
generally include guideline selection and appraisal, guideline mark-up, guideline specification and
integration into clinical workflow. Guideline specification is a multidisciplinary process requiring
cooperation between medical domain experts and knowledge engineers. In document-centric approaches,
the formalisation process results in a transparent translation of a clinical practice guideline into a
computer-interpretable guideline because a link is maintained with the original guideline document. This
generic framework was partly illustrated by applying the methods to the Belgian Consensus Conference
Report on cardiovascular prevention. The consensus group was asked to reach consensus on three clinical
situations in primary cardiovascular prevention. Consensus was reached in two out of three situations. A
total of five specific practice recommendations were composed. Four were extracted from the consensus
document and one was obtained through the Delphi procedure.
Discussion: Computerised clinical decision support systems show promising results, but they are not
always effective. Today, no standard methodology for guideline formalisation exists. The large number of
computerised guideline implementation methodologies available provides an interoperability problem.
The development of a framework for the construction of a generic computer-interpretable document that
can be operationalised by different computerised clinical decision support systems should be addressed. A
small consensus panel, loss of anonymity in one round and low number of rounds were listed as
limitations of the Delphi procedure. Furthermore, the set of specific practice recommendations is not
complete. This issue needs to be addressed and the set of recommendations needs to be completed in the
guideline specification step.
Conclusions: Increased guideline adherence is associated with more quality of care, more cost-effective
care and more patient satisfaction, but guideline compliance is low. The existence of a guideline
implementation gap requires the use of implementation strategies to increase the adherence to clinical
practice guidelines. These strategies are very diverse, but there is consensus that computerised decision
support integrated into existing clinical information systems and integrated in the clinical workflow are
most effective. Multidisciplinary human formalisation of guidelines is a necessary prerequisite for
building a computerised clinical decision support system. The field of cardiovascular prevention, because
of its rich and sophisticated evidence base, is particularly apt to apply document-centered multi-faceted
guideline formalisation methodologies.
2
Summary in Dutch
Achtergrond: Klinische praktijkrichtlijnen bevatten belangrijke informatie over adequate diagnostische
en therapeutische handelingen. Deze informatie wordt verkregen door een systematisch en periodisch
proces van literatuuronderzoek. Op deze manier bevatten richtlijnen bewijs uit voorheen gepubliceerde
klinische trials, reviews en meta-analyses. Jammer genoeg blijkt dat richtlijnen slechts in de helft van de
patiëntencontacten gevolgd worden. Verschillende barrières weerhouden clinici ervan om richtlijnen te
volgen voor hun specifieke patiënt. Het verschil dat geobserveerd wordt tussen de ontwikkeling en
distributie van richtlijnen en hun correct gebruik in de dagelijkse praktijk kan overbrugd worden door
gebruik te maken van implementatie strategieën. Deze strategieën zijn zeer divers, maar in de
internationale literatuur zijn de onderzoekers het erover eens dat richtlijnen voor de klinische praktijk het
beste kunnen geïmplementeerd worden door gebruik te maken van hedendaagse informatietechnologie.
Deze klinische beslissingssystemen kunnen bijvoorbeeld gebruik maken van informatie uit het
elektronische patiëntendossier door deze te koppelen aan geprogrammeerde regels – in de vorm van
“als… dan…” – die opgesteld werden aan de hand van klinische richtlijnen.
Doel: Het verzamelen van recente studies die het aantal klinische beslissingen consistent met de klinische
richtlijnen onderzochten, studies die de attitudes van artsen ten opzichte van klinische richtlijnen en de
barrières voor correcte richtlijnimplementatie onderzochten en studies die de gevolgen van betere
implementatie van klinische richtlijnen onderzochten. Het verzamelen van verschillende methodologieën
om een klinische richtlijn te converteren naar een operationeel document en het toepassen van de eerste
stappen van een dergelijke – meer algemene – methodologie.
Materialen en methode: Er werd gezocht naar relevante publicaties voor de bovenstaande onderwerpen
in Pubmed en Web of Science. De gevonden methodes werden toegepast op de farmacologische
behandeling van het cardiovasculair risico, meer specifiek door het opstellen van specifieke aanbevelingen
voor de behandeling met anti-aggregantia in primaire cardiovasculaire preventie. Ook werden enkele
controversiële onderwerpen verder verduidelijkt door gebruik te maken van de opinie van experten via een
Delphi procedure. De specifieke aanbevelingen werden opgesteld aan de hand van het consensusrapport
van het RIZIV over cardiovasculaire preventie uit 2009. De aanbevelingen uit dit rapport werden ook
ingeschat naar kwaliteit van bewijs en kracht van aanbeveling met het GRADE instrument. In de Delphi
procedure werden enkel controversiële onderwerpen besproken. De consensusgroep bestond uit twee
cardiologen, één cardiovasculair onderzoeker en twee huisartsen. Twee ronden werden uitgevoerd binnen
deze procedure.
3
Resultaten: Het aantal klinische beslissingen in overeenstemming met de richtlijnen is onvoldoende.
Verder nog, de naleving van aanbevolen zorg varieert zeer sterk tussen verschillende aandoeningen,
verschillende richtlijnen, verschillende werkomgevingen, verschillende specifieke aanbevelingen en
tussen verschillende artsen. Barrières die het correct volgen van richtlijnen tegengaan kunnen grofweg
ingedeeld worden in interne en externe barrières. Interne barrières omvatten barrières die gerelateerd zijn
aan het gedrag van de arts, terwijl externe barrières kennis- en houding-gerelateerde barrières bevatten.
Het verhogen van zorg in overeenstemming met de richtlijnen is interessant omdat dit leidt tot een hogere
kwaliteit van zorg, een meer doelmatige zorg en een grotere tevredenheid van de patiënten. Strategieën die
erop gericht zijn de implementatie van richtlijnen in de dagelijkse praktijk te verbeteren, zouden deze drie
aspecten als doel moeten hebben. 62 relevante publicaties werden teruggevonden met betrekking tot het
transformeren van originele richtlijndocumenten in een document dat kan geïnterpreteerd worden door de
hedendaagse informatietechnologie. In het algemeen kunnen hiervoor twee benaderingen onderscheiden
worden, namelijk een aanpak die uitgaat van het originele document en een aanpak waarbij het originele
document eerst wordt omgezet tot een conceptueel model. Beide benaderingen hebben hun specifieke
voordelen in bepaalde medische gebieden. De benaderingen die uitgaan van het originele document
werden in dit werk in detail besproken omdat ze bijzonder compatibel zijn met het bovenstaande
onderwerp. Deze benaderingen maken gebruik van verschillende kleinere stappen om een tekstuele
richtlijn gradueel om te zetten tot een operationeel bestand. Deze stappen omvatten de selectie van een
kwaliteitsvolle richtlijn, mark-up, specificatie en integratie in de workflow van de arts. Specificatie van
richtlijnen is een multidisciplinair proces dat een goede samenwerking tussen artsen en kennisingenieurs
vergt. Doordat een link behouden wordt met het originele document kan het originele document op een
transparante manier omgezet worden in een document dat kan geïnterpreteerd worden door
computersystemen. Deze methodologie werd verder toegepast op het consensusdocument van het RIZIV
en aangevuld met een Delphi procedure. Vijf specifieke aanbevelingen werden geformuleerd voor het
gebruik van anti-aggregantia in de preventie van cardiovasculaire aandoeningen. Vier aanbevelingen
werden onttrokken uit het consensusdocument en één aanbeveling werd aanvaard door het
consensuspanel.
Discussie: Klinische beslissingssystemen die gebruik maken van informatietechnologie tonen
veelbelovende resultaten. Maar men dient er zich van bewust te zijn dat deze systemen niet altijd effectief
zijn. Tot op de dag van vandaag bestaat er geen gouden standaard om dergelijke systemen te construeren.
Er zijn daarentegen wel veel verschillende klinische beslissingssystemen die gebruik maken van
verschillende methodologieën om een toepasbare richtlijn te construeren, waardoor de computerinterpreteerbare richtlijnen niet interoperabel zijn. Er zijn slechts twee projecten (SAGE en SAPHIRE) die
dit probleem trachten aan te pakken. Er is nood aan de ontwikkeling van een methodologie die resulteert
4
in een richtlijn die uitvoerbaar is door de verschillende beslissingssystemen. Een kleine consensusgroep,
verlies van anonimiteit in één ronde en weinig rondes zijn de beperkingen van deze Delphi procedure. De
set van aanbevelingen is ook niet compleet, verdere stappen in het implementatieproces dienen deze set te
vervolledigen.
Conclusie: Zowel de kwaliteit van zorg, de doelmatigheid van zorg en het aantal tevreden patiënten
nemen toe als de zorg in overeenstemming met de richtlijnen toeneemt. Daarom zijn
implementatiestrategieën, die de lage compliance met richtlijnen verhogen, interessant. Deze strategieën
zouden de bovenstaande aspecten als doelen moeten nastreven. Deze strategieën zijn zeer divers, maar
men is het erover eens dat beslissingssystemen die gebruik maken van bestaande klinische
informatietechnologie en die geïntegreerd zijn in de workflow van de arts het meest efficiënt zijn. De
formalisatie van richtlijnen is een multidisciplinair proces dat deels menselijke input en deels input door
informatietechnologie vereist. Cardiovasculaire preventie is bijzonder geschikt – maar tegelijk ook
ingewikkeld – om een benadering die uitgaat van het originele richtlijndocument uit te testen in de
praktijk omwille van de vele evidentie omtrent dit onderwerp.
5
1. Introduction
In the early nineties, the Institute of Medicine defined clinical practice guidelines as “systematically
developed statements to assist practitioner and patient decisions about appropriate health care for specific
clinical circumstances”. [1-4] Today, clinical practice guidelines are considered an important tool in the
practice of medicine. [4-6] Clinical practice guidelines are evidence-based, meaning that they are
developed using a periodic and systematic search for evidence and critical review and evaluation of the
literature. [7-9] They are an inherent and important part of disease management strategies used to improve
patient outcomes for several reasons. [4,10] If clinical practice guidelines are applied correctly, they may
increase quality of care [4,11], decrease inappropriate care [4,12], result in a more efficient use of health
care resources [4,13] and decrease variation in clinical practice [4,11]. A final reason to support the
development and implementation of clinical practice guidelines is the increasing amount of new evidence
available every day. [14,15] Without clinical practice guidelines, health care providers need to process a
large amount of published evidence in order to determine the appropriate diagnostic or therapeutic
intervention for their patient. In guidelines, the evidence is summarised into a narrative document that
eliminates such a time-consuming and labor intensive task and that makes for easier clinical decision
making. [4] However, guidelines have had but little influence on physician behavior since their
introduction. [16-18] In other words, guideline compliance is low.
About 30-40% of the patients do not receive recommended care and 20-25% of patients even receive care
that is potentially hazardous. Furthermore, comparisons between older and more recent studies show that
guideline compliance has not improved. [4,19-26] Apparently, guideline development and distribution is
not enough to have a significant impact on physician behavior. A proverbial gap exists between the
development and the distribution of guidelines and their effective implementation in daily practice,
referred to as the 'guideline implementation gap'. [18,20,27-33] Thus, implementation strategies are
needed.
Implementation strategies are very diverse and include – among many others - leaflets that contain written
recommendations [34], written reminders [34,35], education and training [34-37], oral feedback [35,38]
and computer-based implementation strategies. In turn, computer-based implementation strategies point to
a wide range of different uses of information technology for guideline implementation. The most basic
computer-based implementation of clinical practice guidelines is the electronic dissemination of the
narrative guideline document. [4] Other computerised implementation strategies use computer-based
6
feedback and computer-generated reminders. [39,40] In this context, immediate feedback is superior to
delayed feedback [41] and guideline adherence will revert back to baseline when the computer-based
reminders stop. [42] Most computer-based decision support systems provide feedback to their users or
generate reminders. [4] Next to the electronic distribution of the original guideline document, computerbased feedback and computer-generated reminders; evidence-based point-of-care information summary
providers also exist (e.g. www.uptodate.com). [5] However, the best approach for promoting adequate use
of evidence-based guidelines in daily practice is to incorporate clinical practice guidelines into existing
clinical information systems (e.g. the electronic medical record). [4,41,43] In order to build such a
computerised decision support system, the original guideline document must first be transformed into a
computer-executable file. This work will address the foundation of these real-time decision support
systems, namely computer-interpretable guidelines. In short, a computer-interpretable guideline can be
defined as a formalised representation of a textual clinical practice guideline that can be interpreted by
today's computer technology. Different approaches and tools for the development of computerinterpretable guidelines will be will be further elaborated below. [4]
This work has been divided into four distinct but related sections. First, an overview will be provided of
the recent studies that assessed guideline adherence in a disease-specific or a more general sample
population. Furthermore, an overview will be provided of implementation barriers and characteristics of
effective guidelines. The consequences of increased guideline adherence as potential aims of computerised
clinical decision support systems will also be addressed. Second, a generic framework of computerinterpretable guideline development will be described and the use of computer-interpretable formalisms in
IT-based guideline implementation will be addressed. Third, the implementation of this generic
framework was started by extracting specific practice recommendations from an evidence-base on
cardiovascular prevention composed in 2009. [44] Specific practice recommendations were composed for
antiplatelet therapy in primary cardiovascular prevention. These recommendations serve as initial building
blocks for constructing a computer-interpretable guideline. [28] The composed recommendations were
subsequently assessed according to quality of evidence and strength of recommendation using the
GRADE instrument. And last but not least, a Delphi consensus procedure was performed in order to
clarify any vague and ambiguous evidence regarding the topic mentioned above.
7
2. Methodology
2.1. Guideline Adherence
MEDLINE was systematically searched using PubMed with different combinations of the following
keywords: guideline adherence, United States, Europe, Africa, developmental countries, Asia, Japan,
China, Korea, cost, cost-effectiveness, efficiency, barriers, implementation barriers, characteristics of
effective guidelines, patient satisfaction. The search strategy was limited to English articles about humans
that were published in the last 10 years. Some references were handpicked from related citations in
PubMed. The reference list of relevant articles was also checked for extra publications.
As depicted in figure 1, the search strategy returned 10,386 publications and 4 articles were handpicked.
94 articles were selected based on abstract and, subsequently, fully read. Eventually 42 articles of this
search were used in this work.
MEDLINE
Handpicked
10,386 articles found
4 references
94 articles retrieved and read
42 articles selected for use
Figure 1. Search strategy flowchart
2.2. Computerised Clinical Decision Support Systems
A literature search for relevant publications was conducted in electronic databases. PubMed was used to
search MEDLINE. Web of Science was also searched, but did not provide any additional references.
Again, different search tags in different combinations were used: guidelines as topic [MeSH], clinical
decision support (systems), ontology, semantics, semantic markup, semantic annotation, markup, mark up,
mark-up, guideline implementation gap, formalisation. Other
references were obtained using the
8
following search string: ("Information extraction" OR "Formalization" OR "specification") AND
("Guidelines as topic"[Mesh] OR "Decision Support Systems, Clinical"[Mesh]). No limits were used in
this search strategy.
Some articles were found by hand searching the World Wide Web, by checking the reference list of
relevant articles and by browsing related citations in PubMed. The retrieved evidence was processed as a
narrative review.
As depicted in figure 2, the search strategy returned 3,004 publications and 26 articles were handpicked.
145 articles were selected based on abstract and, subsequently, fully read. Eventually 62 articles of this
search were used in this work.
MEDLINE
Handpicked
3,004 references
26 references
145 articles retrieved and read
62 articles selected for use
Figure 2. Search strategy flowchart
2.3. Specific Practice Recommendations and GRADE
Specific practice recommendations, that can be easily transformed into a computer-readable format
without the need for further interpretation, were composed for the use of antiplatelet therapy in primary
prevention. Usually, these recommendations are – as depicted in figure 3 – distilled from clinical practice
guidelines that contain evidence from previously performed meta-analyses, reviews and individual studies.
Thus, specific practice recommendations are a highly condensed and accessible type of evidence that
could make evidence-based medicine easier to implement in daily practice. Furthermore, these
recommendations can be used as the initial building blocks for constructing a computer-interpretable
guideline. [28] The evidence contained in the 2009 RIZIV consensus report on the efficient use of drugs in
cardiovascular prevention was used to compose these specific recommendations. [44] These
recommendations were discussed with two experts.
9
The recommendations were formulated in the form of “if ... then …”. [45-47] Tierney et al. recommend
that guideline developers systematically formulate evidence-based recommendations with an 'if... then...'
structure. Thus, this highly condensed format of clinical evidence can be most effectively translated into
operational guideline knowledge when turning evidence into practice. [46] Georg et al. structured
guideline knowledge for each decision step in this manner. [45]
Afterwards, the quality of evidence and the strength of
recommendation were assessed for the individual
recommendations using the GRADE instrument (Grading
of Recommendations Assessment, Development and
Evaluation). This instrument has been developed and
introduced in 2004, in reaction to the diversity of rating
tools for quality of evidence and the subsequent need for
standardisation.
It
also
introduces
strength
of
recommendation next to the rating of evidence quality.
Both components of the instrument are clearly separated.
The GRADE instrument is designed for systematic
reviewers and guideline authors. These actors may use
the GRADE instrument for rating management strategies
and, subsequently, solving questions or disputes on
alternative management strategies, interventions or
policies. [48,49] The ratings were discussed with two
medical experts.
Figure 3. Distilling specific practice recommendations from previously published evidence (RCT = randomised
controlled trial, CPG = clinical practice guideline, SPR = specific practice recommendation)
In 2011, the GRADE working group started publishing their new, 20-part guideline on the use of their
instrument. Nine parts have already been published. [48,50-57] In 2008, the group already published a 6part introduction to the instrument. [49,58-62] GRADE is a twofold instrument that assesses both the
quality of the evidence behind different management options and their strength of recommendation for
patient care. Thus, letter and number scores are obtained, referring to quality of evidence and strength of
recommendation respectively (see figure 7).
10
After reviewing the evidence base for the defined question, guideline developers and systematic reviewers
initially determine quality of evidence by study design only. Randomised trials produce high initial quality
of evidence and observational studies are low initial quality. [48,51] Subsequently, the initial low or high
quality is altered based on different methodological aspects. Risk of bias [52], publication bias [53],
imprecision [54], inconsistency [55] and indirectness [56] lower the quality one or two steps, depending
on severity. Large effect of intervention, dose response relation, residual confounding that would reduce a
demonstrated effect and residual confounding that would suggest a false effect if none was observed, raise
the quality of evidence. [57] Thus, a final quality of evidence is obtained, ranging from very low (one
plus) to high (four plus). [48,51,58] See figure 4.
Figure 4. Rating the quality of evidence using GRADE [51]
As depicted in figure 5, quality of evidence is linked to confidence in the demonstrated effect being true.
[48,51] Note that the definition of quality of evidence by the GRADE working group has changed in
comparison to 2008. [48,49,51]
11
Figure 5. Consequences of GRADE quality rating [51]
The second part of the instrument is needed because, in some cases, evidence can suggest a true effect but
combined with significant side effects, toxic effects or with an inacceptable cost. Thus, it is weakly
recommended. On the other hand, some interventions may have low or even no evidence, but may be
strongly recommended. For example, epinephrine may be lifesaving in case of an anaphylactic reaction,
but it is not ethical to perform a randomised controlled trial and refuse epinephrine to the control group.
[49]
Assessing strength of recommendation is less well delineated than determining quality of evidence. The
strength of a recommendation reflects to the confidence that the desirable effects outweigh the undesirable
effects (figure 6). But other factors also play a role in assessing the strength of a recommendation. These
include cost, uncertainty of values and preferences and quality of evidence. Guideline developers – and
not systematic reviewers – review all the evidence and evaluate different aspects to determine whether a
recommendation is – or is not – strongly or weakly recommended. Thus, both the direction and the
strength of a recommendation are determined. [48,59]
12
Figure 6. Assessing the strength of recommendation [59]
Figure 7. Grade summarised [59]
Guyatt et al. [49] explain that quality of evidence is actually a continuum and that categorising it
introduces some arbitrariness and therefore limitations to this rating exist. On that note, GRADE users
must realise and bear in mind that it is not a perfect instrument. The GRADE instrument has five
important limitations. First, it is not designed for solving questions or disputes about risk or prognosis but
only for questions about interventions, management and policy. [48] Secondly, it could prove troublesome
to apply GRADE to a set of vague recommendations or a set of good practice recommendations that are
generally not helpful (e.g. “take a comprehensive history” or “perform a detailed clinical examination”)
because these recommendations could lead to counterproductive and contrary practices. Furthermore,
applying GRADE may not even be necessary if the desirable effects clearly outweigh the undesirable
effects or if no one performed a study to confirm a very obvious effect, even when it comes to helpful
recommendations. In these cases, sifting out the indirect evidence may be laborious, time consuming and
generally not useful. [48] Third, reviewers and guideline developers should understand the specific place
of this instrument in the guideline development process as seen in figure 8. The appraisal of the quality of
the available evidence and its subsequent weak or strong recommendation takes place early on in the
development process, shortly after defining the questions to be addressed and establishing a review and/or
13
guideline team. The use of the instrument is integrated in the literature search for the systematic review or
the guideline. [48] Fourth, GRADE has been particularly used to address clinical questions, mostly on
prevention and treatment. Those researchers using GRADE for public health and health system related
questions or questions about diagnostic tests may be confronted with other problems that are less
described in the literature on the GRADE instrument. The GRADE working group acknowledges that the
instrument still needs continuous refinement and improvement. [48] In 2008 and 2009, the GRADE
working group published two papers on the specific challenges that occur in the use of GRADE for rating
diagnostic tests and strategies when using accuracy studies. Cross-sectional and cohort often provide –
when well performed – high quality evidence for test accuracy (sensitivity and specificity). However,
these study designs are low quality evidence for recommendations because test accuracy is a surrogate for
patient-important outcomes. Information on effective treatment availability, less test related adverse
effects or anxiety and improvement of patient wellbeing from prognostic information is needed to
determine whether data on test accuracy can really improve patient-important outcomes. For example, a
very accurate test that diagnoses lung cancer in an early stage in which an effective treatment is possible
will reduce lung cancer mortality. And thus, the diagnostic intervention is strongly recommended. [60,63]
And last but not least, GRADE makes the assessment of quality of evidence and its strength of
recommendation more standardised, but it will never totally eradicate disagreement about alternative
management strategies, interventions or policy among researchers, physicians and policy makers. [48]
Figure 8. GRADE’s place in the guideline development process [48]
14
2.4. Delphi Procedure
Consensus methods (Delphi technique, Nominal Group technique [64-67], RAND/UCLA appropriateness
method [15,64], Consensus Development Conference [64]) have since long been used in health care to
solve disputes and determine strategic directions. The common objective of these methods is to develop a
certain level of agreement on a controversial subject, problem or question. [68-70] These methods can be
used to distil specific practice recommendations from clinical practice guidelines in the case of
controversial, unresolved, vague or ambiguous evidence. The opinion of domain experts (e.g. medical
specialists, general practitioners, nurses, health economists ...) is used to achieve consensus in most of the
previously mentioned consensus methods, but in some cases other stakeholders may also be involved (e.g.
the Office for Medical Applications of Research (OMAR) in the US, one or more Institutes or Centres
(IC's) and the use of an open discussion in the Consensus Development Conference). [64]
A Delphi procedure was performed in order to clarify a number of controversial issues from the 2009
consensus report and to clarify any new evidence between 2009 and 2011. The consensus group consisted
of two cardiologists, one cardiovascular researcher and two general practitioners. Only two rounds were
performed.
The
questionnaire
was
distributed
electronically
using
Survey
Monkey
(http://www.surveymonkey.com/) and the results were anonymously processed and communicated to the
experts after each round.
The experts were asked whether they agreed with the content, the composition (e.g. “Is the
recommendation unilaterally interpretable?”) and the completeness of the recommendation. The experts
needed to score these different aspects on a nine-point Likert scale (see figure 9). The members of the
consensus group were also asked to reflect on the recommendation of they disagreed in any way. Thus, the
second round of the procedure could be prepared as good as possible. Consensus was reached when
opinions coincided without any outliers and without further commentary about the specific
recommendation.
Figure 9. Nine-point Likert Scale
15
The standard Delphi technique (see figure 10) originated in 1948 and utilises a systematic protocol.
Today, numerous variations on the original technique exist. A consensus group can consist of different
parties, including domain experts, general practitioners and in some cases – depending on the subject –
health economists, nurses and/or patient representatives. The panel tries to reach a certain level of
agreement on a predefined subject over a couple of rounds, usually three or four. Each round, each
individual member of the consensus group anonymously fills out the distributed questionnaire. And each
round, the data is collected, processed, summarised and tabulated by an independent facilitator. The data is
then reported back to each participant. A Delphi procedure ends when a certain level of agreement is met
or when no more gain is to be expected. [64,68-73]
Just like the GRADE instrument, the Delphi technique has its strengths and weaknesses. It is a flexible
instrument because the content and number of rounds can be adjusted, yet also systematic because each
round is performed in the same way. The technique does not know any geographical constraints because a
Delphi procedure is commonly performed by (e-)mail. Thus, large international consensus procedure can
relatively easily be performed through the use of a Delphi procedure. Furthermore, the entire technique is
easy to understand and it can be applied to a large variety of questions, topics and problems. But probably
the greatest advantage to the Delphi technique is the anonymity that is associated with the opinion of each
member of the consensus group. [64,68,69,72,73]
On the other hand, the Delphi technique is not suited for achieving consensus when personal contact
among participants is desirable. Furthermore, panel members often get fatigued after a number of rounds
as opposed to the increasing reliability of the procedure with increasing size of the panel and growing
number of rounds. Thus, the number of rounds is best balanced between remaining feasible for the
consensus panel and having sufficient reliability. [64,68-73]
Keeney et al. concluded in their 2001 critique on the Delphi procedure that variations on the standard
procedure are common and therefore methodological difficulties may arise. Each variation should be
carefully considered according to the study protocol. Furthermore, they concluded that the advantages and
disadvantages of the Delphi method do not outweigh each other. [72] In 2006, the same research group
stated that it remains a good technique for achieving consensus on a subject where no consensus was
previously met. Researchers should use the consensus method (either the Delphi technique, the nominal
group technique, RAND/UCLA, consensus development conference or one of their variations) that best
suits their study design. [73]
16
Nair et al. recommended to use the Delphi technique when the investigated variables are clearly evaluable
(e.g. related to an outcome or response criteria set). They also added the following advantages to the ones
listed above: large group possible, disabling supremacy of very eloquent panel members or supremacy of
important persons or dominance by loud people or dominance by people with a very strong view, bias by
the moderator is unlikely, large amount of time to reflect on the question or topic, possibility to change
ideas and the Delphi technique is generally cheap – although large groups and numerous rounds may
prove costly. They listed low external validity, dependence of questionnaire quality, vulnerability with
respect to who is an “expert”, possible bias with panel selection, possible high cost and growing
complexity with increasing number of rounds and with larger consensus groups as disadvantages. [64]
Figure 10. Delphi Technique flowchart
17
3. Results
3.1. Guideline adherence and Quality of Care
Guideline adherence is more and more regarded as a measure of quality of care. [4,74] Furthermore,
compliance with clinical practice guidelines may be used to determine the payment of physicians. [4]
These pay for performance systems in healthcare insurance, in which healthcare providers are rewarded by
meeting predefined targets, are an emerging movement in healthcare insurance. Pay for performance
originally emerged in the United Kingdom and the United States. Today, pay for performance is a globally
emerging movement. [75-78]
In this section we will provide a systematic overview of recent studies that assessed the adherence to
recommended care in a general or disease-specific population sample. These studies can be roughly
categorised into publications that studied a patient population or publications that studied a population of
physicians. Furthermore, a framework of reasons for not adhering to clinical practice guidelines will be
provided and characteristics of effective clinical practice guidelines will be listed. Third, the different
consequences of increasing guideline adherence will be described. The effects of computerised clinical
decision support should aim to comply with these consequences.
3.1.1. Guideline Adherence
3.1.1.1.
Guideline adherence as assessed in a patient population
In 2003 McGlynn et al. [20] studied the quality of health care in 6637 US adults by means of a telephone
interview and one or more health records. They found that only 55% of their research population received
basic care as recommended by guidelines. There is little or no difference between the adherence rate to
recommended preventive, acute and chronic care. There is also little difference between adherence to
recommendations for screening or follow-up (respectively 52.2% and 58.5%). There is however an
important difference between appropriate care given to different medical conditions. For example: 78.7%
of patients were given recommended care for senile cataract versus 10.5% of patients were given
recommended care for alcohol dependence.
18
In 2006 Asch et al. [21] analysed the same population sample and assessed the quality of care in different
American sociodemographic subgroups. They reported an overall adherence to recommended care of
54.9%. Women received more recommended care than men (56.6% versus 52.3%, P<0.001), patients
younger than 31 years received indicated care more frequently than patients older than 64 years (57.5%
versus 52.1%, P<0.001), Caucasian patients (54.1%) received less recommended care than blacks (57.6%)
and Hispanics (57.5%) (P<0.001 for both). Patients with a household income over 50,000 dollars per year
received better quality of health then families with an income less than 15,000 dollars per year (56.6%
versus 53.1%, P<0.001). They concluded that these are but moderate variations in quality of healthcare. It
is the difference between the observed and desired adherence to recommended care that is important here.
In other words, the overall guideline adherence is unsatisfactory.
Mangione-Smith et al. [22] performed a similar study in 1553 outpatiently managed children in the United
States. Averagely, 44.5% of the included children received recommended care. There was a significant
difference in kind of recommended care given: 67.6% received recommended acute care, 53.4% received
indicated chronic medical care and 40.7% received adequate preventive care. As in adults, the percentage
of appropriate medical care differed greatly depending on the different medical conditions: 92% of
recommended care was given for upper respiratory tract infections whereas only 34.5% of recommended
preventive care was offered to adolescents.
McGlynn's research group concluded that both in children and in adults the magnitudes of deficits in
recommended medical care were the same and that guideline compliance is unsatisfactory. They asked for
adequate strategies to reduce these deficits. [20-22]
Shrank et al. [23] wanted to observe the fraction of appropriate pharmacological interventions among US
residents in 3457 adults because of the lack of information on quality of pharmacological care in the
United States despite the growing annual budget for prescribing medication. In general, 61.9% of patients
received appropriate pharmacological care. Different scores were observed for different aspects of
pharmacological care: 46.2% received appropriate education and documentation at a pharmacological
intervention, 54.7% received adequate follow-up of the prescribed medication and there was a huge 62.6%
use of inappropriate drugs.
A recent French-Italian study (Perrier et al. [79]) assessed the cost-effectiveness of sarcoma management
in 219 patients older than 15 years and with a histological diagnosis of sarcoma. Appropriate guideline use
was found in about half of the patients (54%). Appropriate implementation of clinical practice guidelines
19
for sarcoma was superior both in terms of efficacy (2.46 versus 2.16 relapse-free survival years) and in
terms of cost-effectiveness (23,571€ versus 27,313€ mean cost per patient, ICER of compliance with the
clinical practice guideline dominated ICER of non-compliance with the clinical practice guideline). No
use of the gold standard for the assessment of cost-effectiveness, cost per Quality Adjusted Life Year
(QALY), because of the retrospective study design and a lack of information on confounders so they could
not control for these variables were identified as weaknesses of the study.
A Japanese study by Kirigaya et al. [80] reported an overall 23.3% of adherence to glucocorticoid
guidelines in the treatment of 2,368 patients who took glucocorticoid drugs for more than 90 days.
Guideline adherence varied according to the equivalent dose of prednisolone (8.3% for < 5 mg/d versus
30.5% for ≥ 5mg/d). Guideline adherence was also lower in smaller medical institutes and a higher
adherence to guideline recommendations was observed with internal medicine specialists versus surgery
specialities. Younger, male physicians and lower glucocorticoid doses were correlated with lower
guideline adherence.
In Korea, Oh et al. [81] compared the outcomes of 4994 diabetic patients aged 18 or older to physician’s
adherence to clinical practice guidelines. They found an overall adherence rate of 53.5%. Adherence to
recommendations on patient follow-up were low for eye examination at least once a year (32.8%), renal
function (serum creatinine and urine albumin-to-creatinine ratio (UACR) at least once a year, 33.5%,
serum creatinine was tested more (81.2% at least once a year) than UACR (35.1%) and lipid profiles at
least once a year (45.9%). Testing rate for blood pressure at least once a year and HbA1c at least once a
year were high (93.9% and 84.9% respectively). They concluded that higher guideline adherence is
associated with better prognosis because data on HbA1c and UACR being available are associated with a
better outcome (less mortality and less end-stage renal disease).
Van der Veen et al. [82] reported that 66% of HIV patients in the Namibian private sector received
recommended antiretroviral treatment in 2003. In 2008 91% of the private patients were treated with a
recommended antiretroviral treatment (non-nucleoside reverse transcriptase inhibitor or protease inhibitor
added to two different nucleotide reverse transcriptase inhibitors). Non-recommended and ineffective
treatment protocols lowered from 23% to 2% between 2003 en 2008. The latter is an excellent
development in the management of HIV in Africa, because non-recommended or ineffective treatment
protocols give a fast and significant rise in resistance to treatment. The 25% rise in adherence to
recommended care can be explained by the introduction of low-cost insurances. Most people only opted
for HIV or AIDS insurance. These newly introduced insurances included some conditions for medical
20
doctors: all new patients with an indication for antiretroviral therapy must receive first-line recommended
antiretroviral treatment, frequent laboratory checks and consultation of an HIV-physician is needed for
second-line therapy. Another explanation for the significant rise in adherence is that the HIV Clinicians
Society organised a lot of conferences, meetings, seminars and discussions in order to promote
recommended care.
Cassimjee et al. [83] studied the adherence to standard treatment guidelines (STGs) of the Essential Drugs
Programme (EDP) in a province in South-Africa. They reported an adherence to hypertension guidelines
of 22.05% in 100 prescriptions for patients with hypertension.
3.1.1.2.
Guideline adherence as assessed in a population of practitioners
Lugtenberg et al. [84] found a mean adherence to 16 key recommendations extracted from four different
NHG guidelines (red eye, cerebrovascular accident, urinary tract infections, thyroid disorders) of 77%
among Dutch general practitioners (GPs). The highest rates of guideline adherence were seen in
recommendations on patient referral (94%). The lowest adherence rates were observed in
recommendations on patient rehabilitation and patient education (57%). Intermediate adherence rates were
found with diagnostic and therapeutic recommendations (respectively 78% and 77%). Adherence levels
varied greatly between the different key recommendations.
Another Dutch study, conducted by Albers-Heitner et al. [85], assessed the level of adherence of 264
general practitioners to a national guideline on urinary incontinence. They found that most general
practitioners adhered to recommendations for elaboration of urinary incontinence, with the exclusion of
the use of a bladder diary (only 35%). Adherence to therapeutic recommendations was only low in severe
and complicated cases of urinary incontinence. Adherence to recommended therapy for mild to moderate
urinary incontinence was high. ¼ general practitioners replied that guideline implementation during
practice was difficult. They identified lack of time, lack of staff, lack of diagnostic tools, lack of ability to
this kind of care in a proper way and lack of motivation in patients as guideline implementation barriers.
Rietveld et al. [86] concluded that in most episodes of infectious conjunctivitis general practitioners did
not follow recommended care as described in the red eye guideline of the Dutch College of General
Practitioners. 21% of the medical encounters for infectious conjunctivitis were labelled as bacterial
conjunctivitis, but for more than 2/3 of the general practitioners prescribed antibiotic ointments.
21
A Japanese study by Tsumura et al. [87] studied the relation between adherence of physicians to one
specific practical recommendation (association of gastroprotective medication in patients using NSAIDS
with a high risk on gastric mucosal lesions (≥ 65 years, history of peptic ulcers, use of NSAIDS in
combination with corticosteroids or anticoagulant medication, high doses of NSAIDS)) and the occurrence
of gastric mucosal lesions and gastric ulcers in 254 continuous or on-demand non-steroidal antiinflammatory drugs (NSADS) users. Adherence to the recommendation differed between regular and ondemand NSAID users (31.7% versus 24.6%). A higher occurrence of gastric mucosal lesions was
significantly associated with non-adherence to the recommendation in both NSAID user subgroups (both
P=0.01). Gastric ulcers also occurred more frequently in continuous NSAID users in the non-adherence
group then in the adherence group (29.6% versus 5%, P<0.01).
The goal of the study conducted by Suzuki et al. [88] from October 2002 to October 2004 and published
in 2010, was to observe a possible change in physician compliance, efficacy and cost after the introduction
of the Japanese guidelines on prevention and management of Asthma in 2003 (JGL2003). They reported
an overall guideline adherence of 89.5% before the introduction of JGL2003 and 90.3% after the
introduction of the new guidelines. Suzuki et al. report higher adherences rates than all of the above
because their study was performed at Showa University Hospital, Japan. They reported a significant
decrease of severe asthma complaints after introduction of the 2003 guidelines and a significant difference
in the occurrence of severe asthma symptoms between the non-adherent and adherent group (more severe
asthma symptoms in the non-adherent group, P<0.0001). The number of patients on oral corticosteroids,
long-acting β2-agonists both orally and inhaled, long-acting β2-agonist patches, short-acting β2-agonists,
theophylline and leukotriene receptor antagonists decreased after the introduction of JGL2003, except for
inhaled corticosteroids (ICS). The change in prescribed medication can be explained because of the more
prominent place of ICS as the first-line treatment of the inflammatory component of asthma in the new
guidelines. The total cost of anti-asthmatic drugs per year decreased significantly (P=0.006).
A Korean study by Sun et al. [89] reported an adherence to asthma guidelines of 51.3% for intermittent
asthma, 68.5% for mild persistent asthma, 56.9% for moderate persistent asthma and 85.7% for severe
persistent asthma in 81 paediatricians. The adherence rate for severe persistent asthma is probably
overrated because any answer combination with high dose of ICS was considered correct. A very low
adherence to diagnostic recommendations was observed (peak flow and spirometry, respectively 21.5%
and 10.3%). Mostly external implementation barriers were identified (lack of time, lack of equipment,
lack of staff) and barriers varied according to different recommended care.
22
A German study by Karbach et al. [90] assessed the knowledge of the German guidelines on
cardiovascular prevention in primary care physicians. Out of 2500 German primary care physicians to
whom the survey was sent, 1152 returned the questionnaire. 40% of primary care physicians showed an
adequate knowledge of the guidelines. Guideline knowledge varied according to the topic (e.g. guideline
compliance was higher for questions about chronic coronary heart disease than for questions about arterial
hypertension).
3.1.1.3.
Family Physicians vs. Specialists
Salinas et al. [91] described a difference in familiarity with COPD and hypertension guidelines between
family physicians and internal medicine specialists. Internists were significantly (P<0.05) more familiar
with COPD guidelines than family physicians. Guideline familiarity varied depending on the guideline.
28.7% of family physicians versus 36.5% of internists felt very familiar (8 to 10 on 10-point rating scale)
with the Global Initiative for Chronic Obstructive Lung Disease (GOLD) guidelines, 13.3% versus 26.2%
with the American Thoracic Society/European Respiratory Society (ATS/ERS) guidelines and 17.3 versus
39.4% with the American College of Physicians (ACP) guidelines. Both groups felt more or less (77.5%
versus 74.3%) equally familiar with the Seventh Report of the Joint National Committee on Prevention,
Detection, Evaluation, and Treatment of High Blood Pressure (JNC7) hypertension guidelines. The
authors noted that a higher familiarity with COPD guidelines is a logical consequence in physicians who
are more frequently involved in care for COPD patients. Salinas et al. also noted that practitioners who
were engaged in practice for 15 years or less were more likely to be familiar with ATS/ERS guidelines
and not the GOLD guidelines (mean years in practice = 16.1). It is also noteworthy that, for both groups,
familiarity with the different COPD guidelines is a lot lower than familiarity with the JNC7 hypertension
guidelines. See figure 11.
Gerber et al. [92] analysed 226 surveys from physicians working in both hospital and ambulatory settings
(including general practitioners, internists and cardiologists). Adherence to guidelines on oral
anticoagulation therapy with vitamin K antagonists (OAC) was assessed. Physicians working in an
ambulatory care setting reported an overall lower use of guidelines than clinicians working in a hospital
setting (47.6% versus 12.2%). 97.3% of all physicians made the right decision and followed the
recommendation on stroke prophylaxis using OAC. Correct initiation of anticoagulant therapy however
was significantly lower (overall 60.6%) and physicians working in a hospital setting showed a clearly
higher initiation of anticoagulant therapy as described in the guidelines (79.7% versus 51%). Correct
guideline implementation was not associated with medical specialty, but was significantly associated with
working in a hospital environment (OR 2.8, P = 0.023, controlled for confounders (experience, specialist
23
education)). Both general physicians and specialists working in an ambulatory setting reported a lower
adequate use of clinical practice guidelines.
Figure 11. Difference in familiarity with COPD (GOLD, ATS/ERS, ACP) and hypertension (JNC7)
guidelines in family and internal medicine practitioners (P<0.05). Note the difference between familiarity
with hypertension and COPD guidelines: both family physicians and internists are significantly less
familiar with COPD guidelines than with hypertension guidelines. [91]
Yeh et al. [93] assessed the adherence to asthma treatment guidelines from 526 questionnaires returned by
asthma specialists and general practitioners. They found that 90.4% of asthma specialists as opposed to
63.2% would provide care as recommended by the clinical practice guidelines. Asthma specialists were
more adherent to every aspect of recommended care. These aspects included general asthma knowledge,
instructing a good inhalation technique, using peak flow meters, writing down and giving an action plan.
Apart from the previously described reasons for explaining the difference in guideline compliance
between medical specialists and general practitioners – frequency of involvement in the same kind of care
and working environment, we would like to add that medical specialists only need to master a limited
number of clinical practice guidelines whilst general practitioners must take into account numerous
evidence-based guidelines. Thus, general practitioners actually still do well compared to medical
specialists.
24
3.1.1.4.
Conclusion
It may be concluded that – at a global level – guideline adherence is rather low, both in adults and in
children. There are little but significant differences between sociodemographic subgroups, but they are
overshadowed by guideline compliance being generally unsatisfactory. [21] Guideline adherence varies
greatly between different medical conditions [20,22], different recommendations [84], different clinical
practice guidelines [84] and different geographic locations. There are significant differences between
adherence to recommendations on referral, education and rehabilitation, diagnosis and treatment. [84]
Significant differences were also observed between recommendations on acute, chronic and preventive
care in children. [22] General practitioners and medical specialists also have different adherence rates to
clinical practice guidelines (higher guideline adherence among internal medicine specialists versus general
practitioners [91-93] and versus surgeons [80]). Furthermore, adherence to clinical practice guidelines is
also higher in centralised care than in ambulatory care. [92] And lower adherence to clinical practice
guidelines is associated with smaller medical institutes. [80] Higher adherence to clinical practice
guidelines is associated with a higher quality of care (fewer symptoms, less severe symptoms and less
mortality with higher adherence to recommended care). [4,88]
These findings confirm the existence of a guideline implementation gap. Dissemination of clinical practice
guidelines is clearly not sufficient. Implementation strategies are needed to support the adequate use of
clinical practice guidelines in daily practice.
There are enough studies that assessed guideline adherence in specific medical conditions. However, there
is a shortage on studies that analysed adherence to different guidelines in a fairly general population, such
as the studies performed by McGlynn’s research group in the previous decade. [20-24]
3.1.2. Identifying Guideline Implementation Barriers
In 1999, Cabana et al. [18] provided a framework – depicted in figure 12 – for improving adherence to
clinical guidelines in which they described different barriers to guideline adherence. They reviewed and
pooled every English article that described such implementation barriers. They stated that barriers can be
divided into different categories according to the sequence of behavioral change (knowledge ➞ attitudes
➞ behavior). Internal barriers can be addressed to the physician’s knowledge of or his attitudes towards
clinical guidelines (e.g. lack of knowledge of the guidelines’ details or lack of agreement with certain
25
recommendations). External barriers go beyond the physician (e.g. practical constraints, lack of time …).
[94] Barriers to guideline adherence which act on the knowledge level include lack of familiarity and lack
of awareness (volume of information, time needed to stay informed, and guideline accessibility). Barriers
which act on the attitudes level include lack of agreement with specific guidelines (interpretation of
evidence, applicability to patient, not-cost beneficial, lack of confidence in guideline developer), lack of
agreement with guidelines in general (“too cookbook”, too rigid to apply, biased synthesis, challenge to
autonomy, not practical), lack of outcome expectancy, lack of self-efficacy, lack of motivation and inertia
of previous practice. Barriers to guideline adherence which act on the behavioral level include external
barriers which, in turn, can be divided into external barriers due to patient factors (inability to reconcile,
patient preferences with guideline recommendations), due to guideline factors (guideline characteristics,
presence of contradictory guidelines) and due to environmental factors (lack of time, lack of resources,
organisational constraints, lack of reimbursement, perceived increase in malpractice liability). External
barriers may also contribute to the development of other barriers. For example, lack of time may prevent
the physician from retrieving the recommended care from the clinical practice guidelines during patient
contact and may also prevent him from staying informed and looking the information up afterwards. Thus,
the behavioral cycle is completed.
Figure 12. Guideline implementation barriers [18,94]
In 2011, Lugtenberg et al. [84] used an online survey to assess guideline implementation barriers among
703 general practitioners in the Netherlands. Their results have been tabulated in table 1. The answers to
the surveys were compared to important recommendations in the national guidelines (NHG-standards).
They concluded that most general practitioners (89%) perceive guidelines as an important tool to improve
patient care and that implementation barriers varied greatly depending on the recommendation provided in
the guidelines. Implementation barriers include knowledge related (unawareness, unfamiliarity) and
attitude related barriers (lack of agreement, self-efficacy, outcome expectancy and motivation) and
26
external barriers (patient-related such as preferences, guideline-related and barriers related to work
environment such as lack of time, organisational constraints, lack of reimbursement). Mean adherence to
the selected guidelines was 77%. Recommendations on referral showed the highest adherence rate,
recommendations on education and rehabilitation the lowest and recommendations on diagnosis and
treatment had adherence rates in between the two previous categories. About 30% of general practitioners
perceived patient ability and behavior as a barrier for adequate guideline implementation, 25%
experienced a lack of applicability of the recommendations to the patient, 23% of general practitioners did
not follow the recommendations contained in the guidelines because of specific patient preferences and a
little more than 22% did not apply guideline recommendations because they perceived the
recommendations having a lack of applicability in general.
Table 1. Mean percentage of general practioner's perceivance of different barriers, a cross-sectional
study in the Netherlands [84]
Perez et al. [95] surveyed 154 practitioners from two general medicine facilities in New York City. The
practitioners were asked to clarify which barriers kept them from adhering to 7 recommendations from the
GOLD guidelines. They found that low familiarity with the specified recommendations, low self-efficacy
(practitioner thinks that he or she cannot perform the recommended care) and lack of time as
environmental factor were associated with low adherence to COPD guidelines.
27
Salinas et al. [90] studied a mixed population of 309 family physicians and 191 internal medicine
specialists in the US. They also performed a survey on different COPD guidelines, more specifically on
the recommended use of spirometry (spirometry needed to confirm suspected COPD diagnosis and needed
to assess COPD stage) and long-acting bronchodilator (adding an LABD starting from stage II COPD).
They reported a very low overall adherence to both guidelines (respectively 23.6% and 25.8%). They
found that feeling confident in interpreting the airflow data, being able to incorporate the test in their
patient schedule as an organisational element and expected outcome if the recommendation is followed
was positively associated with adherence to spirometry guidelines. LABD recommendations were
positively associated with agreement with guidelines and confidence in positive pharmacological effect.
But the difference between family physician and internist familiarity with guidelines is a more interesting
finding in this study, as described above.
Van Steenkiste et al. [96] interviewed 15 Dutch speaking general practitioners after they analysed two
recorded patient encounters on cardiovascular prevention. They concluded that general practitioners
perceived barriers not associated to themselves (influence of mass media and marketing strategies of the
pharmaceutical industry (e.g. Becel Pro.activ and lowering cholesterol commercials in Belgium),
prescribing medication in order to prevent future insurance claims, lack of trained staff, no integration
with the electronic health record) most notable in the correct implementation of a cardiovascular risk table
(not SCORE and not Framingham) in daily practice.
Steinman et al. [97] describe five different categories of reasons and their interrelation for not prescribing
angiotensin converting enzyme inhibitors or β-blockers in the treatment of heart failure. These include
therapeutic side effects and the presence of comorbid conditions, nonadherence to drug prescription and
monitoring recommendations (inadequate drug use and drug underuse, inadequate drug monitoring),
patient preferences and beliefs, inadequate transition of care and comanagement (e.g. not starting
recommended medication because of insecurity of adequate follow-up after hospital discharge) and
prioritisation and patient benefits (e.g. first address to more urgent matters before prescribing new
medication).
Carlsen et al. [98] assessed the attitudes of Norwegian medical doctors (240 general practitioners and 725
other medical doctors) to practice guidelines and compared reasons for nonadherence between general
practitioners and medical doctors in other fields of expertise. Most of their research population had a good
understanding of the guidelines for their area of expertise (97.2% and 97.8%) and most doctors followed
the guidelines in general (97.6% and 98.8%). However less physicians (61.8% and 71.3%) had effectively
28
integrated the guidelines into their practice, with a significant difference between general practitioners and
other doctors (P=0.005). Most doctors had confidence in guidelines, except for guidelines originating from
the pharmaceutical industry. A lot of physicians (67.3% and 41.1%) perceived an unclear distinction
between essential information and additional information in guidelines, with a significant difference
between general practitioners and other doctors (P<0.001). About half (57.7% and 41.2%) of the
physicians find guidelines difficult to access. They reported that environmental factors (lack of time,
practical issues) were of significantly (P<0.001) more importance to general practitioners in adhering to
guidelines. Both general practitioners and other medical doctors find it important to adjust care to the
individual patient by using common clinical sense. Economic considerations are also an important reason
to deviate from guideline recommended care. Furthermore, sceptical considerations about the evidence
were reasons for nonadherence to evidence-based practice. The question now rises if different
implementation strategies should be developed for general practitioners and specialists. The results of
Carlsen et al. are tabulated in table 2.
Haagen et al. [99] performed a cross-sectional survey in 860 gynaecologists, residents and fertility
specialists in the Netherlands. They found that the clinicians’' knowledge of a subfertility guideline was
generally adequate, only 12% of unfamiliarity was reported for each of the 31 key recommendations.
Physicians’ lack of self-efficacy regarding communication with the patient and poor outcome expectancy
were listed as attitude-related barriers. External barriers were usually related to patient specific factors.
Guideline adherence varies greatly between different recommendations for clinical practice as described
above. On that note, Hofer et al. [100] explained that – in managing patients with type II diabetes mellitus
– there is a lot to do at the point of care. Thus, they specifically investigated why practitioners (289
general practitioners and 213 endocrinologists in the state of Michigan) adhere to certain
recommendations and why physicians do not stick to other recommendations at the point of care. After
analysing the surveys, they found that both general practitioners and medical specialists adhered more to
high-priority recommendations for patients with uncomplicated diabetes mellitus type II (e.g. HbA1c,
blood pressure and lipids under control). Most practitioners also identified several low-impact
interventions as important because they were used as performance measures. On the other hand,
interventions that had recently proven to be effective (e.g. more strict control of blood pressure and lipids)
were identified as less important by both general practitioners and endocrinologists.
29
Table 2. Different attitudes to guidelines and different reasons for nonadherence to recommended care
between general practitioners and other medical doctors: a comparison. [98]
Assessing barriers that may prevent practitioners from adhering to clinical practice guidelines is an
important first step in improving adherence to clinical guidelines and therefore – as previously described –
improving quality of care [101,102].
In conclusion, implementation barriers can be divided into different categories according to where they
intervene in the behavioral sequence. Thus, implementation barriers can be categorised in knowledgerelated, attitude-related and behavior-related barriers. As depicted in figure 12, the presence of certain
30
barriers may lead to the development of other barriers that may further counteract adequate
implementation of clinical practice guidelines in daily practice. External barriers account for most of the
implementation barriers (e.g. lack of time, patient preferences, hard to organise into workflow …) and are
most frequently listed as the reason why physicians do not adhere to recommended care in daily practice.
Physicians’ attitudes towards guidelines (e.g. lack of agreement, lack of self-efficacy, lack of outcome
expectancy …) are also important reasons for nonadherence. Other barriers – not included in Cabana’s
framework [18] – include influence of mass media, marketing strategies, avoidance of future insurance
claims, no integration with the electronic health record, prioritisation, transition of care, comanagement,
adverse effects of therapy and comorbidities. Different medical doctors mostly report the same reasons for
not adhering to guidelines, but general practitioners report more practical constraints and more time
issues. Several of these barriers can be diminished by using computerised clinical decision support
systems in daily practice. For example, computer-based implementation strategies often contain additional
information for integration in clinical workflow (see below) that make it easier to implement
recommended care in daily practice.
3.1.3. Characteristics of Effective Clinical Practice Guidelines
Grol et al. [103] described different aspects of guidelines that influence their use in clinical practice after
analysing 12,880 decisions by 61 general practitioners. Overall, the physicians followed the
recommendations contained in various guidelines in 61% of the clinical decisions. They found that, if the
recommendations were based on scientific evidence, they were used more in clinical practice. Concrete
and specific recommendations make practice guidelines more effective. Furthermore, recommendations
were used more if not controversial. Effective guideline recommendations change existing work routines
to a lesser extent. Other factors with less influence on guideline use are also described in table 3. All
findings were significant (P≤0.001). 38 recommendations had at least one positive attribute and 12
recommendation at least one negative attribute (out of 47 recommendations).
Grol et al. explained however that developing effective guidelines is just one step in making a guideline
attractive for use in clinical practice. Other important steps include guideline distribution and
implementation strategies.
31
Table 3. Influence of different attributes on recommendation compliance, an observational study in 61
general practitioners. [103]
Burgers et al. [104] found that effective recommendations were less complex, require less new skills, were
more in line with existing norms and were more evidence-based. Other important attributes, specifically
for diagnostic recommendations, were negative reactions of patients and convenience of use. They
concluded that both theoretical (evidence) and practical aspects should be taken into consideration when
formulating new recommendations. When practical aspects cannot be met (e.g. need of new skills or
knowledge for correct execution of the recommendation), implementation strategies should be developed
to address these issues.
In general, recommendations for evidence-based practice should be – as far as possible – easy to interpret
and easy to use. Guideline developers and implementers should be increasingly aware of these influencing
factors when developing or implementing evidence-based guidelines for clinical practice.
3.1.4. Consequences of Increased Guideline Adherence
3.1.4.1.
Increased Guideline Adherence and Cost-effectiveness
As described above, Grimshaw et al. [13] concluded that guidelines have the potential to allocate limited
economic resources to effective interventions and Perrier et al. [79] demonstrated that adherence to
32
guidelines for the management of sarcoma reduced the mean cost per patient (23,571€ versus 27,313€)
and that being compliant with the guidelines was more cost-effective (ICER, no QALY). Suzuki et al. [88]
also reported that cost was significantly lower (P=0.006) when adherence to recommended care was
higher.
Flamm et al. [105] tested the adherence to a web-based guideline for the management of preoperative
patients in 1363 patients in a secondary line hospital in Austria. They reported that 81.7% of all
preoperative tests were not compliant with the guideline. 226 out of 5879 preoperative tests were
duplicates. Only 52 indicated tests were not performed. They concluded that, if the web-based
preoperative guideline was followed exactly, 26,287€ per 1000 patients would be saved. Elimination of
test duplicates would save 1,076€ per 1000 patients.
Wilke et al. [106] studied clinical improvement, survival, days hospitalised and different costs (total, ICU,
hospital wards and drug costs) in relation to guideline adherence for initial intravenous antibiotic therapy
(IIAT) in 221 patients with hospital-acquired (HAP) or ventilator-acquired pneumonia (VAP). They found
that about half of the patients received adequate IIAT (107 out of 221). Adequate IIAT was associated
with a significant better clinical improvement in comparison with inadequate IIAT (P<0.001). Survival
differed between adequately and inadequately treated subgroups (86% versus 74% respectively, P=0.021).
Adequate IIAT was significantly associated with a shorter hospital stay (P=0.022). Guideline-adherent
IIAT was cheaper for all economic variables (total cost: 28,033€ versus 36,139€ with P=0.006, ICUrelated costs: 13,308€ versus 18,666€ with P=0.003, drug costs: 4,069€ versus 4,833€ and not significant).
Hospital ward costs were lower in the nonadherent group (3,062 versus 2,918 and not significant). They
concluded that adhering to the HAP and VAP guidelines in German ICU’s could save 2,042 lives and
125,819,000€ in the German health system per year.
Orrick et al. [107] assessed the economic implications of (in)adequate treatment of community-acquired
pneumonia according to the Infectious Diseases Society of America (IDSA) guidelines in 99 patients aged
18 or older. 75.8% of the patients received care compliant to the IDSA guideline. Patients receiving
recommended care had a lower hospital cost (mean 3,009$ ± 2,682$ versus 4,992$ ± 3,686, P=0.021) and
lower cost of antibacterials (mean 117$ ± 79 versus 301$ ± 409, P=0.038).
Ferrando et al. [108] studied the effect of correct application of guidelines on preoperative evaluation in
702 patients. They reported an average decrease in the number of preoperative laboratory tests from 20 to
3 per patient. Cost for preoperative assessment per patient decreased from 69€ to 26€ (decrease of 63%).
33
In conclusion, achieving a higher quality of life at a lower, similar or an acceptable increase of cost as
compared to the current standard is becoming increasingly important in a world of scarcity. Thus, costeffectiveness should definitely be taken into account in clinical practice. A first step in providing efficient
healthcare may be the adequate us of clinical practice guidelines in clinical practice, because guideline
compliance decreases the cost of care on different levels (hospitalisation cost, drug cost …).
3.1.4.2.
Guideline Adherence and Patient Satisfaction
Feuerstein et al. [109] studied the relation between guideline adherence, patient outcomes, patient
satisfaction and cost in 15,789 US primary care patients with acute low back pain between 1998 and 2002.
Overall guideline adherence was 42.8%. Patients who received guideline adherent care reported higher
satisfaction scores as assessed by means of an annual health care satisfaction survey (7.22 versus 7.12,
P=0.0054), lower costs (222.40$ versus 712.60$, P<0.0001) and better functionality afterwards (85.1%
versus 78.4%, P<0.0001).
Gross et al. [110] performed a study in 135 Israelian diabetes patients to determine the relation between
guideline adherence and patient satisfaction because some authors do not relate adequate use of guidelines
to improved patient satisfaction (but state that personal and communicative skills of the physician are
related to patient satisfaction) while other authors concluded that recommended care does increase patient
satisfaction. Patients were generally more satisfied when physicians adhered to the guidelines. Patients
who received more recommended explanations were more satisfied with the professional level of the
physician (OR=2.82) and patients who received more recommended treatments were more satisfied with
the physician’s attitude (OR=2.68). Both were significant (P<0.05 and P<0.01 respectively). They
concluded that patients were aware of the change in performance while adhering to the guidelines, thus
translating itself into a higher patient satisfaction. They asked for more general studies and studies in other
medical areas and health systems to determine of their conclusions are applicable in the general or other
populations.
Reker et al. [111] assessed patient satisfaction in relation to guideline adherence (Agency for Healthcare
Research and Quality (AHRQ) stroke guidelines) in a prospective cohort study with 288 newly diagnosed
stroke patients. Adherence to AHRQ guidelines for postacute stroke care was significantly associated with
higher patient satisfaction, even after controlling for functional outcome (respectively P=0.002 and
P=0.004). There was no relation between adhering to AHRQ guidelines for management of acute stroke
and patient satisfaction.
34
In conclusion, patient satisfaction is an important indicator for quality of care – especially in primary care.
Furthermore, patient satisfaction may also lead to confidence in the physician and, subsequently, patient
compliance. [110] Providing care as recommended by clinical practice guidelines increases patient
satisfaction according to the studies mentioned above. However, further elaboration is needed in other
medical areas and in more general populations to assess whether these findings are applicable in the
general population.
3.2. Computerised Clinical Decision Support Systems
As described above, guideline compliance is unsatisfying. Furthermore, increasing guideline adherence
leads to more quality of care, more efficient care and increased patient satisfaction. Thus, implementation
strategies are needed to enhance the use of clinical evidence in daily practice. Different implementation
strategies exist (see above), but there is consensus that integrating clinical practice guidelines into existing
clinical information systems – such as the electronic health record – at the point of care is most effective.
[4] Furthermore, integrating decision support into the clinical workflow – for which information
technology is particularly suitable – provides the strongest increase in guideline adherence. [4,28] These
integrated decision support systems match individual patient attributes to an electronic knowledge base
containing computer-interpretable guidelines. Subsequently, patient-specific recommendations (positive
and negative) are generated by software algorithms (see figure 13). The individual patient attributes are
extracted from the electronic medical record after they are manually entered into the electronic medical
record. Recommendations for decision support can come in the form of alerts, reminders, prescribing
advice, critique on the practitioner's orders, overseen preventive measures and others. [112-114] This
section will provide a comprehensive overview of how computerised clinical decision support systems are
constructed and will contain a comprehensive description of their building blocks, namely computerinterpretable guidelines. Computer-interpretable guidelines are even a necessary prerequisite for building
a computerised clinical decision support system. [115]
35
Figure 13. Extracting and integrating information from the Electronic Health Record (EHR) in Clinical
Decision Support Systems that provide do's and don’ts at the point of care
Going from a clinical practice guideline to a computer-interpretable guideline is not done in one day. The
translation of narrative documents into computer-interpretable guidelines is not straightforward and is
liable to variability because human language involves an important element of human interpretation.
Therefore, different recommendations from the same guidelines could be given for the same patient
because computerised guideline representations may vary according to the individual who constructed
them. [28,116,117] Translating clinical practice guidelines into a computer-interpretable representation is
a process that compromises a number of steps. [28,113] These are depicted in figures 17 and 18. A very
important first step is the acquisition of knowledge to process into a computer-interpretable guideline.
[28,113] This includes the selection and appraisal of clinical practice guidelines and the processing of the
medical information by medical experts. Shiffman et al. [28] recommend to extract the essential parts of
the guideline, specific recommendations, to enhance the implementation process. Medical experts play an
important role in the selection and composition of these recommendations because they can clarify any
incomplete, vague or ambiguous knowledge (e.g. consensus methods) and add missing knowledge.
[28,120] The selection and appraisal of clinical practice guidelines is particularly important, because
qualitative guidelines are a great first step towards – but no guarantee for – a qualitative computerinterpretable guideline. Appraising guidelines is also important because guideline implementation is a
time and resource consuming task. In other words, the knowledge source needs to be worth translating
into a computer-interpretable representation. Clinical practice guidelines can be appraised by the
Appraisal of Guidelines REsearch & Evaluation (AGREE) instrument. [28] The AGREE instrument
evaluates the development of clinical practice guidelines and the quality of reporting. In 2009-2010 the
development of AGREE II was finished. [118] AGREE II compromises 23 items organised into 6 quality
domains. These domains include scope and purpose, stakeholder involvement, rigor of development,
36
clarity of the presentation, applicability and editorial independence. At the end, appraisers need to score
two global items, namely the overall quality of the guideline and the recommendation of the guideline for
use in practice. Every item is scored on a 7-point Likert scale. At least two appraisers and preferably four
appraisers are needed to reliably score clinical practice guidelines. [118,119] Because AGREE II is a
generic instrument it can be applied to clinical practice guidelines for every medical condition, guidelines
that focus on any step in the health care continuum (screening, diagnosis, treatment, …) and guidelines
developed at various levels (local, regional, national, international). The instrument can be used by
guideline developers, implementers and users, by policy and decision makers and by educators. [118,119]
AGREE II has been included in annex 1.
After selecting qualitative clinical practice guidelines for use, the information contained in the guidelines
needs to be processed into a computer-interpretable format. This step is performed by knowledge
engineers and results in a formalised guideline. Formalised guideline documents are written in a specific
guideline representation language that can be operationalised by today’s information technology. Multiple
computer-interpretable guidelines form up a knowledge base which contains a modular representation of
computer-executable rules in the form of “if... then...” [45-47] for various tasks in patient care. [113] The
most difficult task of knowledge engineering is to make the information complete. This means that every
threshold, definition, term, value and decision point needs to be specified to ensure full operability. In
other words, this means that every clinical situation should be covered. [113,121] Substantially less
evidence has been published about guideline specification.
Two different approaches for building a computer-interpretable guideline exist, namely the documentcentric approach and the model-centric approach. [4,45,120,122,123] Most of the methodologies puslished
to date use a model-centric approach. The document-centric approach starts from the original guideline
document whereof information is extracted using mark-up or semantic annotation. [45,120,122,123]
Theretofore the knowledge engineer collaborates with the medical expert who clarifies any incomplete,
vague or ambiguous parts of the guideline. Hereto is referred with the terms human computation and
collaborative encoding. [124,125] It is clear that formalisation of guidelines should be a multidisciplinary
process involving both medical experts and IT specialists fostering each other with knowledge and knowhow. [126] The marked-up or annotated document is an intermediate representation of a clinical practice
guideline that needs further translation into a totally formalised document. Here fore, a number of
guideline representation languages have been developed to structure the guideline information.
[4,123,126-129] These include – among others – Arden Syntax [130], Asbru [131], EON [132], GLIF
[133], ProFORMA [134]. These representation languages all have different aspects and strong points that
37
make them more or less suited for a certain task. [123,129] Describing and comparing these different
guideline representation languages would be beyond the scope of this work. The model-centric approach
initially requires a medical expert to transform the clinical practice guideline into a compact conceptual
model (e.g. a decision tree or a flowchart) that is subsequently converted into a computer-interpretable
format using the representation languages described above. [120,122,123] The main difference between
the two approaches is the link with the original guideline document. Whereas, model-centric
methodologies require a great deal of interpretation at the start and thus result in an indirect relation with
the original document, document-centric approaches maintain a close relationship with the original
guideline through – mostly multiple – intermediate representations (see figure 14). [122] This makes the
two approaches more or less suitable for different clinical areas. When adherence to the original guideline
text is wanted – for example decision support in primary prevention or decision support for cardiovascular
prevention or other broad subjects (e.g. oncology [135]) – then document-centric approaches are more
suitable [45,120], when decision support is wanted in a highly specialised branch of medicine or when
decision support is wanted in acute care (intensive care unit, emergency department) [136] or when
constructing a computer-interpretable guideline for evidence-based online point-of-care information
summary providers based in a clinical question then a model-centric approach is more suitable. The
document-centric approach will be further elaborated below because of its complementarity with the
specific practice recommendations formulated for antiplatelet therapy in primary cardiovascular
prevention. Examples of document-centric approaches include the Guideline Elements Model (GEM)
[45,137,138], ActiveGuidelines [139] and the Hypertext Guideline Markup Language (HGML) approach
[140]. Examples of model-centric approaches include the Asgaard project [141], Prodigy [142], the Arden
Syntax framework [143], the EON system [144], ProFORMA [134], GUIDE [145], GLIF3 [133], SAGE
[146,147]. A detailed description and comparison of these different systems is beyond the scope of this
work, but note that the SAGE project is particularly interesting because it strives towards deploying the
same computer-interpretable guidelines into different clinical environments without having to go through
a tedious customisation stage. Thus, it addresses the interoperability problems in the healthcare IT sector.
[146-148]
However model-centric approaches are more common, document-centered guideline formalisation has its
advantages such as adherence to the literal text [4,45,120], easier verification of the formalised guideline
[120], the possibility of performing a compliance analysis [120], easy provision of explanations [28],
reduced risk of information loss [122], diminishing some of the encoding difficulties into a formalised
guideline representation [4] and easy updating of the computer-interpretable guideline with new evidence
[122,149,150]. The latter is especially interesting, because it takes about two years to construct a new
38
clinical practice guideline without even having it implemented. Updating to a new version of the
computer-interpretable guideline by only formalising the new evidence, may significantly shorten this
time period. [122] Note that also model-centric methodologies for versioning computer-interpretable
guidelines exist. [122,151,152] Figure 14 shows that document-centric approaches utilise the original text
from the clinical practice guideline or from a treatment protocol – which is slightly easier to encode into a
computer-interpretable form [135] – to construct a computer-interpretable guideline through different
intermediate representations. [126] These intermediate representations make the formalisation process
more transparent and make it easier to retrace the steps of the formalisation, because a link is maintained
with the original guideline document. [28,120,122,126,153]
Figure 14. Going from clinical practice guidelines or treatment protocols to a formalised representation
of the guideline. [126]
Different tools exist for a document-centered processing of clinical practice guidelines. [4,122,126] Markup tools – such as the Guideline Elements Model (GEM) Cutter [154], Stepper [120], Document
Exploration and Linking Tool/Addons (DELT/A) formerly known as the Guideline Markup Tool (GMT)
[155] and the web-based Uruz tool [156] – are used to break the original guideline document down into its
most basic parts, to specify them, to label them and to structure them. Therefore, document-centric
approaches have a lower risk of losing clinically relevant information contained in the original text. [4] A
detailed description and comparison of these different tools is beyond the scope of this work. First, the
narrative document is converted into HTML (HyperText Markup Language). Subsequently, the document
is marked-up resulting in an XML (eXtensible Markup Language, e.g. when using GEM) [157,158],
39
XHTML (eXtensible HyperText Markup Language, e.g. when using ActiveGuidelines [139] or the
Hypertext Guideline Markup Language (HGML) methodology [140]) document that does not require
advanced programming skills to interpret. [4] Thus, mark-up can also be performed by trained clinicians
without the need for advanced programming skills [4] or by clinical editors, clinically trained editors that
mark-up the guideline. [159] During the mark-up, different parts of the original guideline text receive
labels that imply what kind of information is contained in the text (e.g. definitions, actions, titles,
conditions …). Furthermore, pieces of text that cannot be or need not be marked up, are removed. [120] A
more advanced form of mark-up is semantic annotation. Apart from breaking the text down in its most
basic parts and tagging the necessary information, semantic annotation also provides a structural tree that
contains the relation between the different tags and subtags. This additional information is called
metadata. The addition of metadata to the formalised guideline document extends the functionality of the
formalised guideline document in comparison to electronic textual guideline documents and ordinary
markup. [28,120,160-162] Furthermore, because semantic annotation uses knowledge elements – which
can be understood by non-IT-experts – to mark up specific parts of the guideline text, semantic annotation
has the capacity to bridge the gap between complex computer-interpretable documents and humanreadable documents. [28,120] This is contrary to plain mark-up, in which only structural information is
provided. (see figure 15).
Thus, in order to summarise, semantic annotation is an advanced form of guideline mark-up with extended
functions, which allows the use of the formalised guideline document outside of workflow integrated
decision-support. The different uses of semantic guideline documents are depicted in figure 16.
Afterwards and for both mark-up and semantic annotation, the structured intermediate representation is
more and more converted into a computer-interpretable format. This includes – for document-centric
approaches – stripping language down into its most basic parts by removing expressions, unnecessary
words and by changing passive tenses into the active form. [28,45,120] Because computers are only stupid
machines, they rely on user input to process information using a binary numeral system (either 0 or 1,
either yes or no). Thus, every piece of information must be specified in order to make the information
contained in the clinical practice guideline unilaterally interpretable. [28] Medical experts can be involved
in the precise specification of guideline knowledge, also called semantic refinement. [28,45,120]
40
Figure 15. The difference between plain and semantic mark-up as displayed in the same narrative
document. Plain mark-up is displayed on the left and only provides structural information, whereas
semantic mark-up – displayed on the left – uses knowledge elements that can be understood by humans.
Figure 16. Semantic guideline documents can be used for different purposes. Semantic documents add
workflow-based decision support, semantic search, reasoning and consistency check to the normal
functionality of electronically published guideline documents, namely on-screen viewing and printing.
[160]
After the construction of a presumably complete computer-interpretable guideline, the knowledge base is
formally and clinically tested and feedback is provided to the knowledge engineer. Testing the knowledge
base is crucial to ensure reliable programming. When everything is in order, the operational knowledge
base is programmed into a computerised clinical decision support system. The programmer uses the
knowledge base as a basis to encode a logical process. He also encodes a graphical working environment
with different other modules around the decision support module. The result of the programming process,
41
a clinical decision support system, depends on the working environment as do the necessary programming
tools. Describing the entire programming process is beyond the scope of this work, but note that the
programming process depends on the chosen guideline representation formalism because the logical
process is encoded from the knowledge base. Once the initial programming is complete, another round of
formal and clinical testing is performed and feedback is provided to the programmer. Integration into
clinical workflow, the decision module itself (unit testing) and the entire clinical system (integration
testing) are tested. When everything is in order, users are trained and the decision support is deployed into
practice. Not that rushing a decision support system to practice is bound to fail. Training to all users
(physicians, nurses and others) and a user support plan are two additional conditions for successful
implementation of computerised decision support. The latter can be obtained by training ‘super users’.
Super users received additional training and can provide support to all other users of the system. Providing
user manuals can also provide additional information to the users, but they cannot assist the user at
bedside. And last, but not least, bedside physicians are in the top position to provide feedback for further
refinement and review when the decision support system is routinely implemented in practice (e.g. new
evidence, a clinical situation that has not been encountered so far, feedback regarding patient safety …).
[113]
Figure 17. Building computerised decision support. [113]
Two examples of document-centric approaches will be described to clarify all of the above. Shiffman et
al. [28] proposed a 4-stage, 12-step document-centric approach to guideline implementation in order to
bridge the guideline implementation gap. This approach is depicted in figure 18. They used the Guideline
Elements Model (GEM) to process the guideline knowledge. The four stages include guideline selection,
42
mark-up, specification of guideline knowledge and workflow integration. Guideline selection includes the
selection of the guideline itself, guideline appraisal and selection of the units of clinical practice, namely
individual recommendations. Guideline mark-up is used to identify knowledge components in two
different types of recommendations, namely imperatives in which directives are identified and
conditionals in which decision variables and actions are identified. Mark-up is also used to identify other
guideline information such as target audience, aim of the guideline, methodologies to rate quality of
evidence and strength of recommendations (e.g. GRADE, see above). Shiffman et al. used the GEM
Cutter for the mark-up process. The third stage, specification of guideline knowledge, compromises
different steps that result in clear computer-understandable statements. These steps include atomisation,
deabstraction, disambiguation, verification of completeness, adding explanations and building executable
statements. Medical experts can and should be involved in deabstraction and disambiguation. Atomisation
involves reducing language to its most basic form (e.g. by converting from the active to the passive tense,
by removing expressions and by removing unnecessary words). Deabstraction is the process of replacing
general or abstract formulations with concrete and operational decision variables or actions (e.g. changing
‘correct use of inhaled medication’ to ‘use of a spacer, slow inhalation and repeat inhalation after 1 to 4
minutes’ [28]). Deabstraction is especially important because there is a tradeoff between abstraction and
variability in guideline use. Since the aim of computerised guideline implementation is to reduce variation
in clinical practice, deabstraction must be performed thoroughly and correctly. After deabstraction,
reatomisation may be needed. Furthermore, vague and ambiguous words - that are often an expression of
limited evidence by guideline authors – will lead to inconsistent interpretation of recommended care.
Therefore, they are disambiguated by specifying the information (e.g.’ frequent use’ to’ ≥ 2 times per
week’ or ‘may reduce’ to ‘will reduce’). The result of disambiguation is a single possible interpretation of
a recommendation. Verification of completeness assesses whether all likely clinical situations are covered
with the set of recommendations extracted from the guideline. Incomplete recommendations will lead to
avoidable practice variation. Explanation provides the possibility to add the logic or reasoning behind
specific recommendations. Ever since 1981, this is considered an important feature of clinical decision
support. [163] Explanations can be easily provided using the original guideline text after it has been
marked-up using the <reason> tag. The last step of the specification stage is to rearrange the atomised,
deabstracted, disambiguated, verified and explained knowledge into executable rules in the form of “if…
then…”. The last stage of this methodology focuses on workflow integration. It has been proven that
clinical decision support systems can only be significantly effective if they can be used at the point of care
or integrated in the physician’s workflow. Integration of these systems in clinical workflow is an
important key to acceptance and success. [164,165] Thus, computer-interpretable guidelines should
contain two different types of information, namely medical information and information on integrating
43
medical knowledge in the clinical workflow. In order to do this, points of origin for each decision variable
and points of insertion in the care flow for each action and directive must be identified. Shiffman et al.
propose a non-universal, descriptive framework of the clinical workflow whereto the decision variables
and actions can be mapped. The elements of the framework include patient registration, clinical history,
physical examination, laboratory, further testing, assessment – in which all the information obtained in the
previous elements is synthesised into a possible diagnosis and plan – and further patient management,
either diagnostic or either therapeutic. Next, actions that encode for appropriate clinical behavior are
categorised. Essaihi et al. defined four distinct action types; namely actions for gathering information such
as patient testing and patient monitoring, actions for interpreting information such as determining a
diagnosis or prognosis, actions for performing a task such as prescribing medication or informing the
patient, and actions for arranging and for organising care such as discharging the patient or referring the
patient. [166] The implementer can also associate beneficial services to certain actions to facilitate clinical
practice. Fourth, interface components are chosen for the user interface. This step must ensure optimal
usability in practice. It goes without saying that users should be involved in this step. And last, but not
least, a requirement specification is provided to the personnel that supervises the clinical information
system. These persons usually have a lot of experience in programming, setting up, editing en supporting
the use of these on-site systems, but have little to no knowledge about the medical domain or the
informatics needed to construct decision support. This requirement specification document then serves as
a launching point for further implementation and use in the local information system. [28]
Svátek et al. [120] proposed a step-by-step semantic mark-up strategy using Stepper. They applied this
methodology to the WHO hypertension guideline. No information was provided on the appraisal of the
guideline, the selection of recommendations, the integration into the clinician’s workflow and further
programming apart from the semantic mark-up. They state that different issues are involved in the
transition of textual guideline documents into an operational format. These include removing parts of the
guideline text that cannot be formalised or that are irrelevant to the task, replacing language expressions
with structures that can be formalised, detaching knowledge elements from the surrounding context,
adding missing knowledge and replacing vague expressions involving measurable parameters with
specific values. Svátek et al. designed a methodology which deals with these aspects. Each of the six
proposed steps result in an XML-based document with its own Document Type Definition (DTD). First,
the narrative guideline document is converted to XHTML using simple web page design and used as an
input format for the mark-up tool. Subsequently, relatively large parts of the guideline text - sentences and
larger - are marked up and pieces of text that cannot be marked up are removed. This is called coarsegrained semantic mark-up and can be performed by persons without advanced medical knowledge.
44
Afterwards, the semantic tree is further refined by specifying subelements. Thus, the flow of narrative text
is disconnected without losing information about the order of the knowledge elements in the text. This is
called fine-grained semantic mark-up. In this step, guideline knowledge is also specified, deabstracted and
disambiguated. Furthermore, missing knowledge is added. Of course domain experts play an important
role in this step. In the fourth step, the original document structure is replaced by a systematic ordering of
the knowledge elements to achieve a modular representation of the knowledge contained in the guideline.
Thus, a universal knowledge base is obtained. Collaboration between the knowledge engineer and the
medical expert may be required. This universal intermediate representation still allows to create different
target representations using different guideline representation languages. In order to translate the universal
knowledge base into a specific computational representation, the structure of the knowledge base needs to
be adapted to ease translation into the target representation. Thus, an export-specific knowledge base is
constructed. Finally, the export-specific knowledge base is exported into an operational representation
using either a specific guideline representation language or a common programming language (e.g. JAVA,
C++, PHP, Perl, Lua …).
Figure 18. A systematic, document-centric approach to guideline implementation by Shiffman et al. [28]
45
3.3. Specific Practice Recommendations
Four specific practice recommendations for antiplatelet therapy in primary cardiovascular prevention,
tabulated in table 4, were composed and assessed according to quality of evidence and strength of
recommendation using the GRADE instrument. The evidence contained in the 2009 consensus document
on cardiovascular prevention was used to perform this task. [44] The section of the report on antiplatelet
therapy in cardiovascular prevention is included in annex 2.
Recommendation
Initial quality
of evidence
Influencing factors
Quality of Strength of recommendation
evidence
GRADE
If a patient presents with ≥ 5% 10 year
High
risk of cardiovascular mortality (SCORE)
then lifelong treatment with aspirin 70150 mg/d is indicated.
None
High
Desirable effects outweigh
undesirable effects, high quality
of evidence, cost-effective [167169] => Strong recommendation
for using the intervention
A-1
If a patient presents with an indication for High
aspirin in primary prevention but with a
documented history of aspirin intolerance
(aspirin allergy (Quincke edema, aspirin
asthma), gastro-intestinal bleeding) then
lifelong treatment with clopidogrel 75
mg/d is indicated.
Inconsistency
Moderate
Desirable effects outweigh
undesirable effects, moderate
quality of evidence, no evidence
for cost-effectiveness of
clopidogrel versus aspirin in
primary prevention => Strong
recommendation for using the
intervention
B-1
If a patient presents with an indication for High
aspirin in primary prevention and
uncontrolled hypertension (BP > 140/90
mmHg) then do not administer aspirin 70150 mg/d.
None
High
Undesirable effects outweigh
A-1
desirable effects, high quality of
evidence => Strong
recommendation for not using the
intervention
If a patient presents with an indication for Low
aspirin in primary prevention, in the
presence of an active internal bleeding
process then do not administer aspirin 70150 mg/d.
None
Low
Undesirable effects outweigh
C-1
desirable effects => Strong
recommendation for not using the
intervention
Table 4. Specific practice recommendations based on the evidence contained in the 2009 RIZIV consensus
report on the efficient use of drugs for cardiovascular prevention. [44]
3.4. Delphi Procedure
As described above, a Delphi procedure was performed to clarify some unresolved issues regarding
antiplatelet therapy in primary prevention (e.g. aspirin in patients with diabetes mellitus). The consensus
group consisted of two cardiologists, one cardiovascular researcher and two general practitioners. Two
rounds were performed. Response rate was 100% for both rounds. The experts were asked to score their
46
agreement on a nine-point Likert scale (see figure 9) regarding different statements about a number of
recommendations (e.g. “Do you agree with the content of the recommendations?"). They were also asked
to reflect on the recommendation to whatever extent they disagreed. Which every expert did.
As provided in table 5, nine questions were presented to the experts in the first round of the procedure.
The experts expressed their disagreement with the use of more recent antiplatelet drugs in primary
prevention. They also did not agree to the use of aspirin and clopidogrel for patients with diabetes mellitus
type II in primary prevention. Thus, a consensus was already reached for five out of nine questions. Based
on the commentary provided by the experts, the remaining questions for which no consensus was reached
were modified and re-presented to the experts in the second round of the Delphi procedure. In the second
round, see table 6, nine questions about three specific recommendations were presented to the experts.
Consensus was reached about primary cardiovascular prevention in patients with diabetes mellitus type II
without other cardiovascular risk factors. The consensus panel remained inconclusive about aspirin in
patients with hyperlipidemia and prioritised lifestyle changes and statins above antiplatelet therapy. The
experts agreed with the composition and completeness of recommending aspirin in case of high blood
pressure as a single elevated risk factor, but not with its content. Therefore, consensus was not reached for
this recommendation.
47
Question to experts
Likert scale score
Strongly
Disagree
If a IIf a patient presents with type II diabetes mellitus without
organ damage and in the absence of other cardiovascular
risk factors then:
•
no treatment for cardiovascular prevention is
indicated.
•
lifelong treatment with aspirin 70-150 mg/d is
indicated.
•
lifelong treatment with clopidogrel 75 mg/d is
indicated.
•
lifelong treatment with clopidogrel 75 mg/d is
indicated when a contra-indication for aspirin
treatment is present (documented allergic reaction or
a history of gastro-intestinal bleeding).
•
lifelong treatment with prasugrel 5-10 mg/d is
indicated.
•
lifelong treatment with ticagrelor 2x 90 mg/d (after
one-time 180 mg loading dose) is indicated.
If a patient presents with a single highly elevated
cardiovascular risk factor (total cholesterol > 310 mg/dl
(8 mmol/l) or LDL cholesterol > 230 mg/dl (6 mmol/l) or
blood pressure > 160/100) then lifelong treatment with
aspirin 70-150 mg/d is indicated.
•
Do you agree with the content of the
recommendation?
•
Do you agree with the composition of the
recommendation?
•
Do you think the recommendation is complete?
Disagree
1
Moderately
Disagree
Mildly
Disagree
Undecided
1
Commentary
Mildly
Agree
Moderately
Agree
Agree
1
3
3
2
2
1
2
1
1
Strongly
Agree
Information on safety with the more
recent products is too limited to
make them widely used in
prevention.
Patients with diabetes and a low risk
profile may treated by using nonpharmacological means.
Clopidogrel, even if generic, is too
expensive to be placed in primary
prevention. There is contradictory
evidence for the use of aspirin in
patients with diabetes. Contraindications to aspirin are rare.
With or without organ damage
should be specified in the
recommendation.
1
1
1
1
1
1
1
1
1
This is actually a double question.
The question is not straightforward.
Actually, two questions should be
asked: one for hyperlipidemia and
one for blood pressure. There was
no evidence found in the literature
regarding hyperlipidemia and
aspirin. There is one Cochrane
review regarding and aspirin
showing that aspirin increases risk
of bleeding if blood pressure
remains uncontrolled. So control of
blood pressure should be specified
in the recommendation
There should also be clarified
whether the recommendation applies
to "primary" prevention or
"secondary" prevention.
Table 5. Results from the first round of the Delphi procedure. The text highlighted in grey indicates the recommendations that were excluded from the
second round because consensus was reached.
48
Question to experts
Likert scale score
Strongly
Disagree
Disagree
If a If a patient presents with type II diabetes mellitus
without organ damage and without other elevated
cardiovascular risk factors then no antiplatelet treatment
for cardiovascular prevention is indicated.
•
Do you agree with the content of the
recommendation?
•
Do you agree with the composition of the
recommendation?
•
Do you think the recommendation is complete?
Mildly
Disagree
Undecided
Mildly
Agree
1
1
1
If a patient presents with a single elevated risk factor,
such as total cholesterol > 320 mg/dl (8 mmol/l) or LDL
cholesterol > 240 mg/dl (6 mmol/l), without organ
damage then lifelong treatment with aspirin 70-150 mg/d
is indicated.
•
Do you agree with the content of the
recommendation?
•
Do you agree with the composition of the
recommendation?
•
Do you think the recommendation is complete?
If a patient presents with blood pressure > 180/110 mmHg
without organ damage and without other elevated
cardiovascular risk factors then lifelong treatment with
aspirin 70-150 mg/d is indicated if hypertension is under
control.
•
Do you agree with the content of the
recommendation?
•
Do you agree with the composition of the
recommendation?
•
Do you think the recommendation is complete?
Moderately
Disagree
Commentary
Moderately
Agree
Agree
Strongly
Agree
1
1
2
1
2
1
1
3
The bleeding risk and the intention to
follow (up for organ damage) should
also be mentioned.
For now, it is ok that there is no
specific anti-platelet therapy
specified, but in the future, other
studies might prove something that
requires further specification of this
recommendation.
Await JPAD-like trials. The Japanese
Primary Prevention of
Atherosclerosis with Aspirin for
Diabetes (JPAD) is a trial that was
undertaken to examine the efficacy
of low-dose aspirin in the prevention
of atherosclerotic events (fatal acute
coronary syndromes and fatal
strokes) in patients with diabetes
mellitus type II. The original analysis
showed no reduction of fatal
cardiovascular events. Several
subanalyses have been performed,
showing a benefit of low-dose
aspirin in specific subsamples. [170173]
In the case of hyperlipidemia, life
style changes and giving statins is
more important.
1
1
3
1
1
2
1
1
1
1
1
2
1
2
1
1
1
2
1
1
1
2
1
Table 6. Results from the second round of the Delphi procedure. Consensus was reached for the questions highlighted in grey.
There is no real hard evidence
available about this subject, perhaps
the new guidelines on cardiovascular
prevention will provide more clarity
(ESC, may 2012).
The value of aspirin in addition to the
adequate treatment of the
hypertension without other
cardiovascular risk factors is not
clear.
The HOT trial which showed a
positive effect received criticism.
[174]
49
In order to summarise, both the RIZIV report and the Delphi procedure for antiplatelet therapy in
primary prevention led to the extraction of five positive recommendations:
•
If a patient presents with ≥ 5% 10 year risk of cardiovascular mortality (SCORE) then lifelong treatment
with aspirin 70-150 mg/d is indicated.
•
If a patient presents with an indication for aspirin in primary prevention but with a documented history
of aspirin intolerance (aspirin allergy (Quincke edema, aspirin asthma), gastro-intestinal bleeding) then
lifelong treatment with clopidogrel 75 mg/d is indicated.
•
If a patient presents with an indication for aspirin in primary prevention and uncontrolled hypertension
(BP > 140/90 mmHg) then do not administer aspirin 70-150 mg/d.
•
If a patient presents with an indication for aspirin in primary prevention, in the presence of an active
internal bleeding process then do not administer aspirin 70-150 mg/d.
•
If a patient presents with type II diabetes mellitus without organ damage and without other elevated
cardiovascular risk factors then no antiplatelet treatment for cardiovascular prevention is indicated.
The experts also concluded that the newer antiplatelet products have not yet proven to be safe enough
in order to be widely applied in cardiovascular prevention.
4. Discussion
As described above, there is a guideline implementation gap which can be effectively bridged with
computerised decision support. Goud et al. [94] found that computerised decision support are an
effective strategy to overcome implementation barriers by increasing the familiarity with
recommendations, reducing guideline complexity and by reducing inertia to previous practice.
Furthermore, computerised decision support systems seemed less suited for overcoming organisational
and procedural constraints. The literature about guideline implementation is more and more dominated
by computerised guideline deployment and a lot of them show promising results. [175-178] But
computerised implementation strategies are not always effective, they can fail. Some studies found no
benefit from implementing guidelines using information technology. [4,179-181] Computerised
guideline implementation may fail because of various reasons. These can be categorised as userrelated (e.g. rushing the decision support out into clinical practice without training the users, see
above) or system-related. System-related reasons why guideline implementation strategies do not
prove effective in daily practice can, in turn, be related to the development process (e.g. inadequate
appraisal of clinical practice guidelines, not all of the probable clinical situations were covered in the
construction of the computer-interpretable guideline …) or the clinical practice guidelines themselves.
Guideline implementation in general may not prove to be effective in daily practice because of
different reasons intrinsic to clinical practice guidelines. For example, clinical practice guidelines
provide population-based and diagnosis-oriented recommendations. These, however, may be difficult
to reconcile with specific patients and problem-oriented patient encounters. [4,127,182] Also, clinical
50
practice guidelines may be incomplete and therefore not cover all probable or possible clinical
situations. [4,183] Furthermore, clinical practice guidelines may not contain all definitions necessary
to grasp the information contained in the guideline, there is need for additional external knowledge to
interpret them. This makes the correct use of guidelines in practice more difficult as missing
definitions may lead to variation of interpretation and implementation. [4] Fourth, guidelines may be
of poor quality and do not adequately reflect the current available evidence about the subject. [4]
Shaneyfelt et al. showed that about half of the guidelines they analysed did not meet methodological
standards. [184] And last, clinical practice guidelines contain a noteworthy amount of vague and
ambiguous information. Inadequate clarification of this knowledge may lead to further variation in
clinical practice. [4,185] The – more or less – generic process of building computerised decision
support described above, deals with most of these shortcomings (e.g. guideline appraisal,
disambiguation and deabstraction and verification of completeness). Thus, however computerised
implementation of clinical practice guidelines is a big leap forward towards increasing guideline
compliance in clinical practice, one must not expect it to be a panacea. Just like narrative guidelines,
computer-interpretable guidelines are bound to have some of the same and their own shortcomings.
Apparently, integrating clinical practice guidelines into today's information technology is necessary
but not sufficient to ensure compliance to best clinical practice. [4]
The question may arise whether these systems are accepted for use in daily practice. A recent doctoral
thesis that assessed barriers and facilitators of electronic guideline implementation and development
[186] showed that most users showed a positive attitude towards a newly implemented computerised
decision support system. Furthermore, most users experienced the system as easy to use and most
users were satisfied with the organisational and technical support.
A lot of different methodologies and approaches for creating computerised clinical decision support
systems exist, several examples were provided above. But on the other hand, the whole guideline
scenery is subject to standardisation. One could argue whether there is a certain approach that is
referred to as the standard. As for today there is no standard methodology and system exists, but there
is consensus that integrating computerised decision support in the existing local informatics system is
most efficient. [4] The existing formalisms, systems and approaches all have their specific pros and
cons and a detailed analysis is beyond the scope of this work.
The large number of computerised implementation methodologies available also provides an
interoperability problem. As described above, the export-specific knowledge base is a format
predestined for a specific guideline representation language. Furthermore, the programmer's activities
depend on the chosen formalism. Thus, the last final representation of the original guideline document
is the universal knowledge base which is not an operational document. This means that operational
51
computer-interpretable guidelines cannot be exchanged between different decision support systems
and that a new operational document must be constructed from the universal knowledge basis for a
specific target system. Interoperability is however being addressed by the SAGE project, which is one
of its interesting benefits. [146,147] Another example of computerised guideline implementation that
addresses the interoperability issue is the SAPHIRE project. [148]
There are several limitations to this work. First, the experts were not asked if they felt the set of
recommendations was complete. These recommendations form up a starting point for marking up a
specific guideline. However, these recommendations remain far from covering all probable clinical
scenarios. Therefore, further input of medical domain experts is needed in the course of the
implementation process. Second, the implementation of this work was only performed for antiplatelet
therapy in primary prevention while cardiovascular prevention is a broad topic that also includes
secondary prevention, lifestyle changes, various other classes of medication and many other
considerations. The modular and semantic representation of computer-interpretable guidelines lends
itself extremely well for such a multifaceted topic. Third, the Delphi procedure also had its limitations.
A low number of rounds were performed with a small consensus group. A higher number of rounds
and a bigger consensus group would increase the time and resource effort needed, but would increase
the reliability of the consensus procedure. Furthermore, in the first round of the procedure the three
cardiovascular experts together formed only one response. This broke the anonymity of the Delphi
procedure, which is one of its strong points. Their response also contains three individual votes, so that
their answer to the survey should actually weigh heavier. But because of the anonymity of the survey,
their answer could not be identified and, thus, we could not take into account this weighting.
The GRADE instrument was applied to the specific practice recommendations extracted from the 2009
RIZIV consensus report on the efficient use of drugs for cardiovascular prevention. The
recommendations for which consensus was reached in the Delphi procedure were not assessed
according to quality of evidence and strength of recommendation. Applying GRADE to such a current
controversial subject as antiplatelet therapy in primary prevention for patients with diabetes mellitus
type II would require an accurate review of the topic's evidence-base by different GRADE assessors.
This is in itself a thesis and, thus, is beyond the scope of this work.
5. Conclusion
Guideline adherence is globally unsatisfactory because clinical practice guidelines are inadequately
implemented. This guideline implementation gap requires the use of implementation strategies. They
should aim at improving patient outcome and patient satisfaction and aim at a cost-effective care.
There is a wide diversity of implementation strategies, but there is consensus that computerised
52
decisions support associated with existing informatics systems and integrated into the clinical
workflow are superiorly efficient. In order to construct a computerised clinical decision support
system, the clinical practice guideline must be transformed into a computer-operational format, a
computer-interpretable guideline. Different approaches exist for constructing computer-interpretable
guidelines, namely document-centric and model-centric approaches. Semantic document-centric
approaches for modeling clinical practice guidelines into an operational format are especially
interesting in case of cardiovascular prevention because of the multifaceted nature of the topic. In
order to implement this generic framework for constructing a computer-interpretable guideline,
specific practice recommendations were composed as a very condensed body of clinical evidence.
Most of the recommendations were assessed according to quality of evidence and strength of
recommendation in order to provide a minimal insight into the evidence behind the recommendations.
In this pilot implementation, the feasibility of extracting highly condensed evidence from clinical
practice guidelines was determined and an inventory of the necessary tools and methods to complete
this work for the subject of pharmacological primary prevention of cardiovascular risk was made.
6. References
1. Field MJ, Lohr MJ. Clinical Practice Guidelines: Directions for a New Program. National
Academy Press, Washington, 1990.
2. Field JM, Lohr KN, Editors. Guidelines for Clinical Practice: From Development To Use.
National Academy Press, Washington, 1992.
3. Fieschi M, Dufour JC, Staccini P, Gouvernet J, Bouhaddou O. Medical decision support systems:
Old dilemmas and new paradigms? Methods Inf Med. 2003;42(3):190-8.
4. Sonnenberg FA, Hagerty CG. Computer-interpretable clinical practice guidelines. Where are we
and where are we going? Yearb Med Inform. 2006:145-58.
5. Grimshaw JM, Russell IT. Effect of clinical guidelines on medical practice: a systematic review of
rigorous evaluations. Lancet. 1993 Nov 27;342(8883):1317-22.
6. Sciarra E. The importance of practice guidelines in clinical care. Dimens Crit Care Nurs. 2012
Mar-Apr;31(2):84-5.
7. Banzi R, Liberati A, Moschetti I, Tagliabue L, Moja L. A review of online evidence-based practice
point-of-care information summary providers. J Med Internet Res. 2010 Jul 7;12(3):e26.
8. Alper BS, Hand JA, Elliott SG, Kinkade S, Hauan MJ, Onion DK, Sklar BM. How much effort is
needed to keep up with the literature relevant for primary care? J Med Libr Assoc. 2004
Oct;92(4):429-37.
9. Woolf SH. Practice guidelines, a new reality in medicine. II. Methods of developing guidelines.
Arch Intern Med. 1992 May;152(5):946-52.
10. Ellrodt G, Cook DJ, Lee J, Cho M, Hunt D, Weingarten S. Evidence-based disease management.
53
JAMA. 1997 Nov 26;278(20):1687-92.
11. Grimshaw JM, Russel IT. Implementing clinical practice guidelines: can guidelines be used to
improve clinical practice? Effective Health Care. 1994;8:1-12
12. Leape LL, Park RE, Solomon DH, Chassin MR, Kosecoff J, Brook RH. Does inappropriate use
explain small-area variations in the use of health care services? JAMA. 1990 Feb 2;263(5):669-72.
13. Grimshaw JM, Hutchinson A. Clinical practice guidelines—do they enhance value for money in
health care? Br Med Bull. 1995 Oct;51(4):927-40.
14. Bastian H, Glasziou P, Chalmers I. Seventy-five trials and eleven systematic reviews a day: how
will we ever keep up? PLoS Med. 2010 Sep 21;7(9):e1000326.
15. Legros B, Boon P, Ceulemans B, Coppens T, Geens K, Hauman H, Lagae L, Meurs A, Mol L,
Ossemann M, van Rijckevorsel K, Van Zandijcke M, Vrielynck P, Wagemans D, Grisar T.
Development of an electronic decision tool to support appropriate treatment choice in adult
patients with epilepsy--Epi-Scope(®). Seizure. 2012 Jan;21(1):32-9.
16. Woolf SH. Practice guidelines: a new reality in medicine III: impact on patient care. Arch Intern
Med. 1993 Dec 13;153(23):2646-55.
17. Hayward RS. Clinical practice guidelines on trial. CMAJ. 1997 Jun 15;156(12):1725-7.
18. Cabana MD, Rand CS, Powe NR, Wu AW, Wilson MH, Abboud PC, Rubin HR. Why don’t
physicians follow clinical practice guidelines: a framework for improvement. JAMA.
1999;282(15):1458-1465.
19. Dufour JC, Bouvenot J, Ambrosi P, Fieschi D, Fieschi M. Textual guidelines versus computable
guidelines: a comparative study in the framework of the PRESGUID project in order to appreciate
the impact of guideline format on physician compliance. AMIA Annu Symp Proc. 2006:219-23.
20. McGlynn EA, Asch SM, Adams J, Keesey J, Hicks J, DeCristofaro A, Kerr EA. The quality of
health care delivered to adults in the United States. N Engl J Med. 2003 Jun 26;348(26):2635-45.
21. Asch SM, Kerr EA, Keesey J, Adams JL, Setodji CM, Malik S, McGlynn EA. Who is at greatest
risk for receiving poor-quality health care? N Engl J Med. 2006 Mar 16;354(11):1147-56.
22. Mangione-Smith R, DeCristofaro AH, Setodji CM, Keesey J, Klein DJ, Adams JL, Schuster MA,
McGlynn EA. The Quality of ambulatory care delivered to children in the United States. N Engl J
Med. 2007 Oct 11;357(15):1515-23.
23. Shrank WH, Asch SM, Adams J, Setodji C, Kerr EA, Keesey J, Malik S, McGlynn EA. The
quality of pharmacologic care for adults in the United States. Med Care. 2006 Oct;44(10):936-45.
24. Schuster MA, McGlynn EA, Brook RH. How good is the quality of health care in the United
States? Milbank Q. 1998;76(4):517-63, 509.
25. Schuster MA, McGlynn EA, Brook RH. How good is the quality of health care in the United
States? 1998. Milbank Q. 2005;83(4):843-95.
26. Grol R. Successes and failures in the implementation of evidence-based guidelines for clinical
practice. Med Care. 2001 Aug;39(8 Suppl 2):II46-54.
54
27. Knight AT, Cowling RM, Rouget M, Balmford A, Lombard AT, Campbell BM. Knowing but not
doing: selecting priority conservation areas and the research-implementation gap. Conserv Biol.
2008 Jun;22(3):610-7.
28. Shiffman RN, Michel G, Essaihi A, Thornquist E. Bridging the guideline implementation gap: a
systematic, document-centered approach to guideline implementation. J Am Med Inform Assoc.
2004 Sep-Oct;11(5):418-26.
29. Sjöborg B, Bäckström T, Arvidsson LB, Andersén-Karlsson E, Blomberg LB, Eiermann B,
Eliasson M, Henriksson K, Jacobsson L, Jacobsson U, Julander M, Kaiser PO, Landberg C,
Larsson J, Molin B, Gustafsson LL. Design and implementation of a point-of-care computerized
system for drug therapy in Stockholm metropolitan health region--Bridging the gap between
knowledge and practice. Int J Med Inform. 2007 Jul;76(7):497-506.
30. Feldstein A, Elmer PJ, Orwoll E, Herson M, Hillier T. Bone mineral density measurement and
treatment for osteoporosis in older individuals with fractures: a gap in evidence-based practice
guideline implementation. Arch Intern Med. 2003 Oct 13;163(18):2165-72.
31. Cahill NE, Heyland DK. Bridging the guideline-practice gap in critical care nutrition: a review of
guideline implementation studies. JPEN J Parenter Enteral Nutr. 2010 Nov-Dec;34(6):653-9.
32. Bodenheimer T. The American health care system--the movement for improved quality in health
care. N Engl J Med. 1999 Feb 11;340(6):488-92.
33. Julian DG. Translation of clinical trials into clinical practice. J Intern Med. 2004 Mar;255(3):30916.
34. van der Meer EW, Boot CR, Jungbauer FH, van der Klink JJ, Rustemeyer T, Coenraads PJ, van
der Gulden JW, Anema JR. Hands4U: a multifaceted strategy to implement guideline-based
recommendations to prevent hand eczema in health care workers: design of a randomised
controlled trial and (cost) effectiveness evaluation. BMC Public Health. 2011 Aug 25;11:669.
35. Tello-Bernabé E, Sanz-Cuesta T, del Cura-González I, de Santiago-Hernando ML, Jurado-Sueiro
M, Fernández-Girón M, García-de Blas F, Pensado-Freire H, Góngora-Maldonado F, de la PuenteChamorro MJ, Rodríguez-Pasamontes C, Martín-Iglesias S. Effectiveness of a clinical practice
guideline implementation strategy for patients with anxiety disorders in primary care: cluster
randomized trial. Implement Sci. 2011 Dec 1;6:123.
36. Van der Veen F, Mugala-Mukungu F, Kangudi M, Feris A, Katjitae I, Colebunder R. Antiretroviral
treatment in the private sector in Namibia. Int J STD AIDS. 2011 Oct;22(10):577-80.
37. Zwerver F, Schellart AJ, Knol DL, Anema JR, van der Beek AJ. An implementation strategy to
improve the guideline adherence of insurance physicians: an experiment in a controlled setting.
Implement Sci. 2011 Dec 21;6:131.
38. Abu-Hanna A, Eslami S, Schultz MJ, de Jonge E, de Keizer NF. Analyzing effects of providing
performance feedback at ward rounds on guideline adherence - the importance of feedback usage
55
analysis and statistical control charts. Stud Health Technol Inform. 2010;160(Pt 2):826-30.
39. Hartig JR, Allison J. Physician performance improvement: an overview of methodologies. Clin
Exp Rheumatol. 2007 Nov-Dec;25(6 Suppl 47):50-4.
40. Were MC, Shen C, Tierney WM, Mamlin JJ, Biondich PG, Li X, Kimaiyo S, Mamlin BW.
Evaluation of computer-generated reminders to improve CD4 laboratory monitoring in subSaharan Africa: a prospective comparative study. J Am Med Inform Assoc. 2011 MarApr;18(2):150-5.
41. Tierney WM, Hui SL, McDonald CJ. Delayed feedback of physician performance versus
immediate reminders to perform preventive care. Effects on physician compliance. Med Care.
1986 Aug;24(8):659-66.
42. Weingarten SR, Riedinger MS, Conner L, Lee TH, Hoffman I, Johnson B, Ellrodt AG. Practice
guidelines and reminders to reduce duration of hospital stay for patients with chest pain. An
interventional trial. Ann Intern Med. 1994 Feb 15;120(4):257-63.
43. Safran C, Rind DM, Davis RB, Ives D, Sands DZ, Currier J, Slack WV, Makadon HJ, Cotton DJ.
Guidelines for management of HIV infection with computer-based patient's record. Lancet. 1995
Aug 5;346(8971):341-6.
44. Rijksinstituut voor Ziekte- en Invaliditeitsverzekering. Het doelmatig gebruik van geneesmiddelen
bij de preventie van cardiovasculaire aandoeningen. 2009 May 14 [Cited 2012 Feb 26]. Available
from: http://www.riziv.fgov.be/drug/nl/statistics-scientific-information/consensus/2009-0514/pdf/lv.pdf
45. Georg G, Séroussi B, Bouaud J. Extending the GEM model to support knowledge extraction from
textual guidelines. Int J Med Inform. 2005 Mar;74(2-4):79-87.
46. Tierney WM, Overhage JM, Takesue BY, Harris LE, Murray MD, Vargo DL, McDonald CJ.
Computerizing guidelines to improve care and patient outcomes: the example of heart failure. J
Am Med Inform Assoc. 1995; Sep-Oct;2(5):316-22.
47. Lamy JB, Ebrahiminia V, Riou C, Seroussi B, Bouaud J, Simon C, Dubois S, Butti A, Simon G,
Favre M, Falcoff H, Venot A. How to translate therapeutic recommendations in clinical practice
guidelines into rules for critiquing physician prescriptions? Methods and application to five
guidelines. BMC Med Inform Decis Mak. 2010 May 28;10:31.
48. Guyatt GH, Oxman AD, Sultan S, Glasziou P, Akl EA, Alonso-Coello P, Atkins D, Kunz R,
Brozek J, Montori V, Jaeschke R, Rind D, Dahm P, Meerpohl J, Vist G, Berliner E, Norris S,
Falck-Ytter Y, Murad MH, Schünemann HJ; GRADE Working Group. GRADE guidelines: 1.
Introduction—GRADE evidence profiles and summary of findings tables. J Clin Epidemiol. 2011
Dec;64(12):1311-6.
49. Guyatt GH, Oxman AD, Vist GE, Kunz R, Falck-Ytter Y, Alonso-Coello P, Schünemann HJ;
GRADE Working Group. GRADE: an emerging consensus on rating quality of evidence and
strength of recommendations. BMJ. 2008 Apr 26;336(7650):924-6.
56
50. Guyatt GH, Oxman AD, Kunz R, Atkins D, Brozek J, Vist G, Alderson P, Glasziou P, Falck-Ytter
Y, Schünemann HJ. GRADE guidelines: 2. Framing the question and deciding on important
outcomes. J Clin Epidemiol. 2011 Apr;64(4):395-400.
51. Balshem H, Helfand M, Schünemann HJ, Oxman AD, Kunz R, Brozek J, Vist GE, Falck-Ytter Y,
Merpohl J, Norris S, Guyatt GH. GRADE guidelines: 3. Rating the quality of evidence. J Clin
Epidemiol. 2011 Apr;64(4):401-6.
52. Guyatt GH, Oxman AD, Vist G, Kunz R, Brozek J, Alonso-Coello P, Montori V, Akl EA,
Djulbegovic B, Falck-Ytter Y, Norris SL, Williams JW Jr, Atkins D, Meerpohl J, Schünemann HJ.
GRADE guidelines: 4. Rating the quality of evidence--study limitations (risk of bias). J Clin
Epidemiol. 2011 Apr;64(4):407-15.
53. Guyatt GH, Oxman AD, Montori V, Vist G, Kunz R, Brozek J, Alonso-Coello P, Djulbegovic B,
Atkins D, Falck-Ytter Y, Williams JW Jr, Meerpohl J, Norris SL, Akl EA, Schünemann HJ.
GRADE guidelines: 5. Rating the quality of evidence--publication bias. J Clin Epidemiol. 2011
Dec;64(12):1277-82.
54. Guyatt GH, Oxman AD, Kunz R, Brozek J, Alonso-Coello P, Rind D, Devereaux PJ, Montori VM,
Freyschuss B, Vist G, Jaeschke R, Williams JW Jr, Murad MH, Sinclair D, Falck-Ytter Y,
Meerpohl J, Whittington C, Thorlund K, Andrews J, Schünemann HJ. Grade guidelines 6. Rating
the quality of evidence--imprecision. J Clin Epidemiol. 2011 Dec;64(12):1283-93.
55. Guyatt GH, Oxman AD, Kunz R, Woodcock J, Brozek J, Helfand M, Alonso-Coello P, Glasziou P,
Jaeschke R, Akl EA, Norris S, Vist G, Dahm P, Shukla VK, Higgins J, Falck-Ytter Y, Schünemann
HJ; GRADE Working Group. GRADE guidelines: 7. Rating the quality of evidence-inconsistency. J Clin Epidemiol. 2011 Dec;64(12):1294-302
56. Guyatt GH, Oxman AD, Kunz R, Woodcock J, Brozek J, Helfand M, Alonso-Coello P, Falck-Ytter
Y, Jaeschke R, Vist G, Akl EA, Post PN, Norris S, Meerpohl J, Shukla VK, Nasser M,
Schünemann HJ; GRADE Working Group. GRADE guidelines: 8. Rating the quality of evidence-indirectness. J Clin Epidemiol. 2011 Dec;64(12):1303-10.
57. Guyatt GH, Oxman AD, Sultan S, Glasziou P, Akl EA, Alonso-Coello P, Atkins D, Kunz R,
Brozek J, Montori V, Jaeschke R, Rind D, Dahm P, Meerpohl J, Vist G, Berliner E, Norris S,
Falck-Ytter Y, Murad MH, Schünemann HJ; GRADE Working Group. GRADE guidelines: 9.
Rating up the quality of evidence. J Clin Epidemiol. 2011 Dec;64(12):1311-6.
58. Guyatt GH, Oxman AD, Kunz R, Vist GE, Falck-Ytter Y, Schünemann HJ; GRADE Working
Group. What is "quality of evidence" and why is it important to clinicians? BMJ. 2008 May
3;336(7651):995-8.
59. Guyatt GH, Oxman AD, Kunz R, Falck-Ytter Y, Vist GE, Liberati A, Schünemann HJ; GRADE
Working Group. Going from evidence to recommendations. BMJ. 2008 May 10;336(7652):104951.
60. Schünemann HJ, Oxman AD, Brozek J, Glasziou P, Jaeschke R, Vist GE, Williams JW Jr, Kunz R,
57
Craig J, Montori VM, Bossuyt P, Guyatt GH; GRADE Working Group. GRADE: grading quality
of evidence and strength of recommendations for diagnostic tests and strategies. BMJ. 2008 May
17;336(7653):1106-10.
61. Guyatt GH, Oxman AD, Kunz R, Jaeschke R, Helfand M, Liberati A, Vist GE, Schünemann HJ;
GRADE Working Group. Incorporating considerations of resources use into grading
recommendations. BMJ. 2008 May 24;336(7654):1170-3.
62. Jaeschke R, Guyatt GH, Dellinger P, Schünemann H, Levy MM, Kunz R, Norris S, Bion J;
GRADE Working Group. Use of GRADE grid to reach decisions on clinical practice guidelines
when consensus is elusive. BMJ. 2008 Jul 31;337:a744.
63. Brozek JL, Akl EA, Jaeschke R, Lang DM, Bossuyt P, Glasziou P, Helfand M, Ueffing E, AlonsoCoello P, Meerpohl J, Phillips B, Horvath AR, Bousquet J, Guyatt GH, Schünemann HJ; GRADE
Working Group. Grading quality of evidence and strength of recommendations in clinical practice
guidelines: Part 2 of 3. The GRADE approach to grading quality of evidence about diagnostic tests
and strategies. Allergy. 2009 Aug;64(8):1109-16.
64. Nair R, Aggarwal R, Khanna D. Methods of Formal Consensus in Classification/Diagnostic
Criteria and Guideline Development. Semin Arthritis Rheum. 2011 Oct;41(2):95-105.
65. Van de Ven AH, Delbecq AL. The nominal group as a research instrument for exploratory health
studies. Am J Public Health. 1972 Mar;62(3):337-42.
66. Gallagher M, Hares T, Spencer J, Bradshaw C, Webb I. The nominal group technique: a research
tool for general practice? Fam Pract. 1993 Mar;10(1):76-81.
67. Carney O, McIntosh J, Worth A. The use of the Nominal Group Technique in research with
community nurses. J Adv Nurs. 1996 May;23(5):1024-9.
68. Fink A, Kosecoff J, Chassin M, Brook RH. Consensus methods: characteristics and guidelines for
use. Am J Public Health. 1984 Sep;74(9):979-83.
69. Jones J, Hunter D. Consensus methods for medical and health services research. BMJ. 1995 Aug
5;311(7001):376-80.
70. Black N, Murphy M, Lamping D, McKee M, Sanderson C, Askham J, Marteau T. Consensus
development methods: a review of best practice in creating clinical guidelines. J Health Serv Res
Policy. 1999 Oct;4(4):236-48.
71. Linstone HA, Turoff M. The Delphi Method: techniques and applications. Addison-Wesley Pub.
Co., Advanced Book Program, Reading, 1975.
72. Keeney S, Hasson F, McKenna HP. A critical review of the Delphi technique as a research
methodology for nursing. Int J Nurs Stud. 2001 Apr;38(2):195-200.
73. Keeney S, Hasson F, McKenna H. Consulting the oracle: ten lessons from using the Delphi
technique in nursing research. J Adv Nurs. 2006 Jan;53(2):205-12.
74. Epstein A. Performance reports on quality--prototypes, problems, and prospects. N Engl J Med.
1995 Jul 6;333(1):57-61.
58
75. Unutzer J, Chen YF, Hafer E, Shields A, Powers D, Veith RC. Quality Improvement With Payfor-Performance Incentives in Integrated Behavioral Health Care. Am J Public Health. 2012 Apr
19. [Epub ahead of print]
76. Jha AK, Joynt KE, Orav EJ, Epstein AM. The long-term effect of premier pay for performance on
patient outcomes. N Engl J Med. 2012 Apr 26;366(17):1606-15.
77. Witter S, Fretheim A, Kessy FL, Lindahl AK. Paying for performance to improve the delivery of
health interventions in low- and middle-income countries. Cochrane Database Syst Rev. 2012 Feb
15;2:CD007899.
78. Eijkenaar F. Pay for Performance in Health Care: An International Overview of Initiatives. Med
Care Res Rev. 2012 Feb 6. [Epub ahead of print]
79. Perrier L, Buja A, Mastrangelo G, Vecchiato A, Sandona P, Ducimetiere F, Blay JY, Gilly FN,
Siani C, Biron P, Ranchere-Vince D, Decouvelaere AV, Thiesse P, Bergeron C, Dei Tos AP,
Coindre JM, Rossi CR, Ray-Coquard I. Clinicians’ adherence versus non adherence to practice
guidelines in the management of patients with sarcoma: a cost-effectiveness assessment in two
European regions. BMC Health Serv Res. 2012 Mar 28;12(1):82.
80. Kirigaya D, Nakayama T, Ishizaki T, Ikeda S, Satoh T. Management and treatment of osteoporosis
in patients receiving long-term glucocorticoid treatment: current status of adherence to clinical
guidelines and related factors. Intern Med. 2011;50(22):2793-800.
81. Oh SW, Lee HJ, Chin HJ, Hwang JI. Adherence to clinical practice guidelines and outcomes in
diabetic patients. Int J Qual Health Care. 2011 Aug;23(4):413-9.
82. Van der Veen F, Mugala-Mukungu F, Kangudi M, Feris A, Katjitae I, Colebunder R. Antiretroviral
treatment in the private sector in Namibia. Int J STD AIDS. 2011 Oct;22(10):577-80.
83. Cassimjee M, Suleman F. Adherence to hypertension treatment guidelines in state facilities in
KwaZulu-Natal, South Africa. J Eval Clin Pract. 2009 Dec;15(6):1077-81.
84. Lugtenberg M, Burgers JS, Besters CF, Han D, Westert GP. Perceived barriers to guideline
adherence: a survey among general practitioners. BMC Fam Pract. 2011 Sep 22;12:98.
85. Albers-Heitner P, Berghmans B, Nieman F, Lagro-Janssen T, Winkens R. Adherence to
professional guidelines for patients with urinary incontinence by general practitioners: a crosssectional study. J Eval Clin Pract. 2008 Oct;14(5):807-11.
86. Rietveld RP, ter Riet G, Bindels PJ, Schellevis FG, van Weert HC. Do general practitioners adhere
to the guideline on infectious conjunctivitis? Results of the Second Dutch National Survey of
General Practice. BMC Fam Pract. 2007 Sep 16;8:54.
87. Tsumura H, Fujita T, Tamura I, Morita Y, Yoshida M, Toyonaga T, Mukai H, inokuchi H, Kutsumi
H, Azuma T. Association between adherence to evidence-based guidelines for the prescription of
non-steroidal anti-inflammatory drugs and the incidence of gastric mucosal lesions in Japanese
patients. J Gastroenterol. 2010 Sept;45(9):944-51.
88. Suzuki T, Kaneko M, Saito I, Kokubu F, Kasahara K, Nakajima H, Adachi M, Shimbo T,
59
Sugiyama E, Sato H. Comparison of physicians’ compliance, clinical efficacy, and drug cost
before and after introduction of Asthma Prevention and Management Guidelines in Japan
(JGL2003). Allergol Int. 2010 Mar;59(1):33-41.
89. Sun YH, Eun BW, Sim SY, Cho KH, Ryoo E, Cho DY, Son DW, TChah H, Jeon IS. Poor
adherence and reasons for nonadherence to the asthma guidelines among pediatricians in Korea.
Asian Pac J Allergy Immunol. 2010 Jun-Sep;28(2-3):147-54.
90. Salinas GD, Williamson JC, Kalhan R, Thomashow B, Scheckermann JL, Walsh J, Abdolrasulnia
M, Foster JA. Barriers to adherence to chronic pulmonary disease guidelines by primary care
physicians. Int J Chron Obstruct Pulmon Dis. 2011;6:171-9.
91. Gerber B, Stussi G, Rosemann T, Senn O. The health care setting rather than medical speciality
impacts on physicians adherence to guideline-conform anticoagulation in outpatients with nonvalvular atrial fibrillation: a cross sectional survey. BMC Cardiovasc Disord. 2012 Feb 29;12:12.
92. Yeh KW, Chen SH, Chiang LC, Chen LC, Huang JL. Survey of asthma care in Taiwan: a
comparison of asthma specialists and general practitioners. Ann Allergy Asthma Immunol. 2006
Apr;96(4):593-9.
93. Karbach U, Schubert I, Hagemeister J, Ernstmann N, Pfaff H, Höpp HW. Physicians' knowledge
of and compliance with guidelines: an exploratory study in cardiovascular diseases. Dtsch Arztebl
Int. 2011 Feb;108(5):61-9.
94. Goud R, van Engen-Verheul M, de Keizer NF, Bal R, Hasman A, Hellemans IM, Peek N. The
effect of computerized decision support on barriers to guideline implementation: a qualitative
study in outpatient cardiac rehabilitation. Int J Med Inform. 2010 Jun;79(6):430-7.
95. Perez X, Wisnivesky JP, Lurslurchachai L, Kleinman LC, Kronish IM. Barriers to adherence to
COPD guidelines among primary care providers. Respir Med. 2012 Mar;106(3):374-81.
96. Van Steenkiste B, van der Weijden T, Stoffers He, Grol R. Barriers to implementing cardiovascular
risk tables in routine general practice. Scand J Prim Health Care. 2004 Mar;22(1):32-7.
97. Steinman MA, Patil S, Kamat P, Peterson C, Knight SJ. A taxonomy of reasons for not prescribing
guideline-recommended medications for patients with heart failure. Am J Geriatr Pharmacother.
2010 Dec;8(6):583-94.
98. Carlsen B, Bringedal B. Attitudes to clinical guidelines--do GPs differ from other medical
doctors? BMJ Qual Saf. 2011 Feb;20(2):158-62.
99. Haagen EC, Nelen WL, Hermens RP, Braat DD, Grol RP, Kremer JA. Barriers to physician
adherence to a subfertility guideline. Hum Reprod. 2005 Dec;20(12):3301-6.
100. Hofer TP, Zemencuk JK, Hayward RA. When there is too much to do: how practicing physicians
prioritize among recommended interventions. J Gen Intern Med. 2004 Jun;19(6):646-53.
101. Grol R. Beliefs and evidence in changing clinical practice. BMJ. 1997 Aug 16;315(7105):418-21.
102. Grol R, Grimshaw JM. From best evidence to best practice: effective implementation of change
in patients’ care. Lancet. 2003 Oct 11;362(9391):1225-30.
60
103. Grol R, Dalhuijsen J, Thomas S, Veld C, Rutten G, Mokkink H. Attributes of clinical guidelines
that influence use of guidelines in general practice: observational study. BMJ. 1998 Sep
26;317(7162):858-61.
104. Burgers JS, Grol RP, Zaat JO, Spies TH, van der Bij AK, Mokkink HG. Characteristics of
effective clinical guidelines for general practice. Br J Gen Pract. 2003 Jan;53(486):15-9.
105. Flamm M, Fritsch G, Seer J, Panisch S, Sönnichsen AC. Non-adherence to guidelines for
preoperative testing in a secondary care hospital in Austria: the economic impact of unnecessary
and double testing. Eur. J Anaesthesiol. 2011 Dec;28(12):867-73.
106. Wilke M, Grube RF, Bodmann KF. Guideline-adherent initial intravenous antibiotic therapy for
hospital-acquired/ventilator-associated pneumonia is clinically superior, saves lives and is
cheaper than non guideline adherent therapy. Eur J Med Res. 2011 Jul 25;16(7):315-23.
107. Orrick JJ, Segal R, Johns TE, Russell W, Wang F, Yin DD. Resource use and cost of care for
patients hospitalised with community acquired pneumonia: impact of adherence to infectious
diseases society of america guidelines. Pharmacoeconomics. 2004;22(11):751-7.
108. Ferrando A, Ivaldi C, Buttiglieri A, Pagano E, Bonetto C, Arione R, Scaglione L, Gelormino E,
Merletti F, Ciccone G. Guidelines for preoperative assessment: impact on clinical practice and
costs. Int J Qual Health Care. 2005 Aug;17(4):323-9.
109. Feuerstein M, Hartzell M, Rogers HL, Marcus SC. Evidence-based practice for acute low back
pain in primary care: patient outcomes and cost of care. Pain. 2006 Sep;124(1-2):140-9.
110. Gross R, Tabenkin H, Porath A, Heymann A, Greenstein M, Porter B, Matzliach R. The
relationship between primary care physicians’ adherence to guidelines for the treatment of
diabetes and patient satisfaction: findings from a pilot study. Fam Pract. 2003 Oct;20(5):563-9.
111. Reker DM, Duncan PW, Horner RD, Hoenig H, Samsa Gp, Hamilton BB, Dudley TK. Postacute
stroke guideline compliance is associated with greater patient satisfaction. Arch Phys Med
Rehabil. 2002 Jun;83(6):750-6.
112. Garg AX, Adhikari NKJ, McDonald H, Rosas-Arellano MP, Devereaux PJ, Beyene J, et al.
Effects of computerized clinical decision support systems on practitioner performance and patient
outcomes: a systematic review. JAMA. 2005;293:1223-1238.
113. Sucher JF, Moore FA, Todd SR, Sailors RM, McKinley BA. Computerized clinical decision
support: a technology to implement and validate evidence based guidelines. J Trauma. 2008
Feb;64(2):520-37.
114. Latoszek-Berendsen A, Tange H, van den Herik HJ, Hasman A. From clinical practice guidelines
to computer-interpretable guidelines. A literature overview. Methods Inf Med. 2010;49(6):55070.
115. Eccher C, Seyfang A, Ferro A, Miksch S. Updating a Protocol-based Decision-Support System’s
Knowledge Base: a Breast Cancer Case Study. In: Riano D, ten Teije Anette, Miksch S, Peleg M.
Knowledge representation for Health-Care. Berlin: Springer; 2011. p. 126-138.
61
116. Patel VL, Allen VG, Arocha JF, Shortliffe EH. Representing clinical guidelines in GLIF:
individual and collaborative expertise. J Am Med Inform Assoc. 1998 Sep-Oct;5(5):467-83.
117. Séroussi B, Bouaud J, Chatellier G, Venot A. Development of computerized guidelines for the
management of chronic diseases allowing to position any patient within recommended therapeutic
strategies. Stud Health Technol Inform. 2004;107(Pt 1):154-8.
118. Brouwers MC, Kho ME, Browman GP, Burgers JS, Cluzeau F, Feder G, Fervers B, Graham ID,
Grimshaw J, Hanna SE, Littlejohns P, Makarski J, Zitzelsberger L; AGREE Next Steps
Consortium. AGREE II: advancing guideline development, reporting and evaluation in health
care. CMAJ. 2010 Dec 14;182(18):E839-42.
119. The AGREE Next Step Consortium. Appraisal of Guidelines for REsearch and Evaluation II
Instrument [Internet]. [Place unknown]: [Publisher unknown]; 2009 May [Cited 2012 April 25].
Available from: http://www.agreetrust.org/index.aspx?o=1397.
120. Svátek V, Růzicka M. Step-by-step mark-up of medical guideline documents. Int J Med Inform.
2003 Jul;70(2-3):329-35.
121. McDonald CJ, Overhage JM. Guidelines you can follow and can trust. An ideal and an example.
JAMA. 1994 Mar 16;271(11):872-3.
122. Kaiser K, Miksch S. Versioning computer-interpretable guidelines: semi-automatic modeling of
'Living Guidelines' using an information extraction method. Artif Intell Med. 2009 May;46(1):5566.
123. de Clercq PA, Blom JA, Korsten HH, Hasman A. Approaches for creating computer-interpretable
guidelines that facilitate decision support. Artif Intell Med. 2004 May;31(1):1-27.
124. Rospocher M, Eccher C, Ghidini C, Hasan R, Seyfang A, Ferro A, Miksch S. Collaborative
Encoding of Asbru Clinical Protocols. In: Szmoszor M, Kostkova P. Electronic Healthcare: Third
International Conference; December 13-15 2010: Casablanca, Morocco: Springer; 2012. p. 135143
125. Eccher C, Larcher B, Purin B, Guarrera G, Graiff A, Forti S. A guideline-based co-operative
work framework integrating medical knowledge management services for heart failure care
delivery. Technol. Health Care. 2002 Dec; 10(6):477-479.
126. Kaiser K, Akkaya C, Miksch S. How can information extraction ease formalizing treatment
processes in clinical practice guidelines? A method and its evaluation. Artif Intell Med. 2007
Feb;39(2):151-63.
127. Peleg M, Tu S, Bury J, Ciccarese P, Fox J, Greenes RA, Hall R, Johnson PD, Jones N, Kumar A,
Miksch S, Quaglini S, Seyfang A, Shortliffe EH, Stefanelli M. Comparing computer-interpretable
guideline models: a case-study approach. J Am Med Inform Assoc. 2003 Jan-Feb;10(1):52-68.
128. Mulyar N, van der Aalst WM, Peleg M. A pattern-based analysis of clinical computerinterpretable guideline modeling languages. J Am Med Inform Assoc. 2007 Nov-Dec;14(6):78162
7.
129. De Clercq P, Kaiser K, Hasman A. Computer-Interpretable Guideline formalisms. Stud Health
Technol Inform. 2008;139:22-43.
130. Poikonen J. Arden Syntax: the emerging standard language for representing medical knowledge
in computer systems. Am J Health Syst Pharm. 1997 Feb 1;54(3):281-4.
131. Shahar Y, Miksch S, Johnson P. An intention-based language for representing clinical guidelines.
Proc AMIA Annu Fall Symp. 1996:592-6.
132. Tu SW, Musen MA. The EON model of intervention protocols and guidelines. Proc AMIA Annu
Fall Symp. 1996:587-91.
133. Boxwala AA, Peleg M, Tu S, Ogunyemi O, Zeng QT, Wang D, Patel VL, Greenes RA, Shortliffe
EH. GLIF3: a representation format for sharable computer-interpretable clinical practice
guidelines. J Biomed Inform. 2004 Jun;37(3):147-61.
134. Fox J, Johns N, Lyons C, Rahmanzadeh A, Thomson R, Wilson P. PROforma: a general
technology for clinical decision support systems. Comput Methods Programs Biomed. 1997
Sep;54(1-2):59-67.
135. Eccher C, Seyfang A, Ferro A, Miksch S. Embedding oncologic protocols into the provision of
care: the Oncocure project. Stud Health Technol Inform. 2009;150:663-7.
136. Ongenae F, De Backere F, Steurbaut K, Colpaert K, Kerckhove W, Decruyenaere J, De Turck F.
Towards computerizing intensive care sedation guidelines: design of a rule-based architecture for
automated execution of clinical guidelines. BMC Med Inform Decis Mak. 2010 Jan 18;10:3.
137. Shiffman RN, Agrawal A, Deshpande AM, Gershkovich P. An approach to guideline
implementation with GEM. Stud Health Technol Inform. 2001;84(Pt 1):271-5.
138. Shiffman RN, Karras BT, Agrawal A, Chen R, Marenco L, Nath S. GEM: a proposal for a more
comprehensive guideline document model using XML. J Am Med Inform Assoc. 2000 SepOct;7(5):488-98.
139. Tang PC, Young CY. ActiveGuidelines: integrating Web-based guidelines with computer-based
patient records. Proc AMIA Symp. 2000:843-7.
140. Hagerty CG, Pickens D, Kulikowski C, Sonnenberg F. HGML: a hypertext guideline markup
language. Proc AMIA Symp. 2000:325-9.
141. Shahar Y, Miksch S, Johnson P. The Asgaard project: a task-specific framework for the
application and critiquing of time-oriented clinical guidelines. Artif Intell Med. 1998 SepOct;14(1-2):29-51.
142. Johnson PD, Tu S, Booth N, Sugden B, Purves IN. Using scenarios in chronic disease
management guidelines for primary care. Proc AMIA Symp. 2000:389-93.
143. Hripcsak G, Cimino JJ, Johnson SB, Clayton PD. The Columbia-Presbyterian Medical Center
decision-support system as a model for implementing the Arden Syntax. Proc Annu Symp
63
Comput Appl Med Care. 1991:248-52.
144. Musen MA, Tu SW, Das AK, Shahar Y. EON: a component-based approach to automation of
protocol-directed therapy. J Am Med Inform Assoc. 1996 Nov-Dec;3(6):367-88.
145. Quaglini S, Stefanelli M, Lanzola G, Caporusso V, Panzarasa S. Flexible guideline-based patient
careflow systems. Artif Intell Med. 2001 Apr;22(1):65-80.
146. Tu SW, Musen MA, Shankar R, Campbell J, Hrabak K, McClay J, Huff SM, McClure R, Parker
C, Rocha R, Abarbanel R, Beard N, Glasgow J, Mansfield G, Ram P, Ye Q, Mays E, Weida T,
Chute CG, McDonald K, Molu D, Nyman MA, Scheitel S, Solbrig H, Zill DA, Goldstein MK.
Modeling guidelines for integration into clinical workflow. Stud Health Technol Inform.
2004;107(Pt 1):174-8.
147. Tu SW, Campbell J, Musen MA. The SAGE guideline modeling: motivation and methodology.
Stud Health Technol Inform. 2004;101:167-71.
148. Laleci GB, Dogac A. A semantically enriched clinical guideline model enabling deployment in
heterogeneous healthcare environments. IEEE Trans Inf Technol Biomed. 2009 Mar;13(2):26373.
149. Kumar A, Quaglini S, Stefanelli M, Ciccarese P, Caffi E. Modular representation of the guideline
text: an approach for maintaining and updating the content of medical education. Med Inform
Internet Med. 2003 Jun;28(2):99-115.
150. Seyfang A, Martinez-Salvador B, Serban R, Wittenberg J, Miksch S, Marcos M. Maintaining
formal models of living guidelines efficiently. In: Bellazzi R, Abu-Hanna A, Hunter J, editors.
Proceedings of the 11th conference on Artificial Intelligence in Medicine (AIME'07), LNAI 4594;
2007; Amsterdam, NL: Springer Verlag; p. 441-5.
151. Peleg M, Kantor R. Approaches for guideline versioning using GLIF. In: Musen MA, editor.
Proceedings of the 2003 American Medical Informatics Association (AMIA) annual symposium;
2003; Washington, DC: AMIA. p. 509-13.
152. Peleg M, Wang D, Fodor A, Keren S, Karnieli E. Adaptation of practice guidelines for clinical
decision support: a case study of diabetic foot care. In: ten Teije, A, Lucas P, Miksch S, editors.
Proceedings of the workshop AI techniques in healthcare: evidence-based guidelines and
protocols (at ECAI'06); 2006.
153. Votruba P, Miksch S, Seyfang A, Kosara R. Tracing the formalization steps of textual guidelines.
Stud Health Technol Inform. 2004;101:172-6.
154. Polvani K-A, Agrawal A, Karras B, Deshpande A, Shiffman R. GEM Cutter manual. Yale Center
for Medical Informatics, New Haven, 2000.
155. Votruba P, Miksch S, Kosara R. Facilitating knowledge maintenance of clinical guidelines and
protocols. Stud Health Technol Inform. 2004;107(Pt 1):57-61.
64
156. Shahar Y, Young O, Shalom E, Mayaffit A, Moskovitch R, Hessing A, et al. DEGEL: a hybrid,
multiple-ontology framework for specification and retrieval of clinical guidelines. In: Dojat M,
Keravnou E, Barahona P, editors. Proceedings of the 9th conference on artificial intelligince in
medicine in Europe, AIME 2003, vol. 2780, LNAI. Springer Verlag; 2003. p. 122-31.
157. Hoelzer S, Schweiger RK, Boettcher HA, Tafazzoli AG, Dudeck J. Value of XML in the
implementation of clinical practice guidelines--the issue of content retrieval and presentation.
Med Inform Internet Med. 2001 Apr-Jun;26(2):131-46.
158. Hoelzer S, Schweiger R, Dudeck J. Representation of practice guidelines with XML--modeling
with XML schema. Methods Inf Med. 2002;41(4):305-12.
159. Shalom E, Shahar Y, Taieb-Maimon M, Bar G, Yarkoni A, Young O, Martins SB, Vaszar L,
Goldstein MK, Liel Y, Leibowitz A, Marom T, Lunenfeld E. A quantitative assessment of a
methodology for collaborative specification and evaluation of clinical guidelines. J Biomed
Inform. 2008 Dec;41(6):889-903.
160. Eriksson H, Tu SW, Musen M. Semantic clinical guideline documents. AMIA Annu Symp Proc.
2005:236-40.
161. Roberts A, Gaizauskas R, Hepple M, Demetriou G, Guo Y, Roberts I, Setzer A. Building a
semantically annotated corpus of clinical texts. J Biomed Inform. 2009 Oct;42(5):950-66.
162. Pisanelli DM, Eccher C, Forti S. An ontological approach to support clinical governance: the case
of breast cancer guidelines. AMIA Annu Symp Proc. 2008 Nov 6:1096.
163. Teach RL, Shortliffe EH. An analysis of physician attitudes regarding computer-based clinical
consultation systems. Comput Biomed Res. 1981 Dec;14(6):542-58.
164. Elson RB. Clinical practice guidelines: the role of technology in perspective. Dis Management
Health Outcomes. 1997;1:63-74.
165. Berg M. Patient care information systems and health care work: a sociotechnical approach. Int J
Med Inform. 1999 Aug;55(2):87-101.µ
166. Essaihi A, Michel G, Shiffman RN. Comprehensive categorization of guideline
recommendations: creating an action palette for implementers. In: Musen MA (ed). Proc AMIA
2003. Washington, DC: Hanley and Belfus, 2003.
167. Greving JP, Buskens E, Koffijberg H, Algra A. Cost-effectiveness of aspirin treatment in the
primaryprevention of cardiovascular disease events in subgroups based on age, gender, and
varying cardiovascular risk. Circulation 2008 June 3;117(22):2875-83.
168. Pignone M, Earnshaw S, Pletcher MJ, Tice JA. Aspirin for the primary prevention of
cardiovascular disease in women: a cost-utility analysis. Arch Intern Med. 2007 Feb
12;167(3):290-5.
169. Lamotte M, Annemans L, Evers T, Kubin M. A multi-country economic evaluation of low-dose
aspirin in the primary prevention of cardiovascular disease.Pharmacoeconomics. 2006;24(2):155-
65
69.
170. Ogawa H, Nakayama M, Morimoto T, Uemura S, Kanauchi M, Doi N, Jinnouchi H, Sugiyama S,
Saito Y; Japanese Primary Prevention of Atherosclerosis With Aspirin for Diabetes (JPAD) Trial
Investigators. Low-dose aspirin for primary prevention of atherosclerotic events in patients with
type 2 diabetes: a randomized controlled trial. JAMA. 2008 Nov 12;300(18):2134-41.
171. Saito Y, Morimoto T, Ogawa H, Nakayama M, Uemura S, Doi N, Jinnouchi H, Waki M, Soejima
H, Sugiyama S, Okada S, Akai Y; Japanese Primary Prevention of Atherosclerosis With Aspirin
for Diabetes Trial Investigators. Low-dose aspirin therapy in patients with type 2 diabetes and
reduced glomerular filtration rate: subanalysis from the JPAD trial. Diabetes Care. 2011
Feb;34(2):280-5.
172. Okada S, Morimoto T, Ogawa H, Kanauchi M, Nakayama M, Uemura S, Doi N, Jinnouchi H,
Waki M, Soejima H, Sakuma M, Saito Y; Japanese Primary Prevention of Atherosclerosis With
Aspirin for Diabetes Trial Investigators. Differential effect of low-dose aspirin for primary
prevention of atherosclerotic events in diabetes management: a subanalysis of the JPAD trial.
Diabetes Care. 2011 Jun;34(6):1277-83.
173. Soejima H, Ogawa H, Morimoto T, Nakayama M, Okada S, Uemura S, Kanauchi M, Doi N,
Sakuma M, Jinnouchi H, Sugiyama S, Waki M, Saito Y; for the JPAD Trial Investigators. Aspirin
Reduces Cerebrovascular Events in Type 2 Diabetic Patients With Poorly Controlled Blood
Pressure. Circ J. 2012 Mar 23.
174. Hansson L, Zanchetti A, Carruthers SG, Dahlöf B, Elmfeldt D, Julius S, Ménard J, Rahn KH,
Wedel H, Westerling S. Effects of intensive blood-pressure lowering and low-dose aspirin in
patients with hypertension: principal results of the Hypertension Optimal Treatment (HOT)
randomised trial. HOT Study Group. Lancet. 1998 Jun 13;351(9118):1755-62.
175. Vissers MC, Biert J, van der Linden CJ, Hasman A. Effects of a supportive protocol processing
system (ProtoVIEW) on clinical behaviour of residents in the accident and emergency
department. Comput Methods Programs Biomed. 1996 Mar;49(2):177-84.
176. Rossi RA, Every NR. A computerized intervention to decrease the use of calcium channel
blockers in hypertension. J Gen Intern Med. 1997 Nov;12(11):672-8.
177. Schriger DL, Baraff LJ, Rogers WH, Cretin S. Implementation of clinical guidelines using a
computer charting system. Effect on the initial care of health care workers exposed to body fluids.
JAMA. 1997 Nov 19;278(19):1585-90.
178. Cannon DS, Allen SN. A comparison of the effects of computer and manual reminders on
compliance with a mental health clinical practice guideline. J Am Med Inform Assoc. 2000 MarApr;7(2):196-203.
179. Tierney WM, Overhage JM, Murray MD, Harris LE, Zhou XH, Eckert GJ, Smith FE, Nienaber
N, McDonald CJ, Wolinsky FD. Effects of computerized guidelines for managing heart disease in
primary care. J Gen Intern Med. 2003 Dec;18(12):967-76.
66
180. Montgomery AA, Fahey T, Peters TJ, MacIntosh C, Sharp DJ. Evaluation of computer based
clinical decision support system and risk chart for management of hypertension in primary care:
randomised controlled trial. BMJ. 2000 Mar 11;320(7236):686-90.
181. Ansari M, Shlipak MG, Heidenreich PA, Van Ostaeyen D, Pohl EC, Browner WS, Massie BM.
Improving guideline adherence: a randomized trial evaluating strategies to increase beta-blocker
use in heart failure. Circulation. 2003 Jun 10;107(22):2799-804.
182. Wetter T. Lessons learnt from bringing knowledge-based decision support into routine use. Artif
Intell Med. 2002 Mar;24(3):195-203.
183. Shiffman RN. Representation of clinical practice guidelines in conventional and augmented
decision tables. J Am Med Inform Assoc. 1997 Sep-Oct;4(5):382-93.
184. Shaneyfelt TM, Mayo-Smith MF, Rothwangl J. Are guidelines following guidelines? The
methodological quality of clinical practice guidelines in the peer-reviewed medical literature.
JAMA. 1999 May 26;281(20):1900-5.
185. Codish S, Shiffman RN. A model of ambiguity and vagueness in clinical practice guideline
recommendations. AMIA Annu Symp Proc. 2005:146-5.
186. Heselmans A. Evidence-based Health Care. Barriers and Facilitators of electronic guideline
implementation and development in ambulatory settings. [Doctoral Thesis]. Acco, Leuven, 2012.
67
Annex 1: AGREE II
i
Annex 2: Antiplatelet therapy in
cardiovascular prevention
ii