Improving medical decision making - Stroke prevention in AF

Guideline-related barriers to optimal prescription of oral anticoagulants in primary care Chapter 4

I ntrod u ction

Prevalence of atrial fibrillation (AF), a common arrhythmia, has increased over the last
thirty years. [1] In 2006 its prevalence ranged from 0.7% (age group 55-59) to 17.8%
(age group > 85). AF increases the risk and severity of stroke. [2,3] Antithrombotic therapy
with oral anticoagulation (OAC) decreases this stroke risk, but at the same time increases
the risk of major bleeding. [2] National and international clinical practice guidelines on
AF management provide guidance on how to weigh these risks against expected benefits,
and include recommendations on antithrombotic treatment [3,4] Yet, antithrombotic
treatment is often not in line with these recommendations. [5,6]For example, a study in
the Netherlands estimated that less than 50% of AF patients received treatment accord-
ing to national guidelines. [6]
Various types of barriers may thwart physicians to follow guidelines in clinical practice,
such as lack of familiarity with the guideline’s content, lack of skills or resources to change
current practice or patients not reconciling with guideline recommendations. [7] Subop-
timal guideline features (i.e., format and content) may also hamper implementation. [7,8]
Lugtenberg et al. reported this as one of the barriers that hindered general practitioners
(GPs) to follow the Dutch College of General Practitioners (NHG) AF guideline. [9]
Improving these features may positively affect guideline use. [8,10,11] Therefore, this
study systematically appraised the format and content of the NHG AF guideline [4] to
identify suboptimal features that may hamper its use in Dutch primary care. We focused
on the guideline section related to prescription of OACs for stroke prevention in AF
patients. The results of this appraisal may contribute to improving features of future AF
guideline versions, as well as to developing tools and strategies for AF guideline imple-
mentation.

Material and Methods

The NHG AF guideline
The NHG aims to promote evidence-based primary care by bridging the gap between
theory and practice. With 12.000 member [12], they cover around 80% of all Dutch GPs
[13] and nurse practitioners [14]. The NHG has developed over 100 guidelines covering
the diagnosis and treatment of acute and chronic conditions, with ten guidelines related
to cardiovascular diseases. In the current study, we reviewed the 2013-updated version
of the NHG guideline on diagnosis and treatment of patients with atrial fibrillation. [4]
It consists of 34 pages of background information on AF, recommendations for practice,
endnotes and references. We focused on three key recommendations: (I) Eligibility crite-
ria for OACs; (II) Which type of OAC to prescribe: coumarin derivatives versus new OACs

49

Chapter 4

Table 1. Key recommendations on oral anticoagulants prescription included in the NHG AF guideline
Recommendation 1 [R1] – Eligibility criteria for OACs
The following recommendations apply to patients with atrial fibrillation aged 65 years and older; younger patients are eligible for
assessment by a cardiologist.
• Recommend oral anticoagulants to all women aged 65 and older and all men 75 years and older (i.e. for patients with a

CHA2DS2-Vasc score of 2 or higher).
• Discuss with male patients aged 65 to 75 years without cardiovascular comorbidities (CHA2DS2-Vasc score of 1) that the

benefits of antithrombotic medication (prevention of thromboembolism) are outweighed by the disadvantages (risk of side
effects, such as bleeding) and for that reason antithrombotic medication is not indicated.
• Recommend aspirin when a contraindication for oral anticoagulants is present. See table with most important contraindications
for antithrombotic medication.

Recommendation 2 [R2]– Which type of OAC to prescribe: coumarin derivatives versus NOACs
When an indication for oral anticoagulants is present, coumarin derivatives are preferred.
• Consider NOACs only if all of the following conditions are met:
Ø age below 80 (arbitrary);
Ø relatively little comorbidity
Ø normal kidney function (GFR> 50 ml / min);
Ø high medication adherence.
• Absolute contraindications for NOACs are:
Ø patients with a mechanical artificial heart-valve;
Ø patients with a (currently rare) rheumatic mitral stenosis.

Recommendation 3 [R3] – type and dosage of coumarin derivatives
Choice of coumarin derivative is partly determined by agreement with the local thrombosis service. In the Netherlands short-
acting acenocoumarol1 mg and long-acting 3 mg phenprocoumon are available.
• In general, Thrombosis Services recommend taking the tablets once daily in the evening.
• When starting a coumarin derivative, a loading dose is given for the first days according to the table with loading doses of

coumarin derivatives for different patient populations.
• Self-monitoring of INR may be considered for patients who find regular monitoring by local thrombosis service burdensome.
• Coumarin derivative dosing by a thrombosis service should aim for an INR between 2.0 and 3.0.

© Copyright NHG 2014
Abbreviations used: AF, atrial fibrillation; CHA2DS2-Vasc score, score calculating stroke risk in AF
patients; GFR, glomerular filtration rate; INR, international normalized ratio; NHG, Nederlands huis-
artsgenootschap; NOACs, new oral anticoagulants; OAC, oral anticoagulant; R, recommendation.

(NOACs); and (III) Type and dosage of coumarin derivatives. The recommendations are
displayed in Table 1.

Systematic appraisal of guideline features
To systematically appraise the features of the NHG AF guideline, we used the Appraisal
of Guidelines, Research and Evaluation (AGREE) II tool [15] and the GuideLine Imple-
mentability Appraisal (GLIA) tool [16]. Both tools are publicly available, and have previ-
ously been used for guideline appraisals [17–19].
The AGREE II tool focuses on assessing the methodological rigour and transparency with
which a guideline has been developed. It contains 23 items grouped in six domains (Table
2, first column). Each item reflects a statement that refers to the guideline as a whole (e.g.,
“Key recommendations are easily identifiable”), and is scored on a seven-point Likert
scale, ranging from 1 (strongly disagree) to 7 (strongly agree) [15].

50

Guideline-related barriers to optimal prescription of oral anticoagulants in primary care

Table 2. AGREE II scores of the NHG AF guideline section on oral coagulants prescription

AGREE II domains, domain scoresa), and items Item scores Illustration of suboptimal features per domain
(SD) b)

SCOPE AND PURPOSE (63.9%)

Overall objective of the guideline is specifically 4.8 (1.5) Although some health questions are included as
described 4.0 (1.4) subheadings, the guideline does not provide an easy-to-
5.8 (1.0) access overview of all questions covered;
Health questions covered by the guideline are It is unclear whether children are included in the guideline’s
specifically described target population.

Target patient population of the guideline is
specifically described

STAKEHOLDER INVOLVEMENT (52.8%)

All relevant professional groups were included in 4.0 (0.8) The guideline development group did not include Chapter 4
the guideline development group 2.8 (0.5) cardiologists, pharmacists, neurologists, or representatives of
5.8 (1.0) anticoagulation clinics;
Views and preferences of the target patient The national patient association were asked for external
population have been sought review, but it is unclear if and how their suggestions were
addressed in the final guideline document.
Target users of the guideline are clearly defined

RIGOUR OF DEVELOPMENT(63.0%)

Systematic methods were used to search for 4.8 (1.0) Although the guideline group did apply selection criteria,
evidence 2.5 (1.3) these were not made explicit before evidence selectiond),
3.0 (1.6) and not available within the guideline document;
Criteria for selecting the evidence are clearly 2.5 (1.0) For some studies, the group performed a quality appraisal,
described 6.5 (0.6) but without applying a formal tool. Furthermore,
5.5 (1.0) recommendations lack a summary of the quality of
Strengths and limitations of the body of evidence 6.5 (0.6) underlying evidence that is easy to find and interpret for
are clearly describedc) guideline users.
It is unclear how evidence was translated into
Methods for formulating recommendations are recommendations, and there is no description of how the
clearly described guideline group solved any disagreements arising during
recommendation formulation;
Health benefits, side effects, and risks have been The Netherlands Society of Cardiology did not externally
considered review the guideline prior to publication, but most other
relevant stakeholder groups did.
There is an explicit link between recommendations
and supporting evidence

The guideline has been externally reviewed by
experts prior to publication

Procedure for updating the guideline is provided 7.0 (0.0)

CLARITY OF PRESENTATION(66.7%)

Recommendations are specific and unambiguouse) 5.5 (1.0) Key recommendations are not easy identifiable because they
Different disease management options are clearly 6.0 (0.8) do not have one specific layout or font; some, but not all
presented are (partly) captured in boxes. Further, some text boxes hold
3.5 (0.6) other types of information, such as a description of guideline
Key recommendations are easily identifiable development procedures, or a summary of what has changed
since the previous version of the guideline.

APPLICABILITY(57.3%)

Facilitators and barriers to application of the 4.0 (0.8) The guideline describes several barriers and facilitators.
guideline are described 4.8 (1.5) However, most are not clearly labelled as a such, and they
6.0 (0.0) are scattered throughout the text instead of grouped or
Guideline provides advice and tools for applying 3.0 (0.8) summarised in one section.
recommendations in practice The criteria for monitoring coumarin derivatives dosage
are clearly described, but less so for other OAC types. The
Potential resource implications of applying guideline does not provide any criteria to audit adherence or
recommendations have been considered monitor impact at the GP practice level.

The guideline presents monitoring and/or auditing
criteria

51

Chapter 4

Table 2. AGREE II scores of the NHG AF guideline section on oral coagulants prescription (continued)

AGREE II domains, domain scoresa), and items Item scores Illustration of suboptimal features per domain
(SD) b)

EDITORIAL INDEPENDENCE(87.5%) 6.0 (0.8) [no substantial suboptimal features with regard to the
6.5 (0.6) guideline’s editorial independence]
Views of funding body have not influenced the
guideline content

Competing interests of authors have been
recorded and addressed

Abbreviations used: AF, atrial fibrillation; GP, general practitioner; GRADE, Grading of
Recommendations Assessment, Development and Evaluation; NHG, Nederlands huisartsgenoot-
schap; OAC, oral anticoagulant; SD, standard deviation
a) Domain scores range between 0 and 100% and were calculated by summing up the individual

appraisers’ scores for each item within a domain, and then standardizing this as a percentage of
the possible maximum score for that domain
b) Average items scores of four appraisers. Item scores can range between 1 and 7, with lower scores
indicating less optimal features.
c) Overlaps with GLIA item ‘ Is quality of evidence that supports each recommendation stated ex-
plicitly?’
d) Based on data collected during structured interviews with guideline development group mem-
bers
e) Overlaps with GLIA ‘Executability’ and ‘Decidability’ domains; see Table 3

Table 3. GLIA scores of the three NHG recommendations on oral anticoagulants prescriptiona); ‘Y’
indicates an optimal feature, ‘N’ indicates a suboptimal feature.

Scores per
recommendation

GLIA domains and items ↓ R1 R2 R3 Illustration of suboptimal features b)

EXECUTABILITY

Is the recommended action stated specifically and Y N Y [R2] GPs might not execute the action ‘...consider
unambiguously? NOACs...’ consistently;

Is sufficient detail provided to perform the Y N Y [R2] Type and dose of NOACs are not specified.
recommended action?

DECIDABILITY

Can one consistently determine whether each N N N Examples of conditions that may not be consistently
condition in the recommendation was satisfied? applied by most GPs:

Are all reasonable combinations of conditions [R1 – referenced table] Contraindications, such as
addressed? Y Y Y ‘severe disturbance of liver function’;

[R2] ‘High medication adherence’;
Is the logical relationship between conditions clear? Y
Y Y [R3 – referenced table] ‘Relative contraindications’ to
treatment with warfarin and phenprocoumon.

VALIDITY

Is justification for the recommendation stated N Y Y [R1] Justification is stated in end notes, but not
explicitly? referenced as such in the recommendation;
For none of the recommendations, evidence quality
Is quality of evidence that supports each
recommendation stated explicitly? c) N N N was systematically appraised or stated.

52

Guideline-related barriers to optimal prescription of oral anticoagulants in primary care

Table 3. GLIA scores of the three NHG recommendations on oral anticoagulants prescriptiona); ‘Y’
indicates an optimal feature, ‘N’ indicates a suboptimal feature. (continued)

Scores per
recommendation

GLIA domains and items ↓ R1 R2 R3 Illustration of suboptimal features b)

FLEXIBILITY

Is strength of each recommendation stated NNN
explicitly?

Are patient characteristics specified that permit Y Y Y For all recommendations the strength is unclear due
individualization? to a lack of standardised terminology or labels.

Are practice characteristics specified that permit n.a. n.a. Y
modification?

EFFECT ON PROCESS OF CARE

Can the recommendation be executed without YYY Chapter 4
substantial disruption in current workflow? n.a.

Can the recommendation be pilot tested without YYY
substantial resource commitment?

MEASURABILITY

Can adherence to this recommendation be YYY
measured? n.a.

Can outcomes of this recommendation be YYY
measured?

NOVELTY & INNOVATION

Can the recommendation be performed without Y N Y [R2] GPs need to gain knowledge on the
acquisition of new knowledge/skills? recommended use and (side) effects of NOACs;

Is the recommendation consistent with attitudes/ Y Y Y [R2] NOAC prescription is discouraged, while
beliefs of the intended audience? patients may expect access to the latest drugs, or

Is the recommendation consistent with patient Y N Y explicit room for shared decision making.
expectations?

COMPUTABILITY

Are all patient data available for electronic N N N All recommendations have some conditions for
implementation of the recommendation? which data are unavailable (e.g., artificial heart valve,

Are the recommendation’s conditions defined N N N arrangements between patient and anticoagulation
specific enough for electronic implementation? clinic), or that lack specificity (see Decidability

Is each recommended action defined specific items);
enough for electronic implementation? N Y Y [R1] The action in statement c (‘Discuss with...’)

Is it clear by what means a recommended action can Y Y may not be specific enough to allow electronic
be executed in an electronic setting? Y implementation.

Number (%) of suboptimal features per 7 9 5
recommendation (35) (45) (24)

Abbreviations: GLIA, guideline implementability appraisal; GPs, general practitioners; N, suboptimal
feature/GLIA score ‘No’; NHG, Nederlands huisartsgenootschap; NOACs, new oral anticoagulants; R,
recommendation; Y, optimal feature/GLIA score ‘Yes’; n.a., not applicable
a) See Table 1 for a description of each of the three key recommendations
b) This column contains examples to illustrate why recommendation features were considered sub-

optimal; each example is preceded by the number of the key recommendation—for example,
[R1]—it refers to.
c) This item overlaps with AGREE II item ‘Strengths and limitations of the body of the evidence are
clearly described’.

53

Chapter 4

To identify obstacles to implementation of the guideline’s key recommendations on
OAC prescription (Table 1), we completed GLIA appraisals. GLIA consists of 21 items in
eight dimensions that – in contrast to AGREE II — are scored at the level of individual
recommendations (Table 3, first column). Each item is formulated as a question (e.g., “Is
justification for the recommendation stated explicitly?”) with response categories ‘yes’,
‘no’, ‘not applicable’, and ‘unsure’. We did not assess GLIA’s global dimension that ap-
praises the guideline in its entirety and largely overlaps with the AGREE appraisal.

Data collection and analysis
Following the AGREE II and GLIA manuals [20,21], our appraisal panel consisted of four
experts, representing a mix of clinical and methodological guideline expertise: one
general practitioner (WL), one expert on antithrombotic treatment and stroke prevention
in AF patients (DA), and two experts on guideline development and implementation (AB,
SV). Panel members first individually performed the appraisals, using the online AGREE
(www.agreetrust.org) and GLIA (eGLIA; http://nutmeg.med.yale.edu/glia) tools. They also
provided additional information in free text fields to explain their scores. The appraisal
process was primarily informed by the guideline document itself, but when necessary,
extra information was collected from: i) the NHG website; ii) the booklet on NHG guideline
development procedures [22]; and iii) a structured interview with two members of the
NHG AF guideline development group. The appraisal coordinator (AB) then summarized
the resultsas input for a group discussion based on which panel members could alter
their scores when they considered this appropriate (e.g., to correct for available data that
were overlooked during the initial appraisal). We discussed every item for which scores
differed by more than one point, and every item for which the NHG development group
members provided additional information.
AGREE II domain scores were calculated by summing up the individual appraisers’ scores
for each item within a domain (i.e., obtained score), and then standardizing this as a
percentage of the possible maximum score for that domain [20], as follows:

Domain score in % = (obtained score − minimum possible score)
(maximum possible score − minimum possible score)

As a result of the consensus procedure for GLIA scores, features were categorized as
optimal (‘Y’ in Table 3) or suboptimal (‘N’ in Table 3). Per recommendation, we calculated
the percentage of suboptimal features as follows:

Percentage of suboptimal features = #suboptimal features of recommendation
total # GLIA features − not applicable features

54

Guideline-related barriers to optimal prescription of oral anticoagulants in primary care Chapter 4

R es u lts

AGREE II appraisal of overall guideline features
Appendix 1 presents the individual appraisers’ scores before and after group discus-
sion. The group discussion resulted in 24 out of 92 (26%) scores being changed. Main
reasons for appraisers to change their score were: correction for available data that were
overlooked during the initial appraisal (8 of 24 changes; 33%); a change of opinion fol-
lowing clarification of other appraisers’ opinion (7 of 24; 29%); correction for additional
information provided by the NHG guideline development group members (6 of 24; 25%).
After the group discussion, standard deviations of item scores ranged between 0 and 1.6,
with the majority (75%) being 1 or lower.
Table 2 presents the final item and domain scores assigned during the AGREE appraisal
of the guideline. Domain scores ranged from 52.8% for ‘Stakeholder involvement’ to
87.5% for ‘Editorial independence’.

GLIA appraisal of key recommendation features
Table 3 displays suboptimal features at the recommendation-level that may hinder the
guideline’s implementation and applicability in practice. The percentage of suboptimal
features for the three key recommendations on OAC prescription (R1-3) ranged from
24% to 45%. The panel considered the domains ‘Effect on process of care’ and ‘Measur-
ability’ optimal across all recommendations.
We found that all recommendations suffered from suboptimal decidability due to am-
biguous or unclear conditions for when to apply a recommendation. For example, not all
GPs may agree on what could be considered ‘high’ (R2) levels of medication adherence,
or ‘relative contraindications’ for prescribing (R3 - referenced table). This, together with
the lack of detail on how to prescribe NOACs (i.e., suboptimal executability of R2), led
the panel to score many of the computability features as problematic. This reflects the
panel’s expectation that operationalizing the recommendations in an electronic informa-
tion system may be difficult.
The strength of recommendations, and thus the degree to which they applied to all
patients, was often unclear, resulting in a suboptimal flexibility score. This stemmed from
a lack of standardization of how recommendations were formulated. Terminology ranged
from “absolute contraindications of NOACs are...” (R2) to “Coumarin derivative dosing […]
should aim” (R3), and from “Recommend oral anticoagulants ...” (R1) to “... consider NOACs
only...” (R2). In some cases, actions were only suggested indirectly: “...coumarin deriva-
tives are preferred” (R2); “...a loading dose is given for the first days” (R3).

55

Chapter 4

D isc u ssion

In this study we systematically appraised features of the NHG atrial fibrillation guide-
line to identify guideline-related barriers that may hamper optimal prescription of
oral anticoagulants in Dutch primary care. The editorial independence of the guideline
development group scored high; scores for all other aspects of the guideline’s method-
ological quality were acceptable. At the recommendation level, the main implementation
obstacles were the lack of explicit statements on the quality of the underlying evidence,
unclarity around the strength of recommendations, and suboptimal computability ham-
pering operationalization of recommendations in electronic systems.
The scores for the NHG AF guideline were high in comparison to those assigned in other,
similar guideline appraisal studies. For example, the systematic review of guideline ap-
praisal studies by Alonso-Coello et al. [23] summarised the methodological quality of
626 guidelines, and reported mean AGREE II scores that were lower for all six domains.
The study by Sabharwal et al. [17] appraised 101 cardiac clinical practice guidelines.
Compared to the NHG AF guideline, they found higher mean scores for the ‘Scope and
Purpose’ domain (64% and 85%, respectively) and for ‘Clarity of Presentation’ (67%
versus 82%), but lower scores for the four remaining domains.

Suggestions to improve the NHG AF guideline

Extend stakeholder involvement in the guideline development process
The AGREE II domain ‘Stakeholder involvement’ obtained the lowest score of all domains
(53%). This was partly due to the lack of representation of a wide range of stakeholders
in the guideline development group, which consisted of general practitioners and an
epidemiologist. Other relevant disciplines, such as neurologists and representatives of
anticoagulation clinics, were consulted at the external review stage. Yet, AGREE II and
other accepted guideline standards advocate multidisciplinary development groups be-
cause they tend to generate more balanced views than single-specialty groups. [24,25]
Inviting representatives of other disciplines as members of the NHG guideline develop-
ment group would be one way to increase multidisciplinary involvement. An alternative
approach may be to consult the current panel of external reviewers earlier on in the
process, for example when setting the guideline scope, selecting or rating the evidence,
or when formulating the recommendations.
The low ‘Stakeholder involvement’ score also stemmed from the apparent limited extent
to which patient experiences and expectations informed the NHG AF guideline. The na-
tional patient organisation was invited for external review, but it was unclear if and how
their suggestions were addressed. Similar to the approaches for increased stakeholder
involvement described above, patient involvement may be facilitated by having patient

56

Guideline-related barriers to optimal prescription of oral anticoagulants in primary care Chapter 4

representation in the guideline development group, or by formal patient consultation at
earlier stages of guideline development. [26] Alternatively, a literature review or patient
interviews could inform a guideline section summarising patient views on and experi-
ences with antithrombotic treatment for stroke prevention.

Standardise layout and language of key recommendations
Although the key recommendations are clearly presented in the online summary of
the guideline (https://www.nhg.org/standaarden/samenvatting/atriumfibrilleren), they
remain hidden in the full text version. Using a specific font, framing them in a box, or
(if possible) presenting them as a flowchart would support distinguishing key recom-
mendations from other types of information in the guideline. Additionally, applying
standardised language may help guideline users to recognise recommendations as
such, whilst aligning interpretations of whether recommendations should be considered
relevant for all patients. The Grading of Recommendations, Assessment, Development
and Evaluation (GRADE) framework [27]proposes standardised guideline language:
recommendations can be either strong (level ‘1’) or weak (level ‘2’), which translate into
the phrases ‘we recommend’ or ‘we suggest’, respectively. Adopting GRADE as part of the
guideline development methodology also ensures systematic assessment of the quality
of the underlying evidence, with the letters ‘A’ (indicating high quality) to ‘D’ (very low
quality) explicitly communicating evidence quality at the recommendation level. For
future versions of the AF guideline, the NHG’s updated procedure booklet (published
January 2015) includes guidance on how to use the GRADE framework for assessing and
summarising quality of the underlying evidence.

Suggest criteria for monitoring the guideline’s use in practice
Facilitating local or regional monitoring of the guideline’s use in practice requires
clearly defined criteria derived from the guideline’s key recommendations. For the NHG
AF guideline, examples of criteria may be “the percentage of female patients who are
prescribed oral anticoagulants and are aged 65 years or older”, or “the percentage of
patients on coumarin derivatives with an INR between 2.0 and 3.0”. Suggesting monitor-
ing criteria as part of the guideline would provide a suitable starting point for developing
audit and feedback, which Dutch GPs considered an encouraging strategy to improve
guideline adherence [28].

Prepare electronic implementation in parallel with the guideline development
process
Recent studies have focused on providing GPs with clinical decision support systems
(CDSS) to improve primary care stroke prevention in AF patients. [29–31] These systems
use decision rules to evaluate current treatment of the patient and, if necessary, recom-

57

Chapter 4

mend the GP to modify it. Creating these decision rules involves translating guidelines
into a format that is interpretable by a computer. The GLIA dimensions ‘Decidability’
and ‘Computability’ relate to obstacles for electronic implementation, i.e., translating
guideline recommendations into actionable, computable decision rules. In the current
study, the NHG AF guideline scored poorly for these dimensions, indicating presence
of ambiguous terms. Although this guideline has been included in CDSSs for Dutch
GPs [26,29], the suboptimal computability hampered interpretation and translation of
individual guideline statements into electronic decision rules. Especially the lack of
clear definitions for certain contra-indications and the unavailability of structured data
to identify contra-indications in electronic health records required input from an expert
group of clinicians to fully convert the guideline into unambiguous decision rules [26].
Based on this finding we suggest involving a CDSS specialist when formulating recom-
mendations for future updates of the NHG AF guideline. By preparing electronic imple-
mentation in parallel with the guideline development process, vague and inconsistent
recommendations can be identified and resolved before publication. [30] This may not
only improve overall implementability of the guideline in practice, but also facilitate the
development of effective CDSS interventions.
In conclusion, this study provides pointers for optimising future versions of the NHG
AF guideline. Future research should investigate whether applying these suggestions
indeed positively affects implementation of the guideline in primary care, which in turn
may improve the adequacy of antithrombotic treatment and stroke prevention in patients
with atrial fibrillation.

Acknowledgments

We would like to thank Maureen van der Donk and Wim Opstelten for providing ad-
ditional information on the NHG guideline development process. Sabine van der Veer
is funded by the European Renal Association – European Dialysis Transplant Association
(ERA-EDTA) for a research fellowship on guideline development and implementation.
Wim Lucassen is appointed member of the Authorisation Committee of the NHG.

58

Guideline-related barriers to optimal prescription of oral anticoagulants in primary care

R eferences

1. Heeringa J, van der Kuip D a M, Hofman A, et al. Prevalence, incidence and lifetime risk of Chapter 4
2. atrial fibrillation: the rotterdam study. Eur Hear J. 2006;2​ 7:​949–53.
3. Hart RG, Pearce L a., Aguilar MI. Meta-analysis: Antithrombotic therapy to prevent stroke in
4. patients who have nonvalvular atrial fibrillation. Ann Intern Med. 2007;1​ 46:​857–67.
5. Camm AJ, Lip GYH, De Caterina R, et al. 2012 focused update of the ESC Guidelines for the
management of atrial fibrillation. Eur Heart J. 2012;3​ 3(21):​2719–47.
6. NHG werkgroep atriumfibrilleren. M79 NHG-Standaard Atriumfibrilleren. 2013;5​ 6(8):​
392–401.
7. Nieuwlaat R, Olsson SB, Lip GYH, et al. Guideline-adherent antithrombotic treatment is
8. associated with improved outcomes compared with undertreatment in high-risk patients
with atrial fibrillation. The Euro Heart Survey on Atrial Fibrillation. Am Heart J. 2007;1​ 53(6):​
9. 1006–12.
Arts DL, Visscher S, Opstelten W, Korevaar JC, Abu-Hanna A, van Weert HC. Frequency and
10. risk factors for under- and over-treatment in stroke prevention for patients with non-valvular
atrial fibrillation in general practice. PLoS ONE [Electronic Resour. 2013;8​ (7):e​ 67806.
11. Cabana MD, Rand CS, Powe NR, et al. Why don’t physicians follow clinical practice guidelines?
12. A framework for improvement. JAMA. 1999;2​ 82(15):1​ 458–65.
13. Shekelle PG, Kravitz RL, Beart J, Marger M, Wang M, Lee M. Are nonspecific practice guidelines
14. potentially harmful? A randomized comparison of the effect of nonspecific versus specific
15. guidelines on physician decision making. Heal Serv Res. 2000;​34(7):1​ 429–48.
16. Lugtenberg M, Zegers-van Schaick JM, Westert GP, Burgers JS. Why don’t physicians adhere
to guideline recommendations in practice? An analysis of barriers among Dutch general
17. practitioners. Implement Sci. 2009;4​ (1):5​ 4.
18. Woolf S, Schünemann HJ, Eccles MP, Grimshaw JM, Shekelle P. Developing clinical practice
guidelines: types of evidence and outcomes; values and economics, synthesis, grading, and
presentation and deriving recommendations. Implementation Science. 2012. p. 61.
Gagliardi AR, Brouwers MC, Palda V a, Lemieux-Charles L, Grimshaw JM. How can we improve
guideline use? A conceptual framework of implementability. Implement Sci. 2011;6​ (1):​26.
NHG. Het NHG ter introductie [Internet]. 2016 [cited 2016 Mar 16]. Available from: www.nhg.
org/het-nhg-ter-introductie
Hassel DTP van, Kaseleijn A, Kenens RJ. Cijfers uit de registratie van huisartsen. Nivel. 2014;
Heiligers PJM, Noordman J, Korevaar JC, et al. Kennisvraag: praktijkondersteuners in de
huisartspraktijk (POH’s) klaar voor de toekomst? 2012;
Brouwers MC, Kho ME, Browman GP, et al. AGREE II: advancing guideline development, report-
ing and evaluation in health care. Can Med Assoc J. 2010;​182(18):E​ 839–42.
Shiffman RN, Dixon J, Brandt C, et al. The GuideLine Implementability Appraisal (GLIA): devel-
opment of an instrument to identify obstacles to guideline implementation. BMC Med Inform
Decis Mak. 2005;​5:2​ 3.
Sabharwal S, Patel V, Nijjer SS, et al. Guidelines in cardiac clinical practice: evaluation of their
methodological quality using the AGREE II instrument. J R Soc Med. 2013;1​ 06(8):​315–22.
van Dijk LJ, Nelen WL, D’Hooghe TM, et al. The European Society of Human Reproduction
and Embryology guideline for the diagnosis and treatment of endometriosis: an electronic
guideline implementability appraisal. Implement Sci. 2011;​6(1):7​ .

59

Chapter 4

19. Nagler E V, Vanmassenhove J, van der Veer SN, et al. Diagnosis and treatment of hyponatre-
mia: a systematic review of clinical practice guidelines and consensus statements. BMC Med.
20. 2014;1​ 2:1​ .
21. Brouwers MEA. Appraisal of Guidelines for Research & Evaluation II. Agree Next Steps Con-
22. sort. 2009;(​May):1​ –56.
Workgroup E. eGLIA 2 instructions [Internet]. 2011. Available from: nutmeg.med.yale.edu/
23. eglia2/eGLIAInstructions.pdf
NHG. NHG Procedure booklet [Internet]. 2010 [cited 2015 Jan 20]. Available from: https:/​ /
24. www.nhg.org/sites/default/files/content/nhg_org/uploads/procedureboek_versie_090810.
25. pdf
Alonso-Coello P, Irfan A, Solà I, et al. The quality of clinical practice guidelines over the last
26. two decades: a systematic review of guideline appraisal studies. Qual Saf Health Care. 2010;​
27. 19(6):​e58.
28. Graham R, Mancher M, Wolman DM, Greenfiled S, Steinberg E. Institute of Medicine: Clinical
29. Practice Guidelines We Can Trust. Natl Acad Press. 2011;​1–13.
30. Eccles MP, Grimshaw JM, Shekelle P, Schünemann HJ, Woolf S. Developing clinical practice
guidelines: target audiences, identifying topics for guidelines, guideline group composition
31. and functioning and conflicts of interest. Implementation Science. 2012. p. 60.
(G-I-N) GIN. G-I-N PUBLIC Toolkit : Patient and public involvement in guidelines. [Internet].
32. 2012. Available from: http://www.g-i-n.net/working-groups/gin-public/toolkit
Guyatt GH, Oxman AD, Vist GE, et al. GRADE: An emerging consensus on rating quality of
evidence and strength of recommendations. Chinese J Evidence-Based Med. 2009;​9(1):8​ –11.
Lugtenberg M, Burgers JS, Han D, Westert GP. General practitioners’ preferences for interven-
tions to improve guideline adherence. J Eval Clin Pract. 2014;2​ 0(6):8​ 20–6.
Arts DL, Abu-Hanna A, Büller HR, Peters RJG, Eslami S, van Weert HCPM. Improving stroke
prevention in patients with atrial fibrillation. Trials. 2013;1​ 4:1​ 93.
Chen R, Valladares C, Corbal I, Anani N, Koch S. Early Experiences from a guideline-based
computerized clinical decision support for stroke prevention in atrial fibrillation. Stud Health
Technol Inform. 2013;​192:​244–7.
Bajorek B, Magin P, Hilmer S, Krass I. A cluster-randomized controlled trial of a computerized
antithrombotic risk assessment tool to optimize stroke prevention in general practice: a
study protocol. BMC Health Serv Res. 2014;1​ 4(1):5​ 5.
Goud R, Hasman A, Strijbis A-M, Peek N. A parallel guideline development and formalization
strategy to improve the quality of clinical practice guidelines. Int J Med Inform. 2009;7​ 8(8):​
513–20.

60

Chapter 5

Improving stroke prevention in patients
with atrial fibrillation

D.L. Arts, A. Abu-Hanna, H.R. Büller, R.J.G. Peters, S. Eslami, H.C.P.M. van Weert

Chapter 5

A bstract

Background
Patients with atrial fibrillation (AF) are at increased risk for stroke. Antithrombotic treat-
ment reduces this risk. Antithrombotic treatment consists of either administration of oral
anticoagulants (OAC) or the provision of an antiplatelet drug. International guidelines
provide advice on the preferred treatment, thereby balancing the risks and benefits of
OAC. However, adherence to these guidelines is reported to be as low as 50%. There is
paucity in research on why adherence rates are low. Recent studies have shown decision
support systems can improve guideline adherence. We investigate the use of a clinical
decision support system to improve guideline adherence among general practitioners
(GPs) treating patients with AF and study reasons for guideline non-adherence.

Methods/Design
The study is a randomized controlled trial, which is performed among Dutch general prac-
titioners. Initially, GPs in the vicinity of the Academic Medical Center (AMC) in Amsterdam
will be included, after which other practices will be recruited. We have developed a novel
decision support system that displays a list with pending messages for the on-screen
medical record in real time. Messages are generated on a server that evaluates a decision
rule based on the atrial fibrillation guideline of the Dutch College of General Practitioners.
By interacting with the list, messages can be opened for a description and explanation, or
be ignored. GPs are allocated into three groups: 1) control group; 2) intervention group
A, in which messages can be ignored without justification; and 3) intervention group B, in
which messages can only be ignored with justification.
Our main outcome measure is the between-group difference in the proportion of patients
receiving antithrombotic prescriptions in adherence to the Dutch GP guideline for atrial
fibrillation. Secondary outcomes are reasons GPs state for deviating from the guideline
and the effect on guideline adherence of requiring justification when ignoring a message.

Discussion
This paper describes the protocol for a cluster randomized trial to study the effects of a
clinical decision support system in patients with atrial fibrillation. The system is charac-
terized by a non-interruptive presentation and real-time messages that are updated after
each relevant action the GP performs.

Trial registration
This trial is registered with the Dutch Trial Register under registration number 3570.

62

Improving stroke prevention in patients with atrial fibrillation

B ackgro u nd

Patients with atrial fibrillation (AF) are at high risk for stroke, with up to a fivefold risk Chapter 5
compared to patients without AF [1]. Apart from the increased risk, stroke severity is
increased [2]. Therefore, patients with AF need to be treated with antithrombotic medica-
tion to reduce stroke risk. Oral anticoagulants (OAC), such as warfarin and the new oral
anticoagulants (for example, dabigatran), reduce the risk of stroke by 60%, whereas anti-
platelet drugs reduce the risk by 20% [3,4]. Because OAC increases the risk of bleeding,
not every patient is eligible for this treatment, as there are patients with a pre-existing
increased risk of bleeding. Guidelines have been developed to help the physician decide
which treatment is appropriate for a patient. These guidelines incorporate contraindi-
cations for OAC, and stroke risk stratification schemes to balance risks and benefits.
Estimates of current adherence to stroke risk guidelines in the prevention of stroke in
patients with atrial fibrillation is about 50% internationally, leading to both over- and
undertreatment. This is true for both primary and secondary care [5-7]. Research on the
reasons for these low adherence rates is lacking. We hypothesize that patient prefer-
ence and medical context, lack of time and lack of overview are important drivers in this
process. These are reasons that have been shown to influence guideline adherence in
general in previous studies [8]. More insight into the reasons stated by physicians for
non-adherence could prove valuable for future research and guideline development.
The aim of this study is to investigate the effect of a clinical decision support system
(CDSS) on adherence to the guideline that recommends antithrombotic treatment for
stroke prevention for patients with atrial fibrillation. Clinical decision support systems
have proven to be effective interventions for improving various healthcare processes
[9,10]. However, evidence for clinical improvement is sparse [9]. We hope to expand the
current body of evidence for the effectiveness of CDSS with findings of increased guide-
line adherence and, in follow-up projects, better clinical outcomes. Furthermore, not all
instances of non-adherence are undesirable. Physicians may have good clinical reasons
(for example, comorbidity, social environment) to deviate from recommended treatment.
Therefore, the second objective of this study is to study reasons for deviation. Lastly, we
would like to evaluate whether the requirement for justification influences response to a
clinical decision support system.

M ethods / design

This study will be a cluster randomized controlled trial. Randomization at the patient
level was considered, but deemed inapplicable due to the fact that one GP would
have patients in both the control and intervention groups, thus leading to possible

63

Chapter 5

contamination bias. Randomization at the GP level was not possible because most GPs
work in group practices and share patients, which would lead to a GP in the control group
reading the reports of another GP who had received the intervention. This would also
lead to contamination bias. It was therefore decided to randomize on the GP cluster level,
where GPs treating the same patients will be regarded as a randomization unit (usually
two or three GPs).
GPs will be allocated into one of the following three groups:
1. Will receive no messages (control group)
2. Will receive messages that can be ignored without justification (intervention 1)
3. Will receive messages that can only be ignored with justification (intervention 2)

Intervention
We have developed a plugin (an extra piece of software that is not an integral part of the
main system) for a single GP electronic health records system (EHRS). The CDSS plugin
can be classified as active (it does not wait to be asked), non-interruptive (does not inter-
rupt the workflow), and critiquing (provides feedback based on the current treatment
scheme). The basic functionality of this decision support system is based on detecting
relevant events based on triggers, evaluating the events and responding to them. Several
trigger points in the GPs’ EHR have been designed. Trigger points are events that may
occur in a system that then trigger an action (for instance ‘Closing a record’= > ‘send data
to the server’). These triggers are:
1. opening a patient’s electronic medical record on the computer
2. adding laboratory data (for example, a new glomerular filtration rate (GFR))
3. changing medication (for example, prescribing aspirin)
4. updating the patient’s problem list (for example, registering a diagnosis of atrial fibril-

lation)
Whenever a trigger is activated, information from the on-screen medical record (the
medical record that is currently opened on the GP’s screen) is sent to a remote server
via a secure Internet connection. This information consists of recent laboratory data (12
months), active medication and a list of the patient’s diagnoses. On the remote server a
clinical rule engine (CRE) evaluates the provided information using the computerized de-
cision rule (see decision rule). The evaluation of the decision rule may lead to a response
in the form of a message. This message is formatted as an XML (a file format designed
to exchange data) document and sent to the CDSS plugin that has been installed on
the GP’s computer. The plugin reads the XML file and evaluates the contents. It extracts
the message title and displays part of the title in a thin sidebar that is attached to the
right side of the GP’s screen. The GP can move the mouse cursor over the sidebar to
display the full message title. He or she can then choose to either ignore the message
by clicking on the ‘ignore’ button, or open the message by clicking anywhere on the

64

Improving stroke prevention in patients with atrial fibrillation Chapter 5

message title. When the message is opened, a report is shown that displays the complete
message, as well as the findings that have led to the message, and a clickable link to
online background information. Upon first appearance, the message title is displayed
with a red background in the sidebar, indicating the message is new. If the GP does not
interact with the message, the background will become orange the next time the GP
opens the electronic medical record, indicating the message has been shown before but
not dealt with yet. When the GP changes the treatment plan according to the guideline,
the background will turn green and the message title will be updated. This can happen
without opening the message; that is, the GP might already know that he or she needs
to change the treatment plan and doing so will also result in a green background and
updated message without the GP having to interact with the CDSS plugin.
It is important to note that by using this procedure, real-time feedback to the GP is pro-
vided. This means that the GP will see a direct effect of every relevant action he or she
performs on the screen.
When analyzing baseline data to determine the current proportion of correctly treated
patients, we found that recorded diagnoses of stroke were often incomplete or inac-
curate. Therefore, a single question was added to the messages of the rule engine, which
asks the GP for the exact type of stroke if/when the International Classification of Primary
Care (ICPC) [11] code for stroke (K90) is found on the patient’s diagnoses list. The fol-
lowing options will be presented to the GP: subarachnoid hemorrhage (K90.01), hemor-
rhagic stroke (K90.02) and ischemic stroke (K90.03). Once the question is answered, the
response is saved for future use and sent to the clinical rule engine, where the entire
decision rule can be evaluated.
GPs who are allocated into the third group (intervention 2) will not be able to ignore a
message without providing justification. The GP is presented with several predefined
options and a free text field, which will allow him or her to enter other explanations.
The GP cannot continue without answering this question. When the GP tries to close the
on-screen medical record without providing a reason for deviating from the advice, he
or she is reminded once more of the message and asked to respond. An independent
adjudication committee will value the legitimacy of the reasons provided to deviate from
the advised treatment and will contact GPs when necessary.

Decision rule
The computerized decision rule is based on the Dutch GP guideline for atrial fibrillation
authored by the Dutch College of General Practitioners [12]. This guideline uses the
CHADS2 score, a commonly used stroke risk stratification scheme, which is calculated by
assigning 1 point each for the presence of congestive heart failure, hypertension, age 75
years or older, and diabetes mellitus and by assigning 2 points each for history of stroke
or transient ischemic attack (TIA) [13].

65

Chapter 5

The decision process consists of two phases:
1. The rule engine uses the decision rule to determine the recommended treatment.
2. The rule engine compares the actual treatment to the recommended treatment and
generates the message according to the result.
The first phase is a two-step procedure:
1. Determine whether contraindications for OAC are present. If this is the case, antiplate-
let drugs are the preferred treatment.
2. Calculate the CHADS2 score. For scores of one and lower, the preferred treatment is
antiplatelet drugs; for scores of two and higher, the preferred treatment is OAC.
In the second phase, the recommended treatment is compared to the actual treatment. A
message is generated using this information (for example, ‘It is advised to stop OAC and
start antiplatelet drugs, due to the fact that this patient has a CHADS2 score of 1’). There
are a total of 11 unique messages.

Baseline measurements
Baseline measurements will be performed to determine the current proportion of
patients with AF who are treated in accordance with the guideline. These data will be
gathered by automatically evaluating the decision rule on all patients in an anonymized
copy of the GP’s EHR databases before the start of the study.

Regulatory aspects
The medical ethics committee of the AMC deemed this study exempt from ethical ap-
proval because the intervention is a way of providing valid medical information to GPs
and does not change medical treatment or use invasive procedures. Furthermore, only
anonymized patient data is used for analysis.
Every practice has provided written consent to participate in this study via a designated
GP representative.
This trial is registered with the Dutch trial register under number 3570.

Data collection
Whenever a message is shown, clicked, or ignored in the CDSS plugin, this event is stored
in a database along with the current time and date. This will allow us to analyze the way
the system is used by the GPs and will inform us about possible improvements that can
be made. The responses GPs give to the questions about the type of stroke and the
reason to ignore the message, if any, are saved inside the XML document and sent back
to the remote server where the document is stored. During the study, these XML files
will be opened and the responses will be extracted. To calculate adherence rates we
use anonymized patient data (disease list, medication, GP contacts, lab data), which are
extracted directly from the GP’s EHR databases.

66

Improving stroke prevention in patients with atrial fibrillation Chapter 5

Population
We will include GP practices in the Netherlands that use the EHR for which the plugin
was developed. Initially, we will include every GP practice (18 clusters of GPs, with a total
of 35 GPs) that is part of the GAZO (a network of GP practices in Amsterdam Southeast).
Decision support messages will be shown for every patient with atrial fibrillation (both
new and existing patients) who visits the GP in his or her practice.

Sample size and analysis
A power analysis showed that 300 patients are needed for a power of 0.8 when assuming
a base adherence of 50%, an absolute effect size of 10% for intervention group 2 and
20% for intervention group 3, with a test significance level of 0.05. Because GPs are clus-
tered for randomization, a correction for these clustering effects was introduced. For the
cluster correction analysis, an intraclass correlation of .1 was assumed, based on studies
using similar clustering [14]. This resulted in a required sample size of approximately
500 patients to result in an effective sample size of 300.
When data collection has been concluded, differences in adherence percentages will be
calculated per GP cluster. A chi-square test will be used to compare the differences in
the three groups.

Outcomes
The primary outcome is the difference in the proportion of patients with AF, treated in
adherence to the guideline between groups 1, 2 and 3.
The secondary outcomes are the reasons GPs provide for deviating from the guideline
and the manner in which they respond to required justification (intervention 2).

D isc u ssion

This study will investigate the effectiveness of a decision support system in primary care
by implementing a system that advises the general practitioner on antithrombotic treat-
ment for stroke prevention in patients with atrial fibrillation. We will use a novel system
that does not rely on pop-ups, but rather a sidebar that displays real-time messages
concerning the actual patient, as studies have shown this to be the most effective timing
for a CDSS [10].
The success of the study depends on the ability of the system to attract the attention of
the GP and on the quality of data stored in the EHR. One of our challenges was finding
the right mode of presentation, where messages are visible to the GP without interfering
with their usual workspace. This meant we had to investigate computer use among GPs,
especially which screen resolutions were used and how much space was available on

67

Chapter 5

their screen for our system. As it turned out, many GPs were working on older systems
with low-resolution screens. We therefore had to adapt our system to actively show part
of the message titles, the rest being entirely visible on demand. We hope this mode of
presentation will not interfere significantly with the GP’s workspace, while remaining
visible enough for the GP to notice.

Guideline conversion
We could not fully convert the complete guideline into our decision rule, as some ele-
ments were not available as structured data in the GP’s EHR. For instance, the finding
‘Fundus bleeds of the eye’ has no code that can be used to identify it. Thus, we had
to adapt our decision rule to work with the data that were available. Also, severe renal
failure and liver failure were not defined with cut-off values, so we had to define these.
These are important factors to consider during future guideline development, as decision
support systems are expected to play an increasingly important role in healthcare. The
recently introduced Logical Elements Rule Method (LERM) could play an important role
in ensuring guideline-CDSS compatibility [15].

Reasons for deviation from the guideline
Based on findings in literature on this subject, we expect GPs will have a number of valid
reasons to provide treatment different from the guideline [8,16]. These are expected to
be in the following categories: patient preference, diagnostic uncertainty and contextual
factors (for example, medication prescribed by specialist). Almost all the other reasons
we expect GPs to report will be related to incomplete registration. For instance, a contra-
indication for oral anticoagulation cannot be a reason to disregard the system’s advice,
because the system checks for contraindications. Thus, the GP needs to update EHR to
reflect the latest clinical situation.

Effect size and reasons for non-adherence
We expect a 10% and 20% absolute effect size in intervention groups A and B, respec-
tively. We hypothesize that the main reason for non-adherence is that there are too many
parameters in this guideline to be evaluated effectively during a 10-minute visit. If a
patient develops diabetes and reaches the age of 75, that has an impact on multiple
guidelines. It is virtually impossible to know every guideline by heart, as there are cur-
rently one hundred Dutch GP guidelines, 10% of which are revised every year. Hence,
the GP will often not be aware that the patient in fact needs to receive OAC instead of
aspirin due to the fact that his or her CHADS2 score increased by two points.

68

Improving stroke prevention in patients with atrial fibrillation

C oncl u sion

This paper describes the protocol for a cluster randomized trial to study the effects of
a clinical decision support system in patients with atrial fibrillation. The system is char-
acterized by a non-interruptive presentation and real-time messages that are updated
after each action the GP performs (if the action is part of the decision rule). We expect an
absolute effect size of 10 to 20%, because current guideline adherence is low (around
50%) and the guideline might be missed or is difficult to evaluate during a patient visit
in a busy practice.

A bbre v iations

AF: Atrial fibrillation; CDSS: Clinical decision support system; CRE: Clinical rule engine;
EHRS: Electronic health records system; GP: General practitioner; ICPC: International
Classification of Primary Care; OAC: Oral anticoagulant; XML: Extensible markup language.

Authors’ contributions Chapter 5

DLA: Converting the guideline to a decision rule, coordinating development with the
involved parties, substantial programming for the plugin, writing this protocol. HCPMW:
The original study protocol, acquiring funds, guiding the project, co-authoring the pro-
tocol, taking part in the justification committee.AA-H: The concept of the CDSS plugin,
getting the necessary parties involved, guiding the project, co-authoring the protocol.
SE: The concept of the CDSS plugin, working out the details of the CDSS plugin concept,
coordinating with the involved parties. HRB: Co-authoring the protocol, taking part in
the justification committee. RJGP: Co-authoring the protocol, taking part in the justifica-
tion committee. All authors provided revisions to the paper, and all authors read and
approved the final manuscript.

69

Chapter 5

R eferences

1. Wolf PA, Abbott RD, Kannel WB. Atrial fibrillation as an independent risk factor for stroke: the
2. Framingham Study. Stroke. 1991;2​ 2(8):9​ 83‑8.
3. Lin HJ, Wolf PA, Kelly-Hayes M, Beiser AS, Kase CS, Benjamin EJ, et al. Stroke severity in atrial
4. fibrillation. The Framingham Study. Stroke. 1996;​27(10):1​ 760-4. Epub 1996/10/01.
5. Hart RG. Meta-analysis Antithrombotic Therapy to Prevent Stroke in Patients who have noval-
vular atrial fibrillation. Annals of internal medicine. 2007.
6. Ogilvie IM, Welner SA, Cowell W, Lip GY. Ischaemic stroke and bleeding rates in ‘real-
7. world’ atrial fibrillation patients. Thrombosis and haemostasis. 2011;1​ 06(1):​34-44. Epub
8. 2011/05/27.
9. Dinh T, Nieuwlaat R, Tieleman RG, Buller HR, van Charante NA, Prins MH, et al. Antithrombotic
10. drug prescription in atrial fibrillation and its rationale among general practitioners, internists
11. and cardiologists in The Netherlands--The EXAMINE-AF study. A questionnaire survey. Inter-
national journal of clinical practice. 2007;6​ 1(1):​24-31. Epub 2007/01/19.
12. Nieuwlaat R, Capucci A, Camm AJ, Olsson SB, Andresen D, Davies DW, et al. Atrial fibrillation
management: a prospective survey in ESC Member Countries - The Euro Heart Survey on
13. Atrial Fibrillation. Eur Heart J. 2005;​26(22):​2422‑34.
14. Ogilvie IM, Welner SA, Cowell W, Lip GY. Characterization of the proportion of untreated and
15. antiplatelet therapy treated patients with atrial fibrillation. The American journal of cardiol-
ogy. 2011;1​ 08(1):1​ 51-61. Epub 2011/05/07.
Cabana, Rand CS, Powe NR, Wu AW, Wilson MH. Why Don ’ t Physicians Follow guidlines, A
Framework for Improvement. October. 1999;0​ 718.
Bright TJ, Wong A, Dhurjati R, Bristow E, Bastian L, Coeytaux RR, et al. Effect of clinical
decision-support systems: a systematic review. Ann Intern Med. 2012;​157(1):2​ 9-43. Epub
2012/07/04.
Kawamoto K, Houlihan CA, Balas EA, Lobach DF. Improving clinical practice using clinical
decision support systems: a systematic review of trials to identify features critical to success.
Bmj. 2005;​330(7494):​765. Epub 2005/03/16.
An international glossary for primary care. Report of the Classification Committee of the
World Organization of National Colleges, Academies and Academic Associations of General
Practitioners/Family Physicians (WONCA). The Journal of family practice. 1981;​13(5):6​ 71-81.
Epub 1981/10/01.
Opstelten W, Boode BS, Heeringa J, Rutten FH, Goudswaard AN. [Summary of the practice
guideline ‘Atrial fibrillation’ (first revision) from the Dutch College of General Practitioners].
Nederlands tijdschrift voor geneeskunde. 2010;1​ 54:​A1570. Epub 2010/10/30. Samenvat-
ting van de standaard ‘Atriumfibrilleren’ (eerste herziening) van het Nederlands Huisartsen
Genootschap.
Gage BF, Waterman AD, Shannon W, Boechler M, Rich MW, Radford MJ. Validation of Clinical
Classification Schemes for Predicting Stroke. JAMA: The Journal of the American Medical As-
sociation. 2001;2​ 85(22):​2864‑70.
Campbell MK GJ, Steen N. For the Changing Professional Pratice in Europe Group: Sample size
calculations for cluster randomised trials. J Health Serv Res Policy. 2000(5):1​ 2‑6.
Medlock S, Opondo D, Eslami S, Askari M, Wierenga P, de Rooij SE, et al. LERM (Logical
Elements Rule Method): a method for assessing and formalizing clinical rules for decision
support. International journal of medical informatics. 2011;​80(4):​286-95. Epub 2011/02/22.

70

Improving stroke prevention in patients with atrial fibrillation

16. Ardery G, Carter BL, Milchak JL, Bergus GR, Dawson JD, James Pa, et al. Explicit and implicit
evaluation of physician adherence to hypertension guidelines. Journal of clinical hyperten-
sion (Greenwich, Conn). 2007;​9(2):1​ 13‑9.

Chapter 5

71



Chapter 6

Effectiveness and usage of a decision
support system to improve stroke

prevention in general practice: a cluster
randomized controlled trial

D.L. Arts, A. Abu-Hanna, S.K. Medlock, H.C.P.M. van Weert

Chapter 6

A bstract

Background and aims
Adherence to guidelines pertaining to stroke prevention in patients with atrial fibrillation
is poor. Decision support systems have shown promise in increasing guideline adher-
ence. To improve guideline adherence with a non-obtrusive clinical decision support
system integrated in the workflow. Secondly, we seek to capture reasons for guideline
non-adherence.

Design and Setting
A cluster randomized controlled trial in Dutch general practices.

Method
A decision support system was developed that implemented properties positively asso-
ciated with effectiveness: real-time, non-interruptive and based on data from electronic
health records. Recommendations were based on the Dutch general practitioners guide-
line for atrial fibrillation that uses the CHA2DS2-VAsc for stroke risk stratification. Usage
data and responses to the recommendations were logged. Effectiveness was measured
as adherence to the guideline. We used a chi square to test for group differences and a
mixed effects model to correct for clustering and baseline adherence.

Results
Our analyses included 781 patients. Usage of the system was low (5%) and declined over
time. In total, 76 notifications received a response: 58% dismissal and 42% acceptance.
At the end of the study, both groups had improved, by 8% and 5% respectively. There
was no statistically significant difference between groups (Control: 50%, Intervention:
55% P=0.23). Clustered analysis revealed similar results. Only one usable reason for
non-adherence was captured.

Conclusion
Our study could not demonstrate the effectiveness of a decision support system in gen-
eral practice, which was likely due to lack of use. Our findings should be used to develop
next generation decision support systems that are effective in the challenging setting of
general practice.

74

Effectiveness and usage of a decision support system to improve stroke prevention in general practice Chapter 6

I ntrod u ction

The burden of atrial fibrillation (AF) increases year by year in societies with aging popula-
tions, such as Western Europe and the United States [1-3]. This burden is largely due to the
five-fold increased risk for stroke that is associated with AF [4]. There is ample evidence
that the efficacy of oral anticoagulation (OAC) in stroke prevention is excellent [5-7].
Nonetheless overtreatment should be avoided, as this can result in hemorrhaging, mainly
intracerebral and gastrointestinal. Therefore, treatment strategies for stroke prevention
have been optimized over recent years, and clinical practice guidelines describing these
treatment strategies are publicly available [8]. However, adherence to these guidelines
remains poor [9-11]. This non-adherence is mainly related to underuse of OAC in patients
with medium to high stroke risk. Two important reasons for OAC underuse are 1) the
complexity of the decision rules used and 2) physicians’ concerns with the bleeding risk
associated with OAC. However, the benefit of stroke prevention greatly outweighs the risk
of bleeding due to OAC, and OAC should therefore not be withheld when indicated [6].
Studies have shown that clinical decision support systems (CDSSs) have the potential
to improve guideline adherence [12-14], as these systems can make physicians aware
of the guidelines when they lack the appropriate knowledge and eliminate inappropri-
ate use of clinical decision rules. The overall effectiveness of CDSS on mortality has not
been established, but a recent review did find a moderate improvement (RR = 0.82; 95%
CI = [0.68, 0.99]) in morbidity outcomes [15]. CDSSs therefore present a possible solution
for the issue of guideline non-adherence in stroke prevention among AF patients.
CDSS effectiveness varies by study and domain, and several studies have tried to de-
termine what factors predict success or failure,[16-18] although it is hard to pinpoint
these [19]. We have developed a CDSS that implements properties that appeared to
have a positive effect on CDSS effectiveness, most notably: “implementing decision
support as part of the workflow”, “during the time of decision making” and “optimized
human-computer interface“ [17, 20]. Too many alerts will tend to result in all alerts being
ignored, a phenomenon known as “alert fatigue” [21]. Given the possible adverse effects
of “alert fatigue” and interruption [22], we considered the optimal interface to be one
which minimized these effects.
In this study we investigate the effectiveness of our CDSS as measured by general
practitioners’ adherence to the Dutch GP guideline for patients with atrial fibrillation. A
secondary objective of this study is to gain insight into reasons for deviations from the
guideline and to adjudicate these reasons with peers.
We hypothesized that the implementation of a non-obtrusive CDSS, integrated in the
GP’s workflow, will increase guideline adherence. Secondly, we expect GPs will often
have valid reasons for guideline non-adherence.

75

Chapter 6

Materials and methods

The trial protocol for this study, which was dubbed the “Expert-AF” trial, has been pub-
lished elsewhere [23]. Below, we will provide a short overview of the most important
aspects of the methodology used.

Design
A cluster-randomized controlled design was used. Randomization was done at the GP
practice level to reduce contamination bias. GPs were allocated into one of the following
three groups:
1. Received no messages (control group)
2. Received messages that could be declined without documenting justification (inter-

vention 1)
3. Received messages that could only be declined if justification was documented

(intervention 2)
The planned allocation ratio was 2:1:1 respectively. Randomization was performed in the
statistical environment R [24] using the ‘sample’ function to create a random sequence
from the list of GP practices by DA. GPs were aware that they were allocated to a variant
of the system, but were blinded as to their allocation and how the variants differed.

Regulatory aspects
The medical ethics committee of the AMC deemed this study exempt from ethical ap-
proval because the intervention is a way of providing valid medical information to GPs
and does not change medical treatment or use invasive procedures. Furthermore, only
anonymized patient data was used for analysis. All GPs consented to participation in the
trial.

Population
We planned to include GP practices in the Netherlands that used the electronic health
record (EHR) for which we developed a decision support plugin (an extra piece of soft-
ware providing feedback that is not an integral part of the main system). All patients with
AF (both incident and pre-existing) who had been in contact with their GPs during the
length of the trial were evaluated. The GPs use ICPC-coded diagnoses for both care and
reimbursement, thus we considered a patient to have AF if a coded diagnosis (‘K78’) for
AF was documented [25]. This method was chosen to exclude care avoiders (patients
who actively avoid contact with their GP despite their condition) that could bias results.

76

Effectiveness and usage of a decision support system to improve stroke prevention in general practice

Intervention
We developed a CDSS for a single EHR system. This CDSS was automatically activated us-
ing event-based triggers, and responded to actions of the GP in real time [23]. Our study
ran concurrently with the ICOVE trial that contained 15 decision rules per randomization
group [26]. For each decision rule that was not satisfied, notifications were shown in a
floating notification window. The window could contain multiple notifications if multiple
rules applied tothe patient. Notifications contained a short (1- to 3-word) title (Figure 1).
By clicking on a notification, a popup appeared that contained: background information,
an actionable recommendation and two response buttons that allowed the GP to either
accept or decline the advice (Figure 2). Accepting or declining was logged by the plugin
but had no effect on the EHR. Due to the fact that there were other notifications present
in the system, GPs were likely unaware of their allocation. The Expert-AF notification
was positioned at the top of the notification window. For intervention group 2 there
was a small free text box that required the GP to enter a reason for deviating from the
recommendation. The plugin was only active for GPs.

Atrial fibrillation Clinical Rules Chapter 6
Stroke prevention in AF
Yearly medication revision

Risk of falling

Figure 1. The notification window in its expanded state, showing three notifications, with the AF no-
tification at the top of the list that includes other notifications originating from other clinical rules.

77

Chapter 6

Stroke prevention in patients with atrial fibrillation

Recommendation
(517) Start oral anticoagulation.

Background
The patient is not receiving antithrombotic treatment for the prevention of stroke. The patient has a CHA2DS2-VAsc score
of more than 1: Antithrombotic treatment is recommended.
Reasons for scoring more than 1 point on CHA2DS2-VAsc:
Patient characteristics:
Age: 95 years old
Gender: Male

Risk factors:
Hypertension

Response to recommendation

I will follow the recommendation I will ignore the recommendation

Source
This recommendation is based on the Dutch General Practitioners guideline for Atrial Fibrillation.

Figure 2. The popup after clicking on a notification, containing background information, an action-
able recommendation and response buttons that allowed the GP to either accept (1) or decline (2)
the advice.

Training and support
GPs were trained using three methods: a live demonstration, a PDF manual and a video
demonstration. One live demonstration per GP practice was provided. Additionally, a de-
tailed manual of the system was distributed in PDF format among all participants. Lastly,
GPs were made aware per email of a 10-minute video demonstration of the system avail-
able online. Support was available during office hours via email and phone.

Decision rule
The computerized decision rule used in the Expert-AF system was based on the Revised
Dutch GP guideline for atrial fibrillation, authored by the Dutch College of General Prac-
titioners [27]. This guideline uses the CHA2DS2-VASc score, which consists of 7 variables
to predict risk of stroke [28]. Patients can score from 0 to 9 points. All patients with a
score >1 have an indication for OAC according to the current GP guideline. These same
recommendations are made in the 2012 European Society of Cardiology guideline
on atrial fibrillation [8]. The HAS-BLED bleeding risk score [29] was not yet formally
introduced in the Dutch GP guideline for AF, and thus not included in the decision rule.

78

Effectiveness and usage of a decision support system to improve stroke prevention in general practice Chapter 6

However previous large bleeds, untreated hypertension, kidney failure and thrombotic
disorders were included as contra-indications for VKA.

Baseline data collection
Baseline measurements were performed to determine the proportion of patients with
AF who were treated in accordance with the guideline. Guideline adherence was deter-
mined by automatically evaluating the computerized decision rule on all patients in an
anonymized copy of the GP’s EHR databases.

Data collection during and after the study
Every action that related to the plugin was stored. Anonymized patient files were saved
to confirm the system functioned properly and to determine what triggered the notifica-
tions. Guideline adherence by the participating GPs was evaluated using an anonymized
copy of the GP EHR databases, which was made after the trial was completed. The trial
was conducted from October 1st, 2013 to September 1st, 2014, resulting in 240 active
workdays.

Outcomes
We defined the primary outcome as the difference in the proportion of patients with AF
treated in accordance with the guideline between the intervention and control groups.
A secondary outcome of the study was the reasons GPs provided for deviating from the
guideline and the manner in which they responded to required justification (intervention
2).

Analysis
A power analysis accounting for clustering resulted in a required sample size of 500
patients for an effective sample size of 300 [23].
A chi-square test was used to compare between-group differences at baseline and dif-
ferences in adherence rates post-study. Additionally, a random effects model was used
with the GP practice as random effect to correct for clustering at the practice level and
baseline adherence.
Intervention 2 was an extension of intervention 1; the only difference between the two
was the addition of a free text box that required a reason for ignoring recommendation.
For the quantitative analyses, these groups were merged to increase statistical power.
Analyses were performed using R [24].

79

Chapter 6

R es u lts

For the baseline analyses 731 patients with AF were included (table 1). In total, 781
patients with AF were in contact with the GP practice during the trial and included in the
analysis.

Table 1. Population characteristics and guideline adherence before and after the study

Baseline control Baseline Post-study Post-study
intervention control intervention

N 235 496 259 522

Mean age 74.61 (13.63) 72.13 (12.46) P=0.08 73.73 (14.7) 72.79 (12.61) P=0.52
(SD) P=0.13

Mean 3.06 (1.8) 3.00 (1.72) P=0.52 2.27 (0.82) 2.25 (0.86) P=0.02
CHA2DS2-VAsc (SD) P=0.05 P=0.23
0.8 0.78 0.76 0.76
CHA2DS2-VAsc > 1 0.4 0.48 0.51 0.60

on OAC

Adherence 0.42 0.5 P=0.04 0.5 0.55

Usage
Table 2 contains usage statistics. The notification was shown 3848 times, and clicked
188 times to open the pop-up with the information and advice (5%). Usage over time
declined, with the most activity in the first month, and a steady decline of usage over
the following 8 months (figure 3). In total, GPs actively responded to 76 notifications by
either dismissing them (N=44, 58%) or indicating they would follow the advice (N=32,
42%). Some GPs did not click on the notification at all, while others clicked nearly all of
them (figure 4).

Table 2. Usage of the system during the 9 months 3848
Notifications shown 188
Notifications clicked 76
Responses to recommendations 0.05
Clicks per notification 0.07
Clicks per consultation 240
Days usage 16
Avg. # notifications per workday 0.45
Avg. # notifications per workday per GP

In the second intervention group, GPs clicked on twenty-two notifications and accepted
two. Five notifications were declined. Four reasons for declining were related to the plu-
gin not being able to detect Warfarin as active medication or not related to the guideline,

80

250

26­6­E2f0f1e6ctiveness and usage of a decision supporPtHsPy cslatses mfor tGoooigmle pCrhoarvt Teoosltsroke prevention in general practice

Times rules triggered 50 100 150

6.0 Days since start of trial
log10(times triggered)

User activity

Notications clicked Responses Notification window expanded/10

4.5

75
User rules act / times triggered

0.20 times opened / TT
3.0

50

0.15
1.5

250.10
0.0
expeIrCtAOVEIC01OVEIC02OVEIC03OVEIC04OVEIC05OVEIC06OVEIC07OVEIC08OVEIC09OVEIC10OVEIC11OVEIC12OVEIC13OVEIC15OVEIC16OVEIC17OVEIC18OVEIC19OVEIC20OVEIC21OVEIC22OVEIC23OVEIC24OVEIC26OVEIC27OVEIC28OVE29

0.05 50 100 150

Figure 3. Usage of the CDDS plugin over time. Blue dots indicate clicked notifications, red dots in-

dicate responses to recommendations (clicking accept or decline), orange dots indicate the number
onfo0I.ttC0iiO0fmVicEIeC1a9OstViEoItC0h3OneVEw0Ge8axpPseIrCthoAOVopEIveC20OenVreEIeC2d1Od.VEIoC24OvVeEICr06OtVhEIC1e5OVnEIC0o1OtViEIfC0i4OcVaEItC2i2OoVnEIC18OwVEIiC1n6OdVEoIC10OwVEICw11OiVtEIhC05OVthEIC1e2OVirEIC2m7OVoEIC2u8OVsEIeC09OrVeEIC2g9OaVErICd07OlVeEICs23OsVEoIC1f3OVwEIC1h7OVeEItC0h2OVeEr26the

GP activity

0.12 responses / notifications (per GP)

0.09 Chapter 6
http:0//l.o0c6alhost/EAF%20charts/
1/3

0.03

0.00
769 3 4 44 7 31 1474358 38 443 745 24 47 11 408122163122879 5 7651715 488 764 1 30 763 102 2 8 45 52 614365

Figure 4. User activity
ANvoeticrea: gUenkunoswang: Seki(prpeinsgp noumnesriec ske/y 1n ion tUifnikcnaotwino onn) lipnee 0r  general practitioner

Notice: Unknown: Skipping numeric key 2 in Unknown on line 0 

Notice: Unknown: Skipping numeric key 3 in Unknown on line 0 

thus only one reason for guideline non-adherence was presented, which was insufficient

Notice: Unknown: Skipping numeric key 4 in Unknown on line 0 

fNoorticfeu: Urtnkhneowrna: Snkiappliyngs nisum. eric key 5 in Unknown on line 0 

htEtp:f//floecaclhotsit/vEAeF%n2e0cshasrts/ 2/3

Guideline adherence, defined as the percentage of patients treated in accordance with

the guideline, differed between the control and intervention groups at baseline (Con-

81

Chapter 6

trol: 42%, Intervention: 50% chisq P = 0.04). At the end of the study, both groups had
improved, by 8% and 5% respectively. There was no statistically significant difference
between groups (Control: 50%, Intervention: 55% chisq P = 0.23) (table 1). Clustered
analysis revealed similar results (P=0.21); correcting for baseline adherence did not alter
these results (P=0.25).

D isc u ssion

Summary
We evaluated the effectiveness of a real-time CDSS in general practice, specifically for
increasing adherence to antithrombotic guidelines for atrial fibrillation. We did not find
a significant difference in guideline adherence between the intervention and control
groups.

Strengths and limitations
Strengths of this study included the decision support system feedback characteristics
that built on evidence pertaining to effectiveness of decision support systems [16, 17].
We implemented a real-time CDSS that supported the GP during their contacts with
patients by showing alerts in a non-obtrusive way. Our trial implemented many of the
features that we should expect from decision support in the future. We reduced contami-
nation by using clustered randomization. Lastly we studied a topic (prevention of stroke),
that is considered highly relevant by the participating GPs.
However, this study was hampered in several ways. First, the vendor of the GP system
that implemented the CDSS plugin stopped supporting it, thereby preventing us from
enrolling more GPs in our trial. As initially, more GPs were included in the intervention
arm, this lead to asymmetric randomization groups, reducing potential power of the
study. Nonetheless the calculated sample size (n=500) was reached, with more than 250
inclusions in each group post study. Lastly, the new guideline for the management of
AF by the Dutch College of General Practitioners was introduced during the start of the
trial, which might have attracted some extra attention, resulting in increased adherence
in both groups. We would have preferred to develop a generic system that worked on
any platform but were due to funding and organizational restrictions we were limited to
working with a single vendor.

82

Effectiveness and usage of a decision support system to improve stroke prevention in general practice Chapter 6

Comparison with existing literature

Guideline adherence
Guideline adherence was poor both before and after the study, which is in line with other
studies that specifically investigated this topic [9, 11, 30]. Guideline non-adherence
consisted of both under- and overtreatment, which was also present in other studies
investigating OAC use [9, 11, 31]. Nonetheless, the increased OAC use in both groups
could indicate awareness of OAC underuse is increasing.

Effectiveness
We attribute the lack of effectiveness mainly to low usage (measured by interaction with
the notifications), which had many reasons. These will be discussed in detail in a separate
qualitative system evaluation. Briefly, barriers were mainly related to lack of time, too
many alert notifications and limitations of the system’s functionality. Participants in ac-
knowledged the potential of CDSSs for the future of healthcare, but implementing these
systems in daily practice for multiple domains remains challenging. Alert prioritization,
user customization, tight EHR integration and strict selection of alerts might improve
CDSS effectiveness. Eccles et. al. found similar low usage rates and attributed the limited
effectiveness of their system to this lack of use [32]. Lugtenberg et. al. recently published
results on a process evaluation of ‘NHGDoc’, an EHR-integrated decision support system
that was implemented in 65% of GP practices in the Netherlands. Usage of the system
was as low as 0.24%, and it also did not lead to changes in outcomes [33]. A recent
trial by Cook et. al. that attempted to improve prescription in AF by way of a CDSS also
failed to show effectiveness of their intervention [34]. While Cook et al did not report on
usage, low usage may have also hampered their study. All four studies (including ours)
attempted to provide the user with a non-obtrusive system, integrated in daily workflow.
It is still unclear what can be considered ‘optimal timing’ in the busy daily practice. Its
non-obtrusive nature may have been disadvantageous in a setting where patients, staff,
e-mails and other matters compete for the GP’s attention. Recent studies have indicated
that systems which force the user to respond to the alert are more effective [18], al-
though clearly only a limited number of alerts can be presented in this way. Furthermore,
physicians do not always agree with the guidelines these systems are based on,3 or have
(valid) reasons for non-adherence [35-37]. The number of declined recommendations
(N=44, 54%) we found in our study suggests that this is at least part of the problem.
‘In terms of the recently-described “Two Stream Model” [38], the lack of use can be at-
tributed mainly to data quality problems and sub-optimal presentation.

83

Chapter 6

Implications for Research
Guideline adherence in both the intervention and control groups increased during this
trial investigating the effectiveness of a CDSS for stroke prevention. Our study could
not demonstrate the effectiveness of our intervention, which was likely due to lack of
use. The implementation of multi-domain CDSSs in clinical practice is challenging and
future studies should investigate further improvements to facilitate effectiveness in a
real-world setting. Stroke prevention in patients with AF is a field where much can be
gained by following guidelines, and we therefore urge other researchers to dedicate their
efforts to investigate how to effectively increase system use and guideline adherence.

84

Effectiveness and usage of a decision support system to improve stroke prevention in general practice

R eferences

1. Chugh SS, Havmoeller R, Narayanan K, Singh D, Rienstra M, Benjamin EJ, et al. Worldwide Chapter 6
Epidemiology of Atrial Fibrillation A Global Burden of Disease 2010 Study. Circulation.
2. 2014;1​ 29(8):​837-47. doi: 10.1161/Circulationaha.113.005119. PubMed PMID: WOS:​
3. 000331776900008.
Zoni-Berisso M, Lercari F, Carazza T, Domenicucci S. Epidemiology of atrial fibrillation: Euro-
4. pean perspective. Clin Epidemiol. 2014;6​ :2​ 13-20. doi: 10.2147/CLEP.S47385. PubMed PMID:
5. 24966695; PubMed Central PMCID: PMC4064952.
6. Lip GY, Brechin CM, Lane DA. The global burden of atrial fibrillation and stroke: a systematic
review of the epidemiology of atrial fibrillation in regions outside North America and Europe.
7. Chest. 2012;1​ 42(6):1​ 489-98. Epub 2012/03/31. doi: 10.1378/chest.11-2888. PubMed
PMID: 22459778.
8. Wolf PA, Abbott RD, Kannel WB. Atrial fibrillation as an independent risk factor for stroke: the
9. Framingham Study. Stroke. 1991;2​ 2(8):​983-8. doi: 10.1161/01.str.22.8.983.
Hart RG, Pearce LA, Aguilar MI. Meta-analysis: Antithrombotic therapy to prevent stroke in
10. patients who have nonvalvular atrial fibrillation. Ann Intern Med. 2007;1​ 46(12):​857-67.
11. PubMed PMID: WOS:​000247347100004.
10. Ruff CT, Giugliano RP, Braunwald E, Hoffman EB, Deenadayalu N, Ezekowitz MD, et al. Com-
12. parison of the efficacy and safety of new oral anticoagulants with warfarin in patients with
atrial fibrillation: a meta-analysis of randomised trials. Lancet. 2014;​383(9921):9​ 55-62. doi:
10.1016/S0140-6736(13)62343-0. PubMed PMID: WOS:0​ 00333520700026.
Nieuwlaat R, Olsson SB, Lip GY, Camm AJ, Breithardt G, Capucci A, et al. Guideline-adherent
antithrombotic treatment is associated with improved outcomes compared with undertreat-
ment in high-risk patients with atrial fibrillation. The Euro Heart Survey on Atrial Fibrillation.
Am Heart J. 2007;​153(6):1​ 006-12. Epub 2007/06/02. doi: 10.1016/j.ahj.2007.03.008.
PubMed PMID: 17540203.
Camm AJ, Lip GYH, De Caterina R, Savelieva I, Atar D, Hohnloser SH, et al. 2012 focused
update of the ESC Guidelines for the management of atrial fibrillation. Eur Heart J. 2012;​
33(21):2​ 719-47. doi: 10.1093/eurheartj/ehs253. PubMed PMID: WOS:​000310628700020.
Dinh T, Nieuwlaat R, Tieleman RG, Buller HR, van Charante NA, Prins MH, et al. Antithrombotic
drug prescription in atrial fibrillation and its rationale among general practitioners, internists
and cardiologists in The Netherlands--The EXAMINE-AF study. A questionnaire survey. Int J
Clin Pract. 2007;​61(1):​24-31. Epub 2007/01/19. doi: 10.1111/j.1742-1241.2006.01241.x.
PubMed PMID: 17229177.
Lip GYH, Lane DA. Stroke Prevention in Atrial Fibrillation A Systematic Review. Jama-Journal
of the American Medical Association. 2015;3​ 13(19):​1950-62. doi: 10.1001/jama.2015.4369.
PubMed PMID: WOS:​000354691200016.
Arts DL, Visscher S, Opstelten W, Korevaar JC, Abu-Hanna A, van Weert HCPM. Frequency
and Risk Factors for Under- and Over-Treatment in Stroke Prevention for Patients with Non-
Valvular Atrial Fibrillation in General Practice. PLoS One. 2013;8​ (7). doi: ARTN e67806
1371/journal.pone.0067806. PubMed PMID: WOS:0​ 00321425300031.
Goud R, van Engen-Verheul M, de Keizer NF, Bal R, Hasman A, Hellemans IM, et al. The effect of
computerized decision support on barriers to guideline implementation: a qualitative study
in outpatient cardiac rehabilitation. Int J Med Inform. 2010;​79(6):4​ 30-7. Epub 2010/04/10.
doi: 10.1016/j.ijmedinf.2010.03.001. PubMed PMID: 20378396.

85

Chapter 6

13. Wilson FP, Shashaty M, Testani J, Aqeel I, Borovskiy Y, Ellenberg SS, et al. Automated, elec-
tronic alerts for acute kidney injury: a single-blind, parallel-group, randomised controlled
14. trial. Lancet. 2015;3​ 85(9981):1​ 966-74. doi: 10.1016/S0140-6736(15)60266-5. PubMed
PMID: WOS:0​ 00354541700030.
15. Bright TJ, Wong A, Dhurjati R, Bristow E, Bastian L, Coeytaux RR, et al. Effect of clinical
decision-support systems: a systematic review. Ann Intern Med. 2012;​157(1):​29-43.
16. Epub 2012/07/04. doi: 10.7326/0003-4819-157-1-201207030-00450. PubMed PMID:
22751758.
17. Moja L, Kwag KH, Lytras T, Bertizzolo L, Brandt L, Pecoraro V, et al. Effectiveness of Computer-
18. ized Decision Support Systems Linked to Electronic Health Records: A Systematic Review and
19. Meta-Analysis. Am J Public Health. 2014;​104(12):​E12-E22. doi: 10.2105/Ajph.2014.302164.
20. PubMed PMID: WOS:​000347210500011.
Kawamoto K, Houlihan CA, Balas EA, Lobach DF. Improving clinical practice using clinical
21. decision support systems: a systematic review of trials to identify features critical to success.
BMJ. 2005;3​ 30(7494):​765. Epub 2005/03/16. doi: 10.1136/bmj.38398.500764.8F. PubMed
22. PMID: 15767266; PubMed Central PMCID: PMC555881.
23. Sittig DF, Wright A, Osheroff JA, Middleton B, Teich JM, Ash JS, et al. Grand challenges in clini-
10. cal decision support. Journal of Biomedical Informatics. 2008;​41(2):3​ 87-92. doi: 10.1016/j.
24. jbi.2007.09.003. PubMed PMID: WOS:0​ 00255360000015.
25. Roshanov PS, Fernandes N, Wilczynski JM, Hemens BJ, You JJ, Handler SM, et al. Features
of effective computerised clinical decision support systems: meta-regression of 162 ran-
26. domised trials. BMJ. 2013;​346:f​ 657. doi: 10.1136/bmj.f657. PubMed PMID: 23412440.
Medlock S, Wyatt JC, Patel VL, Shortliffe EH, Abu-Hanna A. Modeling information flows in
clinical decision support: key insights for enhancing system effectiveness. J Am Med Inform
Assoc. 2016. doi: 10.1093/jamia/ocv177. PubMed PMID: 26911809.
Kawamoto K, Lobach DF, Willard HF, Ginsburg GS. A national clinical decision support infra-
structure to enable the widespread and consistent practice of genomic and personalized
medicine. BMC Med Inform Decis Mak. 2009;​9:1​ 7. Epub 2009/03/25. doi: 10.1186/1472-
6947-9-17. PubMed PMID: 19309514; PubMed Central PMCID: PMC2666673.
McCoy AB, Waitman LR, Lewis JB, Wright JA, Choma DP, Miller RA, et al. A framework for evalu-
ating the appropriateness of clinical decision support alerts and responses. J Am Med Inform
Assoc. 2012;​19(3):3​ 46-52. doi: 10.1136/amiajnl-2011-000185. PubMed PMID: 21849334;
PubMed Central PMCID: PMCPMC3341775.
Ash JS, Sittig DF, Campbell EM, Guappone KP, Dykstra RH. Some unintended consequences
of clinical decision support systems. AMIA Annu Symp Proc. 2007:​26-30. PubMed PMID:
18693791; PubMed Central PMCID: PMCPMC2813668.
Arts DL, Abu-Hanna A, Buller HR, Peters RJG, Eslami S, van Weert HCPM. Improving stroke
prevention in patients with atrial fibrillation. Trials. 2013;1​ 4. doi: Artn 193
1186/1745-6215-14-193. PubMed PMID: WOS:​000321341100001.
Team RDC. R: A Language and Environment for Statistical Computing. 2011.
An international glossary for primary care. Report of the Classification Committee of the
World Organization of National Colleges, Academies and Academic Associations of General
Practitioners/Family Physicians (WONCA). J Fam Pract. 1981;​13(5):​671-81. Epub 1981/10/01.
PubMed PMID: 7024465.
Eslami S, Askari M, Medlock S, Arts DL, Wyatt JC, van Weert HCPM, et al. From assessment to
improvement of elderly care in general practice using decision support to increase adher-

86

Effectiveness and usage of a decision support system to improve stroke prevention in general practice

10. ence to ACOVE quality indicators: study protocol for randomized control trial. Trials. 2014;1​ 5. Chapter 6
27. doi: Artn 81
56. 1186/1745-6215-15-81. PubMed PMID: WOS:​000333474700001.
28. fibrillation DCoGPGDGfA. Guideline Atrial fibrillation (second partial revision). Huisarts &
Wetenschap. 2013;
29. (8):3​ 92‑401.
Lip GY, Nieuwlaat R, Pisters R, Lane DA, Crijns HJ. Refining clinical risk stratification for
30. predicting stroke and thromboembolism in atrial fibrillation using a novel risk factor-based
31. approach: the euro heart survey on atrial fibrillation. Chest. 2010;​137(2):​263-72. Epub
32. 2009/09/19. doi: 10.1378/chest.09-1584. PubMed PMID: 19762550.
Pisters R, Lane DA, Nieuwlaat R, de Vos CB, Crijns HJ, Lip GY. A novel user-friendly score (HAS-
33. BLED) to assess 1-year risk of major bleeding in patients with atrial fibrillation: the Euro Heart
Survey. Chest. 2010;1​ 38(5):1​ 093-100. Epub 2010/03/20. doi: 10.1378/chest.10-0134.
34. PubMed PMID: 20299623.
35. Ogilvie IM, Newton N, Welner SA, Cowell W, Lip GYH. Underuse of Oral Anticoagulants in
36. Atrial Fibrillation: A Systematic Review. Am J Med. 2010;​123(7):​638-U86. doi: 10.1016/j.
amjmed.2009.11.025. PubMed PMID: WOS:0​ 00279067100015.
37. Palomaki A, Mustonen P, Hartikainen JE, Nuotio I, Kiviniemi T, Ylitalo A, et al. Underuse of
38. anticoagulation in stroke patients with atrial fibrillation - the FibStroke Study. Eur J Neurol.
2015. doi: 10.1111/ene.12820. PubMed PMID: 26263442.
Eccles M, McColl E, Steen N, Rousseau N, Grimshaw J, Parkin D, et al. Effect of computerised
evidence based guidelines on management of asthma and angina in adults in primary care:
cluster randomised controlled trial. Br Med J. 2002;​325(7370):​941-4. doi: DOI 10.1136/
bmj.325.7370.941. PubMed PMID: WOS:0​ 00178986300018.
Lugtenberg M, Weenink JW, van der Weijden T, Westert GP, Kool RB. Implementation of
multiple-domain covering computerized decision support systems in primary care: a focus
group study on perceived barriers. BMC Med Inform Decis Mak. 2015;1​ 5:​82. doi: 10.1186/
s12911-015-0205-z. PubMed PMID: 26459233; PubMed Central PMCID: PMCPMC4603732.
Cook DA, Enders F, Caraballo PJ, Nishimura RA, Lloyd FJ. An automated clinical alert system for
newly-diagnosed atrial fibrillation. PLoS One. 2015;​10(4):e​ 0122153. doi: 10.1371/journal.
pone.0122153. PubMed PMID: 25849969; PubMed Central PMCID: PMCPMC4388495.
Cabana, Rand CS, Powe NR, Wu AW, Wilson MH. Why Don ’ t Physicians Follow guidlines, A
Framework for Improvement. October. 1999;​0718.
Lugtenberg M, Zegers-van Schaick J, Westert G, Burgers J. Why don’t physicians adhere to
guideline recommendations in practice? An analysis of barriers among Dutch general prac-
titioners. Implementation Science. 2009;​4(1):5​ 4. PubMed PMID: doi:1​ 0.1186/1748-5908-
4‑54.
Arts DL, Voncken AG, Medlock S, Abu-Hanna A, van Weert HC. Reasons for intentional guide-
line non-adherence: A systematic review. Int J Med Inform. 2016;​89:5​ 5-62. doi: 10.1016/j.
ijmedinf.2016.02.009. PubMed PMID: 26980359.
Medlock S, Wyatt JC, Patel VL, Shortliffe EH, Abu-Hanna A. Modeling information flows in
clinical decision support: key insights for enhancing system effectiveness. J Am Med Inform
Assoc. 2016.

87



Chapter 7

Acceptance and barriers pertaining to a
general practice decision support system
for multiple clinical conditions: a mixed

methods evaluation

D.L. Arts, S.K. Medlock, J. C. Wyatt, H.C.P.M. van Weert, A. Abu-Hanna

Chapter 7

A bstract

Background
Many studies have investigated the use of clinical decision support systems as a means
to improve care, but have thus far failed to show significant effects on patient-related
outcomes. We developed a clinical decision support system that attempted to address
issues that were identified in these studies. The system was implemented in Dutch
general practice and was designed to be both unobtrusive and to respond in real time.
Despite our efforts, usage of the system was low. In the current study we perform a mixed
methods evaluation to identify remediable barriers which led to disappointing usage
rates for our system.

Methods
A mixed methods evaluation employing an online questionnaire and focus group. The
focus group was organized to clarify free text comments and receive more detailed
feedback from general practitioners. Topics consisted of items based on results from the
survey and additional open questions.

Results
The response rate for the questionnaire was 94%. Results from the questionnaire and
focus group can be summarized as follows: The system was perceived as interruptive,
despite its design. Participants felt that there were too many recommendations and that
the relevance of the recommendations varied. Demographic based recommendations
(e.g. age) were often irrelevant, while specific risk-based recommendations (e.g. diagno-
sis) were more relevant. The other main barrier to use was lack of time during the patient
visit.

Conclusion
These results are likely to be useful to other researchers who are attempting to address
the problems of interruption and alert fatigue in decision support.

90

Acceptance and barriers pertaining to a decision support system for multiple clinical conditions Chapter 7

B ackgro u nd

Clinical decision support systems (CDSS) are computerized tools that assist in making
clinical decisions.[1] In recent years, many studies have investigated the use of CDSS as
a means to improve care.[2-5] The interest in CDSS as a strategy to implement guidelines
is not surprising given the ever-increasing number of clinical practice guidelines. For
example, the Dutch College of General Practitioners (NHG) currently offers more than
100 guidelines to its members. It is not unlikely that, for a given patient, more than 15
guidelines might apply. It is this complexity that creates the need for CDSS, especially in
older patients with many comorbidities where guidelines often overlap or even contra-
dict each other.[6]
Despite the apparent advantages of computer-based CDSSs, most systems have thus far
failed to show significant effects on patient-related outcomes.[2-4] A recent review did
however find moderate improvements in morbidity outcomes.[5] Reviews that studied
CDSSs in long term conditions such as asthma, diabetes and hypertension did not show
an impact on patient outcomes.[7-9] Therefore evidence for the cost-effectiveness of
CDSSs is lacking.[2] A limitation reported in all reviews investigating the effects of CDSSs
is the fact that studies are often of poor methodological quality. Another limitation is that
many studies focus on systems that provide support for a single disease, while future
systems will be required to support multiple guidelines and the care of patients with
multiple diseases.
Many reasons for the low usage and/or effectiveness of CDSSs have been identified.
These include: lack of usability, lack of integration with host systems, lack of time to
effectuate advice, inapplicability to the patient, lack of integration with current workflow,
and alert fatigue.[10, 11] These reasons partly explain the limited effectiveness of CDSSs,
but often we do not know why apparently high-quality systems go unused or are inef-
fective.[12]
We attempted to address many of these issues in a CDSS that was developed for the
current project. This system was implemented in general practice and was designed
to be both unobtrusive and to respond in real time to the user’s actions in the record
system, drawing on up to 15 validated clinical decision rules.[13] For instance, when a
general practitioner (GP) added a diagnosis of atrial fibrillation, the system would be trig-
gered and evaluate antithrombotic treatment for stroke prevention. Thus the GP would
receive immediate feedback and could respond by modifying the prescription, unlike
most systems where the user receives feedback only after the order is completed and
must backtrack to follow the advice. Our hypothesis was that this pre-emptive mode of
operation would increase user adherence to system advice, as the GP would not have to
modify already prescribed medication.

91

Chapter 7

Despite the features mentioned above, which attempted to address the problems re-
ported in many CDSS studies [10, 11], we found low overall usage, which declined over
time: The CDSS generated an average of 15 notifications per working day for each GP in
the field test. However, GPs only clicked on a total of 4119 of the 126158 notifications,
a click rate of 3%. The effectiveness of the system was limited and differed greatly be-
tween decision rules. Statistical evaluation of the usage and effectiveness of the system
will be reported elsewhere; however, that analysis gives no insights into the reasons for
the GPs’ limited usage of the system, nor how to improve it in future. Thus, the objective
of the current study was to perform a qualitative investigation as a follow up to our
CDSS clinical trial, using a survey and focus group evaluation, with the primary goal of
identifying remediable barriers which led to disappointing usage rates for our system.

M ethods

Clinical decision support system
The trial protocol for our randomized controlled trial is described elsewhere.[14, 15]
Briefly, the system included two guideline domains, one relating to care of older adults
[13] and the other to anticoagulant management in atrial fibrillation.[16] All these
guidelines were implemented as clinical decision rules in a single rule-based system
and coupled with coded data in the GP’s electronic patient record system. The rules to be
included in the system were based on the Dutch version of ACOVE and selected with in-
put from target users by way of a survey. The final rule set consisted of 30 decision rules
relating to the two domains covering diagnoses such as atrial fibrillation (AF), diabetes,
hypertension and medication prescriptions.
User notifications about CDSS recommendations were shown in a floating window that
could contain up to 15 items. Each item contained a short (1 to 3 word) description of
the corresponding recommendation (figure 1). It was not uncommon for there to be more
than 5 recommendations for an elderly patient (median: 4). The notification window was
collapsed by default, and in this state only two letters of each notification item could be
seen. The window could be expanded temporarily by moving the mouse cursor over the
window (mouse over). The user could drag the window to either the left or the right side
of the screen. On clicking a notification item, a window appeared containing information
about the recommendation: background information, the recommendation itself, and
buttons to allow the GP to either accept or ignore the advice (figure 2). All activity in the
system was logged, this included mouse movements, opened recommendation windows
and responses to recommendations.

92

Acceptance and barriers pertaining to a decision support system for multiple clinical conditions

Atrial fibrillation Clinical Rules
Stroke prevention in AF
Yearly medication revision

Risk of falling

Figure 1. The notification window in its expanded state, showing three notification items.

Stroke prevention in patients with atrial fibrillation Chapter 7

Recommendation
(517) Start oral anticoagulation.

Background
The patient is not receiving antithrombotic treatment for the prevention of stroke. The patient has a CHA2DS2-VAsc score
of more than 1: Antithrombotic treatment is recommended.
Reasons for scoring more than 1 point on CHA2DS2-VAsc:
Patient characteristics:
Age: 95 years old
Gender: Male

Risk factors:
Hypertension

Response to recommendation

I will follow the recommendation I will ignore the recommendation

Source
This recommendation is based on the Dutch General Practitioners guideline for Atrial Fibrillation.

Figure 2. The window after clicking on a notification item, containing background information, an
actionable recommendation and response buttons to allow the GP to indicate whether they accept
(1) or decline (2) the advice (3) close the window (no action).

93

Chapter 7

Data collection
Figure 3 shows an overview of the activities related to the development of the question-
naire and focus group topic guides.

UTAUT based 2 GPs verified content 1 GP piloted
questionnaire validity questionnaire

Questionnaire 30 completed Emergent themes &
distributed to 34 GPs questionnaires noteworthy results

Focus group topic Focus group with 8 Analyses & coding of
guide GPs focus group transcript

Figure 3. Overview of activities related to data collection

Questionnaire
A questionnaire was developed to investigate user attitudes about the CDSS. Of the
40 survey questions, 28 represented operationalisations of concepts from the unified
theory of acceptance and use of technology (UTAUT).[17] The question phrasing was
based on known barriers and facilitators for CDSS use.[10, 11] An additional 6 questions
were added to explore new areas of interest identified in the pre-implementation survey
[18], and 6 more were added to explore user views on specific system features, such as
the real-time feedback. A validation focus group with two GPs was performed to review
content validity which was then piloted on one GP. The questionnaire contained three
free text questions. The full (translated) questionnaire can be found in appendix 2.

Questionnaire distribution
The online questionnaire was distributed by email to all GPs enrolled in the trial. To
maximise response rates, GPs received a small reward (~12 euros) for completing the
questionnaire, and an additional similar reward if an overall response rate of 85% was
reached. E-mail reminders were sent every two weeks for a maximum of four times, after
which the remaining non-responders were approached by phone to remind them about
our questionnaire. Participants were required to enter their name so we could associ-
ate their age and gender with the survey results using demographics data from the GP
centres.

94

Acceptance and barriers pertaining to a decision support system for multiple clinical conditions Chapter 7

Questionnaire analyses
Two authors (DA, SM) independently reviewed the survey free text comments and ex-
tracted themes, representing clusters of topics, by induction.[19] Quantitative survey
results were also independently reviewed by two researchers, who selected results from
the survey which might be clarified by discussing them in the focus group that followed.
These selections were compared and merged during a consensus meeting between DA
and SM, and used as input for the focus group.

Focus group
To further clarify the free text comments and receive more detailed feedback from GPs, a
focus group was organized. We created a topic guide structured according to the Informa-
tion Systems Success Model by DeLone and McLean, which consists of six dimensions of
information systems success.[20] The topic guide consisted of items based on results from
the survey and additional open questions aimed at encouraging free discussion about the
system. The 8 invited GPs were sampled purposively, based on their answers to the survey.
All GPs had used the system more than once, but we specifically chose GPs with either a
lot or a little self-reported experience using the system. Furthermore, we prioritised GPs as
focus group participants if they had given more detailed feedback on the free text survey
questions. The focus group started shortly after normal clinic hours, and food and drinks
were provided. The focus group took 90 minutes, and was recorded using video and audio.
Recordings were transcribed verbatim, and the transcriptions coded independently by
two researchers using the previously-identified topics as an initial coding scheme (from
the Information Systems Success model, topics from the survey, and known issues with
the system) and coded using MaxQDA 12.[21] Comments that were relevant to the central
questions (why the GPs did or did not use the system, and suggestions for improvement)
but did not fit into any of the previously identified categories were also noted, then
classified into emergent themes.

R es u lts

Questionnaire

Respondents and response rates
The response rate was 94% with 32 responses out of 34 eligible respondents. Four ad-
ditional respondents were invited but were unable to respond due to switching jobs,
retirement or maternity leave. Two surveys were not completed fully, resulting in a total
of 30 surveys that were included in the analyses. Respondents were on average 51 years
old (SD: 9.5 years) and 63% were female.

95

Chapter 7

Self-reported usage
Most participants (23, 77%) used the system more than once, but 18 (60%) stopped
using the system during the trial. Only 3 (10%) users reported having had more than
50 interactions with the system. Figures 4 and 5 contain additional statistics about
self-reported usage of the system. We excluded survey results from 7 participants who
reported not using the system more than once from further analyses, leaving 23 out of
30 surveys to be analysed.

I stopped using 60%
the system

I used the system 77%
more than once

0.0 0.2 0.4 0.6 0.8 1.0

Figure 4. Self-reported use of the system
Percentage of users that answered yes to the listed questions.

50+ 10%

30−50 23%

10−30 30%

< 10 37%

0.0 0.2 0.4 0.6 0.8 1.0

Figure 5. Self-reported usage rates of the system
Answers to the question: “How often did you use the system, e.g. clicked a notification, accepted a rec-
ommendation etc.?”
96

Acceptance and barriers pertaining to a decision support system for multiple clinical conditions

Use of instruction materials
The participants indicated that 70% had read the manual, 74% attended the live demo,
and 39% watched the instruction video.

Quantitative survey results
The two researchers independently selected the same four survey results as possible
input for further discussion and each selected one additional result. Upon discussion it
was decided to include both of these results. The selected results are listed in Figure 6.

The system reminded me about tasks that I
would have otherwise forgotten

I could do my job quicker as a result of the
system

Handling the recommendations shown by the
system, did not take too much time

englishThe system did not interrupt my usual way of
working
Chapter 7
I think the use of clinical decision support
on my computer is a good idea

I think the number of recommendations
generated by the system was approriate

20 10 0 10 20
Disagree Count Agree Comp. agree
Comp. disagree
Neutral
Figure 6. Selected questions and Likert scales.

Survey free text comments
There were four free text fields in the survey. Both researchers independently identified
the same emergent themes in these comments.
The first theme that emerged in response to the question “What was the main advantage
of using the system?” was that the system reminded and triggered GPs about things they
would have or had forgotten. E.g. “Being notified of medical issues that sometimes fade
into the background”. A second theme was the way the system encouraged the GP to
systematically analyse the patient, e.g. “More - and more structural - attention for several
issues that arise often in the elderly”.

97

Chapter 7

Answers to the second free text question “What change would you want to make to
the system” often related to the fact that the recommendation window floated always
on top, i.e. would float over every other window the GP was using. Another often-made
remark pertained to the long list of notifications for some patients, and the fact that
GPs had no control over which notifications were shown. Other remarks included “Some
recommendations overlapped with current protocols for diabetes and hypertension” and
“The recommendations did not relate to a patient’s reason for visiting”.

Focus group
Both coders identified the same topics as frequently-recurring. Each dimension of the In-
formation Systems Success model is listed below with related topics and quotes for each
topic to illustrate responses. One emergent theme was identified: “Requested features.”

System Quality
The most frequently mentioned dimension (36% of coded phrases) was System Quality,
particularly the quality of the interface. The primary issue was that the floating window
was always on top and thus blocked access to other applications. This frustrated many
participants. The expanding notification window was experienced by some users as
a “popup”. This is interesting as it only expanded on request (mouse over), indicating
that any interruption on screen can register as a “popup”, i.e. something interrupting
regular computer usage. The use of 2 letter abbreviations in the collapsed state wasn’t
understood by all participants. Participants were divided on the utility of asking the user
to document a reason when declining a recommendation. Some felt that the documenta-
tion was helpful in communicating their reasoning with other physicians in their practice
and that feedback could be used to improve the guidelines; others felt it was “defensive
medicine” and did not contribute to patient care. Participants also noted that it was clear
that the CDSS was not fully integrated into the patient record system (e.g. medication
could not be prescribed directly from the CDSS) and that this adversely affected the
utility of the CDSS. Almost all participants felt that better integration with their EHR was
essential, as they now felt they had to do the work twice: once in the CDSS, and then
again in their own EHR.
Regarding our hypothesis about improving system usability, we note that (1) real-time
feedback did not prompt specific discussion in the focus group, and (2) the interface
design that we considered unobtrusive was still experienced by users as interruptive. To
elaborate on the first point; the CDSS was designed to operate in real-time, i.e. using (on-
screen) data that had been newly added to the patient file by the GP. This feature simply
went unnoticed by the users, i.e. GPs did not notice that a new notification item ap-
peared whenever they changed something on screen. Second, the system was designed
to be non-interruptive by presenting a collapsed notification window on the side of the

98