Open access

Knowledge Bases for Clinical Decision Support in Drug Prescribing – Development, Quality Assurance, Management, Integration, Implementation and Evaluation of Clinical Value

Written By

Birgit Eiermann, Pia Bastholm Rahmner, Seher Korkmaz, Carina Landberg, Birgitta Lilja, Tero Shemeikka, Aniko Veg, Björn Wettermark and Lars L Gustafsson

Published: 01 January 2010

DOI: 10.5772/39458

From the Edited Volume

Decision Support Systems

Edited by Chiang S. Jao

Chapter metrics overview

3,556 Chapter Downloads

View Full Metrics

1. Introduction

The access and use of information technology is increasing in all parts of society and in particular in the health care sector in developed and developing countries (Bates & Gawanda, 2003; Lucas, 2008). The integration of health information technology into health care institutions governs the agenda in most countries presently (Lucas, 2008; Gustafsson et al. 2003). The US has recently enacted a $ 19 billion program to promote the use and adoption of health information technology (Blumenthal, 2009) and information systems including electronic health records (EHR). This program is seen as an essential component to improve the health of every American. Challenges discussed span over the whole area of installing electronic health records, supporting and updating the systems, assistance with the interoperability, training the personal, and implementation of the systems as well as medical education (Blumenthal, 2009). Information technology, in particular computerized decision support systems, is also seen in the recent report by the Institute of Medicine in the US as a key way to address the identified great risk of medication errors in American health care institutions (Aspen, 2006).

A recent European report published by the Swedish government analyses health care in 6 European member states. The report describes the impact of health technology on several political goals such as increasing the availability of health care, continuity, empowerment of patients, patient safety and quality of care. It states that in the 6 European member states studied, 100 000 yearly inpatient adverse drug events (ADE´s) could be avoided through usage of computerised physician order entry systems (CPOEs) with clinical decision support (CDS), which would correspond to a yearly saving of 300 million € (Gartner, 2009). This report combined with other studies and reviews (Sjöborg et al, 2007; Kelly et al., 2006) underlines the complexity of integration and implementation, including local conditions, the involvement of stakeholders and adoption and measurements of changes, all of which have to be tackled for a beneficial usage of the technology. The Gartner report envisages that increasing costs within the health care sector will accelerate the efforts to develop new technologies as well as lead to a beneficial usage of existing systems.

Computerised physician order entry systems (CPOEs) are one step towards increased safety in patient safety. They allow physicians and other health care staff to prescribe patient medication directly by using a computer, replacing hand-written orders, and thereby eliminating possible interpretation and transcription faults. Transcription and/or interpretation errors have been shown to cause 11% of all medication errors resulting in adverse drug events in hospitals (Krahenbuhl-Melcher et al., 2007). An additional step to improving patient safety and efficacy in the prescribing process is the integration of clinical decision support systems (CDS systems) within the CPOEs. This allows physicians to retrieve up-to-date medical knowledge of the optimal/recommended management of the diseases and drugs, thereby improving patient care through enhancing compliance with recent guidelines and recommendations. CDS systems deliver their information through knowledge bases (e.g. drug-drug or drug-food interactions, drugs & pregnancy, drugs & lactation, drug dosage according to kidney function and genotype and in risk groups), which are integrated through software algorithms, that will generate alerts, warnings and recommendations during drug prescribing. For optimisation of the effect of CDS systems, they should be integrated into EHRs´ resulting in patient specific recommendations and alerts using patient characteristics available in the EHRs.

Numerous studies have demonstrated positive effects of CDS systems in various settings including hospital or ambulatory care, intensive care units and in pediatric care (Ammenwerth et al. 2008, Eslami et al. 2008, Wolfstadt et al. 2008, van Rosse et al. 2009). Areas of improvement identified include costs, safety, adherence, alerts, user satisfaction and time. Reduction of medication errors have been demonstrated with the introduction of CDS systems as well as reduction of ADE´s. However, further studies are needed which focus directly on patient outcomes rather than the surrogate outcome such as “practitioners’ performance” to further accelerate their introduction (Garg et al., 2006). Many studies pinpoint improvements of the knowledge bases or CDS systems including optimization of the content (Luna et al. 2007), introduction of classification systems to knowledge bases (Böttiger et al., 2009), and tiering alerts through introduction of severity levels (Paterno et al., 2009). Their introduction is likely to further improve CDS systems.

Evidence is growing though that CDS systems might not only lead to improved quality in health care but they can themselves create unintended errors jeopardizing patient safety (Ash et al., 2004). Introduction of CDS systems might cause diminished medical judgement, letting the computer overrule physicians´ own professional knowledge. Additional work tasks might create disturbances in the already burdened physicians work flow resulting in inefficiencies of the systems. Also the complexity of the systems increases the potential in design flaws thereby actually introducing new errors rather than preventing them (Bates et al. 2001). Therefore, the implementation and use of any CDS system should be linked to the establishment of a medical management, maintenance and quality assurance system, which leads to discovering, analysing and foreseeing possible errors.

Being responsible for paying the drug bill Stockholm County Council, the largest health care provider in Sweden, implemented a health care strategy in 1997 including the development of an IT architecture (Sjöborg et al., 2007). The aim was to provide numerous services to the prescribers to ensure safe and effective drug prescribing. Additional initiatives where started, like the formation of drug expert groups providing a Wise Drug List, which contains a list of about 240 first line drugs for common diseases incorporating therapeutic ladders or guidelines. Recommendations from 23 expert groups and 5 local drug and therapeutic committees are used to produce and refine the guidance. On the IT site Stockholm County Council in collaboration with Karolinska Institutet and other academic partners has designed, developed and implemented a prescribing tool (Eliasson et al., 2006) and the content for medical knowledge bases for drug-drug interactions, Sfinx (Swedish Finnish Interaction X-Referencing), drugs & pregnancy, and drugs & breast feeding (Nörby et al., 2006, Böttiger et al., 2009). The knowledge bases are integrated into clinical decision support tools (Janus toolbar described below) or are accessible through the web ( This strategy has been combined with a range of initiatives to promote rational use of drugs as described by Godman et al., (2009). The different knowledge bases and their life cycle from development to evaluation are used as examples for our own experiences in the following parts.

The review is based on more than 10 years experiences from joint efforts to develop, implement and evaluate user friendly and effective decision support systems for drug prescribing in Stockholm. It summarizes state-of-the art knowledge on development, integration, maintenance, implementation and evaluation of knowledge bases and CDS systems used for rational drug prescribing. Consequently, we see this review as a first step in the process of creating robust future models and international standards for the retrieval of medical and pharmacological knowledge, its conversion and organisation into knowledge bases, as well as their integration into CDS systems, their management and evaluation of user satisfaction and treatment outcome.


2. Development of knowledge databases

Why are knowledge bases needed and what advantages do they offer compared to other sources like e.g. the official product SPC (summary of products characteristics) issued as part of the registration of a drug product? One advantage with knowledge bases is the standardisation of information for all drugs containing the same substance. For instance the content of individual SPCs or physician desk references may vary considerably between drugs containing the same substance and with identical drug formulations produced by different pharmaceutical companies. This can cause confusion for the prescribing physician. For example information about drug-drug interactions can be found in the SPC for drug A from provider 1 but is missing in the SPC for the same product from provider 2. Alternatively, the drug-drug interaction between drug A and B can be found in the SPC text for drug A but not for drug B (Bergk et al., 2005). Another example for inconsistencies is the classification for drug and pregnancy alerts for pharmaceutical products. One provider may state, that the drug should be avoided during pregnancy, but another drug company may state, that the drug can be used without any problems. Other examples are variations in dosing information (maximum recommended therapeutic dose) between SPCs´ from different providers or in information published by the US Food and Drug Administration (Seidling et al., 2007). Consequently, knowledge bases should help by providing more consistent information about the substances and drugs related to that substance.

The starting point for development of any knowledge base is the analysis of the perceived needs of the potential users in the health care system (Revere et al., 2007). Likewise it is important to assess the potential of a new knowledge base to improve efficacy and safety in drug prescribing (Gustafsson et al., 2003; Schiff & Rucker, 1998). We believe the formation of user groups should be mandatory to explore the functional and content needs for a knowledge base and decision support system before other activities are undertaken (Eliasson et al., 2006). Consequently, a multidisciplinary group of clinical experts within the medical field the knowledge base should be aimed for (e.g. nephrologists for a database about drug dosage in patients with reduced kidney function) together with drug experts (e.g. clinical pharmacologists specialised on drug dosage, drug-drug interactions or drugs & lactation depending on the knowledge base), future users, experts within existing drug registries and software developers, should be convened to discuss the potential and obstacles for the knowledge base (Ash et al., 2004). Our own experience is that there often is a mismatch between users’ expectations and the clinical and medical research basis or the availability of certain parameters or features within existing registries. For example, the clinical specialists will focus on one specific recommendation for the most common indications for both drugs for a certain drug-drug interaction. However, this recommendation might not fit all patient cases for which this pertinent drug-drug interaction alert will be shown, leading sometimes to suboptimal recommendations.

Another example is that the recommendation to achieve a certain therapeutic drug concentration interval can only be given, if there is scientific evidence. In addition, even though the potential user of a knowledge base (the general practitioner or any other physician) and the drug expert have the same basic medical education, they do not “talk the same language” and medical and clinical expertise differs, with clinical experts having more knowledge about patient treatment while drug experts possess more information about the properties of the drugs used. Medical advice given by the drug expert might not suit the practical needs of the physician and on the other hand the specialist physicians´ needs may not be fulfilled due to missing medical evidence.

It is also very important to clarify when CDS systems or knowledge bases can help and when they can’t. For example, during the development of the drug-drug interaction database Sfinx one of the future users mentioned, that he now finally can detect all the drug interactions for herbal drugs his patients always take. But since this physician never enters herbal drugs to the patient’s drug list, because he is not prescribing them, he will never get a warning for these drugs.

Prior to the development of the content of a knowledge database the multidisciplinary group needs to define its structure. For example developing the drug-drug interaction database (DDI db), Sfinx, physicians wished not only to receive warnings on certain drug interactions, but also recommendations on how to avoid and handle this interaction (Böttiger et al., 2009). The recommendation part is extremely important, since physicians do not only want warnings on avoiding certain drug combinations, but would like a recommendation how to handle the situation. In a survey among prescribers and pharmacists in the US both groups demanded that drug-drug interaction alerts should be accompanied by management options of the DDI (Ko et al., 2007). In a recent Australian study (Sweidan et al., 2009) recommendations for handling of drug-drug interactions were seen as a quality measure for the DDI databases. However, comparing 9 drug interaction systems used in primary care only 1 out of 9 systems provided useful management advices.

A number of studies have demonstrated, that physicians need timely, easy to digest and up-to-date information, which is filtered, summarized, and synthesized from reliable sources by clinical respected experts (Revere et al., 2007; Grol & Grimshaw, 2003; Schiff & Rucker, 1998). The expert group needs to define the relevant sources to be used for the knowledge base, which might consist of recent research publications, legal documents, information from pharmaceutical companies, textbooks, and other databases. It is critical for the integrity of the knowledge base to use scientifically rigorous methods for evaluation of scientific data by applying critical drug evaluation principles (Godman et al., 2009). Search strategies have to be developed and documented in standard operation procedure protocols (SOP´s) to assure reproducibility of the search results (Böttiger et al., 2009). This is critical in all cases but especially if different people are executing the same task or if expert groups are located in different places and can not communicate with each other on a daily basis.

Figure 1 describes the process from filling a knowledge base with data to providing it to the end user. Literature searched will be evaluated by different experts regarding their clinical relevance and their level of documentation according to standardised rules. It will then be synthesized into short text messages, according to a predefined structure (Böttiger et al., 2009). Different content providers have to use the same tool for data entrance.

Figure 1.

The design and process of building and maintaining a knowledge base to be integrated into CDS systems for drug prescribing at point of care or accessible through the web. User feed back triggers new literature searches and improves the quality of the knowledge base. This figure outlines the data management process for the drug-drug interaction database Sfinx (Böttiger et al., 2009).

It is advisable to define in advance certain standard terms for the text messages to avoid heterogenicity in the text content. It is easier for the end user to recognise standard phrases for certain conditions or recommendations.

Data are entered into the knowledge base and connected to various other registries or databases to assure optimal usage in the CDS system used by the prescribers. We have experienced, that access to experts is always the bottleneck in the production of knowledge bases, which agrees with the experiences by Kuperman et al. (2006). Consequently, data entrance into the knowledge base has to be simplified to save valuable expert time. For example through an easy text sharing function the experts should be able to reuse the same texts in different documents (e.g. interactions), which follow the same interaction mechanisms and rules, as a result reducing both the time for entering data and the size of the database. We have developed a terminology model for substances so that substances within a class such as different salts of the same substance belong to the same mother substance, if they react in the same way (Böttiger et al., 2009). The grouping of substances and text sharing function results in an effective and easy way to use the tool which simplifies data entrance and database updates.

Knowledge bases need to be connected to certain registries through software algorithms for their optimal use (see Fig. 1). Because texts in knowledge bases usually are written on a substance basis they need to be linked to specific drugs which contain the substance. This linkage can be done using substance registries, which contain substance names and drugs connected to the substance. Key fields for the linkage can be:

  1. ATC (Anatomical, therapeutic, chemical classification) codes

  2. CAS (Chemical Abstract Service) numbers

  3. other nationally available unique identifiers (Böttiger et al. 2009).

All systems have advantages and disadvantages. The ATC code system is valuable since it takes indications of drugs into account. This can be used to link or exclude drugs containing the same substance, but with different formulations or used in different strengths. A disadvantage of the ATC code system is the handling of combinational drugs, where the content of the drug most often is not specifically defined by the code. CAS numbers identify each substance in a unique way, which allows correct linkage. Problems within the system are its complexity. For example a substance which appears to be the same might have a different CAS number due to its varying content of crystal water, which is not obvious from the description of the drug. Another disadvantage is the limited use of CAS numbers in national registries. National identifiers might be the optimal way for linkage of knowledge bases to drug registries. However, substance based national identifiers do not take drug dosages into account. A substance can have a different interaction profile due to variations in dose (for example: high-dose versus low-dose acetylsalicylic acid). Consequently linkage just by a substance identifier would lead to wrong interactions alerts. Another disadvantage of national identifiers is, that the national identifier can’t be used across nations, a problem we faced for the Sfinx database distributed in Sweden and Finland. Whatever registry or system is used for linkage it is of great importance to ensure correct update and maintenance of the registries as described in the next chapter.

If drug formulations are relevant for the triggered alert, these should be taken into account. Even here it is important to simplify matters for the knowledge expert and create drug formulation groups (e.g. all sorts of tablets, capsules or oral solutions should be grouped under the term “peroral”). The Swedish drug registry contains about 650 different drug formulations, which we have grouped into 5 different groups in the “drug formulation classification file” (Fig.1) to support data entrance. International standard terms are needed to reduce the work load for a single country and to facilitate the integration with other drug registries from other nations. To our knowledge there is no European or worldwide registry with standardised drug formulations available, which could facilitate integration of knowledge bases across countries.

Finally database updates have to follow the same procedures and rules as defined for the starting phase. Ideally, they should include incorporating the handling of end user comments and feedback for further improvement and refinement (Böttiger et al., 2009). Each specific update should be tested, documented and saved in order to be able to trace back incorrect alerts reported by the end users.


3. Combined quality assurance for knowledge bases and linked registries

Quality assurance of the knowledge bases and their linkage to other registries is an essential task often forgotten as it is time consuming, labour-intensive and requires significant effort and expertise. Quality assurance does not only refer to the medical content in question but stretches over the whole procedure from literature searches, the evaluation process to the linkage of the knowledge base to local or national registries and thereby requires experienced multidisciplinary staff.

Literature about quality assurance processes within clinical databases is limited. Quality assurance papers in medicine mainly deal with securing the quality for a certain medical treatment or procedure, but are not extended to databases and CDS systems. It is amazing that still today EHRs or CDS systems do not need to be certified by health or medical agencies. However, due to the increasing awareness of the possibility that information technology implemented into health care can actually increase the error rate even with risk for higher mortality rate (Han et al. 2005), changes are on their way both in the US (Blumenthal, 2009) and Europe (EU directive; 2007/47/EC;2007). Certification should cover not only the technical part of these systems, but should include even the medical content of knowledge bases integrated into CDS systems and implementation of the systems.

Quality assurance is mostly self evolving during the development phase of any database system including the handling of external registries for linkage purposes. Baorto et al. (2009) describe the experiences they made with the maintenance of a large medical ontology at one of the larger hospitals in New York. They state that the methods described even though developed specifically for their system can be used for carrying out similar tasks at other institutions. Many of the problems and procedures mentioned mirror exactly the situation with the development of our knowledge base systems. In our mind resources and expertise for quality assurance processes are needed for integration and maintenance of high quality knowledge bases. Standards need to be developed in this area.

Combining different registries or other knowledge sources has to be performed using “key fields” like ATC codes, specific identifiers, or CAS-numbers. We were surprised though when comparing different registries that the information in key fields could vary. For example a drug could be assigned to a specific ATC code in one registry and this could vary from the code in another registry. This could be due to simple typing mistakes, system requirements of the registry owner, delays in the update process or other possibilities. We, like Baorto and colleagues (2009), used the “diff approach” for detection of theses variations, where you compare two registries regarding the information in predetermined fields using the information in key fields to link the registries. For example we assume that a drug with a specific registration number has to have the same name, ATC code and drug formulation in all registries. Another approach to detect variations is to compare an older version of the same file with the newer one discovering changes for already existing fields, and new posts entered to the file. Logs are produced during the comparison process, which mainly have to be evaluated manually. Possible mistakes are corrected in the registries and reported to the source owners for correction in the original source.

Over the years we have discovered many mistakes at the point of acquisition of the data including missing information in essential fields (they were either completely empty or omitted), changes in the meaning of existing codes, existence of wrong characters in the master file or creation of redundant terms. The “diff approach” is also used for updating the knowledge base system, e.g. to identify new substances on the market, which will then be added and grouped into the mother child terminology, to discover new drug formulations, which have to be included into the drug formulation file, or to seek for new drugs on the market, which have to be linked to certain knowledge bases. The linkage has to be correct both technically and content wise else care will subsequently be compromised. For example, you can’t link some new ear drops, containing a substance you already have in your knowledge base, to that base, if the text document is irrelevant for this new drug.

Auditing terminology and data structure of the registries linked to the knowledge base is mainly performed manually through reviewing log files created during the import process, which flag for changes and differences. These processes are labour-intensive and time consuming. Some of these audit processes though can be automated or at least semi-automated to save time and resources. For example, if you want to add a new substance child to the registry the hierarchy principle within your database requires the existence of the mother substance to be consistent with previously existing structures. Other examples are rules you create for maintenance purposes, like no two medications with the same registry number are allowed with different names.

As Baorto et al. (2009) stated quality assurance and maintenance of the knowledge base and its linked registries is a “mission critical” task that cannot tolerant errors. If we do not add one specific, new ATC code to a document the new drug assigned to that code will fail to be considered by the alerting system. It must be recognised though that all quality assurance processes rely at least partly on human surveillance so they are inevitably prone for mistakes. One can never be sure, that the knowledge base is completely correct. However, we can increase our confidence in the database through implementation of audits, rules and log files. This will help to create a system, which is detecting and minimising a large percentage of potential errors.

Any errors that occur through the usage of the knowledge base have to be handled by the medical management and maintenance system in a systematic way to enhance the utility of the database. This is described below.


4. Medical management and maintenance system for knowledge bases and CDS systems

The development and implementation of several knowledge bases, CDS systems and other IT applications within health care required the introduction of a surveillance system for possible errors introduced by its applications as an essential pre-requisite for the management of these systems. Our department has implemented a maintenance system, which allowed smooth handling of all procedures linked to its databases and depending registries e.g. regular update processes of the medical content, improvements or changes in the graphical interface and IT structure or adapting to new external registries. At the same time the EU Directive, 2007/47/EC, amending among others the Directive 93/42/EEC ( concerning medical devices is under implementation in Sweden, and supports the process by raising the requirements for software and information systems used for clinical decisions regarding individual patients.

Important parts for the function of knowledge bases and CDS systems are management, maintenance and quality assurance of these applications after their implementation, together with handling of possible errors introduced by the systems, which could be of either technical or medical nature. Clinical, medical, and pharmaceutical competences, as well as competences in various IT-areas and in implementation are needed. Additionally, complete technical documentation of the knowledge base and the CDS system as well as guidelines (standard operation procedures = SOPs´) for producing their content and its distribution have to be part of the management plan to secure standardized procedures and avoid occurrence of mistakes.

Documented incidents include all kinds of subjects i.e. requests for further information, e-services, training, as well as reporting of major or minor errors. Minor or major errors include discussions of diverse opinions about recommendations or conclusions in the knowledge base, wishes for changes in classification levels or inclusion criteria. Technically it could be problems in the applications, or its documentation, or errors regarding the technical integration including design and functionality of user interfaces. All incidents are documented in the management system. Within the management and maintenance system the experiences we have made through real incidents and errors enable us to perform risk analysis on a regular basis. This helps us to foresee and judge possible incidents, which might occur through changes in the content, the graphical interface or the technical solution of our systems.

4.1. Management of errors using root cause analysis

The medical management of incidents or errors involves the processes of discovering the incidents, collecting documentation, performing event analysis and, if required, reporting of the error as a medical event - named Lex Maria - to the authorities in Sweden (Shemeikka et al., 2008). Root cause analysis (RCA) is a technique originally developed in psychology and systems engineering to identify “the basic and casual factors that underlie variation in performance”. We use RCA to investigate errors after they are discovered. It involves critical incident reporting followed by self-managed investigation of the event involving all staff in charge. It should answer three basic questions:

  1. what happened?

  2. why did it happen?

  3. what could be done to prevent it from happening again?

The investigation team consists of colleagues from the department and includes everybody involved in the processes related to the incident.

In the US root cause analysis for investigations of medical errors became mandatory in 1997 for hospitals accredited by the US Joint Commission on Health Care Safety. Models used for RCA were further developed and adopted for by health care systems in other countries like Australia (Iedema et al., 2005). Though RCA used in the US and other countries only included medical procedures and not handling of errors introduced by decision support systems, we have applied the technique for handling of our knowledge bases and decision support tools as well as the e-prescribing system. We have relied on experiences by team members from using the method in pharmaceutical companies to handle reports on adverse drug reactions, and from the health care system reporting events when harm or risk of harm for the patient has occurred during medical treatment.

Once an error of the CDS system is reported an initial rapid assessment is performed of the potential immediate and long term clinical consequences. If there is any risk for the safety of the patient or other patients due to the error, a decision is taken to shut down the e-services or keep it going whilst performing immediate changes. In these cases a report is sent to the national authorities in charge of monitoring and guarding patient safety during clinical care. The error is documented in detail often by requesting additional information from the reporter. The next step is to perform RCA to investigate the reason for the error (Iedema et al., 2005) and to suggests changes in for example the system, content, procedures or technical and user interfaces.

Incidents can be due to medical (e.g. wrong medical recommendation), pharmacological (e.g. wrong pharmacological mechanism thought to be cause for a DDI) or pharmaceutical (e.g. drugs with wrong formulations can be linked to a text) errors in the content of the knowledge base, or due to an unclear text, leading to misinterpretation. Errors in drug linkage can result in wrong alerts for a certain drug or missing alerts. The reason for the error could also be of technical nature. RCA may lead to organisational changes like education of the personal or policy changes, though they have a lower probability of reducing risk (Wu et al. 2008). It may also lead to changes of the content or processes for producing the knowledge bases or CDS systems or in redesign of the product or processes linked to knowledge base or CDS system, which are actions with a high probability of reducing risk (Wu et al., 2008). Procedural changes may lead to updates in the documentation or SOPs´ for the knowledge bases. Any changes in the device will be followed by extensive tests of the modified application before reintegration into the work environment. If the incident does not depend on one’s own systems but on the EHR the CDS system is implemented in, the health record system owner has to solve the problem and document and proof changes. These changes are performed in close contact with vendors and producers of electronic health record systems.

Other incidents like inappropriate handling of the CDS system by the user may lead to a modification of the system and if necessary, user training must be performed. An example of a RCA is shown in figure 2 and 3. It describes an incident, where an ATC code was connected to a medical document by mistake. Drug name and ATC code was incorrectly send to the authors of the knowledge base. This led to the addition of the code to the document by the authors and a wrong linkage of drugs to the document. Processes for quality assurance of linkage of drugs to documents failed due to various reasons (technical equipment; frequent change of personal involved in the process). Consequently, users searching on the web for one of these drugs in one of our knowledge bases ended up in a document which had nothing to do with the drug searched for. Even if RCA has some benefits, including increased awareness of faulty processes and fixes to specific problems, more emphasis should be placed on drawing lessons across investigations rather than to approach each RCA independently. Most important, follow-up for implementation and outcome of each RCA and its actions should become a standard element of the process (Wu et al. 2008)

Figure 2.

RCA part 1: On the top of each RCA the incidents following each other and leading to the mistake are stated. Next line gives the reasoning for each incident. These are followed by the causes, grouping the reasons into categories. Each reason is followed by one or several actions suggested.

4.2. Analyses of risks

Risk analyses are also included into the management and maintenance system. Using the experiences made with existing systems we apply this knowledge to other parts of the knowledge base and the CDS system to foresee possible risks. On a regular base we perform preventive risk analysis to identify and classify different kinds of risks. The method has been adapted and is now used even during development of new CDS systems or knowledge bases in our setting in Stockholm. It improves our possibilities to evaluate the costs, risks, and improvements made with the implementation of new knowledge bases or decision support tools. For example when the graphical interface of the decision support system is changed risk analysis can be performed on possible effects for end user performance.


5. Providing medical knowledge bases at point of care

The knowledge base can either be provided:

  1. as a website solution

  2. integrated into EHR systems

  3. used in learning tools.

The integration into EHR systems facilitate the exchange of patient-specific data with the knowledge base, thereby creating patient-specific alerts or reminders during the process of drug prescribing.

Figure 3.

RCA part 2: the second part of the incident chain explains, how the document with the wrong ATC code was added to the database without proper controls of the document and the effect it had on the linkage of drugs to the document. Actions suggested include changes in routines and policies, education and even changes in the technical tools used.

The integration into an EHR system should be performed in collaboration between the providers of the knowledge base and the owners of the EHR systems. Contracts should specify the implementation of the database and how it is to be used and presented to the end user. The organizations implementing CDS systems must have detailed knowledge of the structure of the knowledge base and the architecture of the CDS system so that it is clear, how the systems interact (Kuperman et al.2006). Intensive testing of its integration following predefined protocols should be required to avoid unintended errors or mistakes due to lack of experiences and knowledge of the product. One must be sure that the knowledge base is behaving as intended (Kuperman et al. 2006).

The knowledge bases for drug-drug interactions, Sfinx, drugs & pregnancy and drugs & lactation produced by Stockholm County Council are provided free of charge through the county website on The website is aimed at health care personal. Physicians or nurses can search various knowledge bases by typing in the patient’s medication and receive advice, whether specific drugs can be used during pregnancy or breast feeding or should be avoided (Norby et al., 2006). Drug-drug interactions can be searched for in Sfinx by either substance or drug names.

However, for optimal use knowledge bases should be implemented into a CDS system linked to an EHR, which will send patient specific data such as age, sex, height, weight, parameters for kidney function and the current drugs a patient is being prescribed to the knowledge base. Through certain software algorithms an alert or reminder could then be triggered or not, providing patient specific warnings for e.g. drug-drug interactions, drugs & lactation or drugs & breast feeding.

The DDI database Sfinx is integrated into the CDS system Janus toolbar, providing patient specific automatic alerts during drug prescribing (Sjöborg et al., 2007). In figure 4 we describe an example of the decision support system provided through Janus toolbar integrated into one EHR system in Stockholm County Council. The patient’s name, sex and age can be seen at the top of the screen. The prescribing module within the EHR contains the current drug list, consisting of 4 different drugs. Sending those data to the knowledge base for DDIs´, pregnancy and breast feeding the alert buttons will be illuminated, if there is any information to be retrieved (Eliasson et al., 2006, Sjöborg et al., 2007). It is of great importance that the EHR and the knowledge base interact in an optimal and correct way. For example in a survey among ambulatory care clinicians in Massachusetts it was observed, that the local CDS system often delivered alerts with out-of-date medications, which led to scepticism towards the system among users (Weingart et al. 2009).

Figure 4.

Implementation of Janus toolbar into an EHR. Patient specific alerts are illuminated related to the patient’s age, sex and current list of drugs. Several different knowledge bases are the basis of the decision support system. For every new order of medication a new drug list will be send to the knowledge base, evaluated and may lead to changes in the alerts.

To further improve user friendliness, accessibility, and speed of the CDS system the most important information of the knowledge base should be short and concise only one click away. This principle is implemented into the Janus toolbar with the most important message being provided immediately, while for information about possible mechanisms, background studies for the statement and references users have to click further (Eliasson et al, 2007, Böttiger et al. 2009). We believe this quick access to pertinent information enhances the utilisation of the support tool. Even other surveys have shown that important information should be easily accessible and speed of use is a critical factor for the successful use of medical information systems (Dawes & Sampson, 2003; Bates et al., 2001).

Figure 5 shows the information provided by the knowledge base for drug-drug interactions, - Sfinx. Sfinx was developed by us together with partners from clinical pharmacology in Finland and in Sweden (Böttiger et al., 2009). Clicking on the yellow alert button, which is illuminated according to the patients’ drug list, short and concise information about the medical consequence and recommendations can be seen immediately. Additional more educational information is available through clicking on the “read more” button.

Figure 5.

Warning texts of the drug-drug interaction database, Sfinx. The yellow colour code is used for interactions classified as “C” which means, that the interaction is clinical relevant but the drug combination can be handled by for example dose adjustment (Böttiger et al., 2009). A short consequence text describes, what can be expected medically. This is followed by a recommendation part, stating how to handle the interaction.

The Janus toolbar alert system delivers non-intrusive reminders. This means that the illuminated warnings are optional not forcing the physician to take any action and not disturbing the workflow for the practitioners. Shah et al. (2006) showed that acceptance of drug alerts was improved by minimizing workflow disruptions, designating only high severity alerts to be interruptive to clinicians work. Disadvantages with intrusive alerts are disruption of physicians’ workflow and increased tendency to ignore, work around or override these warnings. In a survey by Krall & Sittig (2001) physicians indicated that intrusive or active alerts might be more useful but less easy to use. It was also stated that another important factor for increased compliance and effectiveness of a CDS system is the interface design in relation to the workflow process. Alerts showing up too early or too late in the workflow process might lead to decreased compliance and reliability of the users in the system or even worse, lead to errors and harm for the patient (Krall & Sittig, 2001; Khajouei & Jaspers, 2008).

Studies on the effectiveness of non-intrusive versus intrusive alerts are contradictory. One study (Palen et al., 2006) showed no significant difference between control and intervention groups in the overall rate of compliance to ordering certain laboratory monitoring values when prescribing certain medications. They used non-intrusive alerts in their intervention group. Another study (Tamblyn et al., 2008) compared the effectiveness of on-demand versus computer triggered decision supports regarding dosing information, drug-drug, drug-age, -allergy and -disease interactions. They found that physicians in the computer-triggered group saw more alerts, and made more changes. However, they also ignored more of the alerts shown (87.8%). The on-demand group requested less than 1 % of all alerts provided by the CDS, but ignored only 24.4%. There was no difference in the overall result of existing prescribing problems after intervention between both groups.

We believe that CDS systems need to keep a balance between producing too many alerts and reminders and delivering the message in a straight-forward manner. Too many alerts are likely to be overridden and cause “alert-fatigue”, which leads to underestimation of the CDS systems as useful tools in the daily practice (Shah et al. 2006). To avoid too many uncritical alerts classification of the content of knowledge bases regarding clinical significance is of great importance. Numerous studies have shown that compliance to CDS systems and user satisfaction is related to the balance between useful alerting and overalerting (Paterno et al. 2009; Shah et al. 2006). Therefore, we have implemented classification systems in all our knowledge bases. Classification is performed regarding the clinical significance of the content and the level of documentation for the alerts. Colour codes are provided additionally to knowledge base specific classifications (letter or number codes) thereby supporting the prescriber, to identify the urgency of the information retrieved from the knowledge base. The red colour signalises very important messages (e.g. for drug- drug interactions it means: avoid combination) (Böttiger et al., 2009). A yellow colour code indicates information, which should be retrieved and could influence the prescribing (e.g. dose adjustment for a DDI warning). White colour means that information of more theoretical value is available but it has no clinical relevance which has to be considered during prescribing.

Isaac and colleagues (2009) recently showed, that physician’s tendency to override alerts was less pronounced for the alerts with high-severity / high risk compared to medium or low severity alerts. Tiered alerting for severity for drug-drug interaction information, like in Sfinx, is one possibility to increase compliance rates for interaction warnings. That was confirmed in a study by Paterno et al (2009), where compliance in the tiered DDI alert group was significantly higher than in the non-tiered group (29% vs. 10%). Additionally, the most severe alerts were accepted to 100% in the tiered group while only 34% in the non-tiered group.

Commercially available DDI databases tend to put more emphasis on covering the whole medical domain rather than differentiating between clinical important and non-important messages. So there is a need for increased specificity to reduce extraneous workload and reduce “alert-fatigue”. Luna et al. (2007) described the need to “clean” the content of their commercially purchased knowledge base according to the clinical significance of drug-drug interactions. By creating a classification for DDIs in the system they customized the knowledge base for their organisation.

Spina et al. (2005) investigating the usefulness of different types of alerts in a CDS system in a group of primary care physicians stated that more tailored systems are needed, where DDI warnings on topical drugs should be avoided, when not relevant. Therefore drug formulations should be taken into account in a DDI knowledge base (Böttiger et al., 2009). Also interaction warnings should be suppressed, when drug monitoring is already in place.

Another option can be to suppress warnings on reorders for patients’ medications as shown by Abookire et al. (2000). They found that overriding rates for drug allergy warnings increased from 48% to 83% for drugs being reordered for a single patient over a certain time period, suggesting that physicians tend to ignore warnings for the patients permanent medications, since they have handled and considered these alerts already once before. Consequently, tailoring systems focussing more on new ordered medications rather than on drug renewals would be another possibility to increase usefulness of CDS systems.

However, it will not be possible to develop knowledge bases and CDS systems fitting all needs. Personal adjustments seem to be necessary since physicians´ needs and their varying level of knowledge result in different perceptions of any CDS system.


6. Implementation of CDS systems

Healthcare agencies spend significant amounts of money on the development of clinical information systems, though often failing with successful implementation. Designing an effective approach for increasing end-user acceptance and subsequent use of IT- systems is a fundamental challenge. Successful implementation needs comprehensive approaches tailored to clinical settings and target groups taking individual, health care team, and organizational variations into account.

Wears & Berg (2005) described how implementation of any new technology into a clinical workplace triggers both changes in the workplace and in the use of technology, which itself triggers development of the technology (Figure 6). A workplace is described as a field where social behaviour meets technology and both influence each other.

It is also of great importance to consider the different interests in and views on a CDS system from users, administrators and vendors. Ash and colleagues (2003) described the complex interplay of physicians, administrators and IT- staff when implementing a computerized physician order entry (CPOE) system into a hospital setting. They looked at three important parts, which are always influenced by an implementation: the technical, organizational and personal part. Physicians thought the CPOE as technically cumbersome and time-consuming, forcing them to think like computers and click through various screens. They also felt that the CPOE was “forced” on them by hospital administration not taking into account the work situation which they believed was already overburdened. However, on a personal note they felt a need to master the system. The hospital administration thought the system technically to be cost-effective and delivering great statistics. People in the organisation felt pride in being at the forefront of technology. Personally they felt pride in having overcome the clinicians’ resistance. The information technology staff perspective on the technical system was the urge and tendency to make the system even more useful, train the users and fulfil and develop the system according to the users wishes. Organisationally they tried to identify the right staff members for the implementation to reach everybody in the hospital. Personally they described enthusiasm for the benefits of the system, but at the same time they felt implementation as difficult and painful but useful in the long run. This study reflects the difficulties of a successful implementation taking into account the various expectations of different “interest groups”.

Figure 6.

Influences of technology in a clinical workplace environment. This figure shows that any new technology integrated into a clinical workplace will change work practice, which then will result in use of the technology different as planned from the beginning. This will trigger the development and change of the technical tool implemented (Wears & Berg, 2005).

Ash and colleagues (2003) derived four categories of principles for a successful implementation:

  1. computer technology

  2. personal principles

  3. organizational principles

  4. environmental issues.

These principles reflect the need to consider multiple issues during implementation and they highlight the relationship between technology, clinical information, people and organizational issues. Callen et al. (2008) described a Contextual Implementation Model (CIM), which is based on data from sites, where physicians use an existing CPOE system. The model acknowledges the complexity of the clinical environment and the requirements of the users. They concluded that implementation should start with a thorough analysis of the context where the CDS system will be implemented into. This analysis should include all three levels namely organisational, departmental and individual. Work practices have to be studied on an individual and department level. Computer literacy and keyboard skills have to be investigated among potential users and work requirements between departments have to be clarified to take the differences between organizations into account. Requirements of the CDS systems on individual and workplace level have to be investigated and differences can be included in the implementation plan so that during implementation one can accommodate the different needs. Targeted training programs can eliminate the problem with different keyboard skills and computer literacy. Analysis of organizational and team cultures will assist with modifying the cultures to increase receptiveness. They concluded that using the CIM model for implementation will facilitate the usage and benefit of any CDS system.

In a systematic review (Gruber et al., 2009), it was stated, that no single implementation strategy has proved to be completely effective. The authors defined a theoretical model for a computerised decision support system including five major steps in the life cycle of any CDS system (= Expanded Systems Life Cycle = ESLC):

  1. planning

  2. analysis

  3. design

  4. implementation

  5. maintenance

They identified risk zones for each phase and corresponding risk factors. Their analysis revealed that the highest number of failure and success were in the implementation zone focusing on preimplementation and “go-live” of the system. They also identified that training and education, attention to training, policy, process changes, and training to clinical content are key factors influencing the success or failure of a CDS system.

However, more research is needed to avoid costly errors in implementation. Studies focussing on barriers and incentives for changes should be performed focussing on various levels (namely the innovation itself, the professional, the patient, the social context, the organisational context, and the economic and political context) as suggested (Grol & Wensing, 2004).


7. Evaluation

Rigorously designed evaluations and research on the effectiveness of decision support systems are needed to assess their value in clinical practice and to identify areas for improvement in design and implementation. Kirkpatrick described four levels of evaluation in which the complexity of the behavioural change increases as evaluation strategies ascend to each higher level (Kirkpatrick, 1967). The four levels measure

  1. reaction to information

  2. learning

  3. behaviour

  4. results

Studies assessing effects of CDS systems on patient outcome are urgently needed. They are difficult to perform due to the length of time needed for the evaluation, the lack of reliable objective measures, and the number of potential confounding factors.

The selection of methodology to investigate an implementation of decision support systems is no different from choosing methods in any other type of research. A variety of study designs can be used to evaluate if decision support systems influence prescribing behaviour and patient outcomes. These studies include quasi-experimental designs (uncontrolled or controlled before-and-after studies and interrupted time series) and randomized controlled trials (RCTs) (Grimshaw et al., 2000). The RCT has the highest degree of evidence as non-randomized designs might introduce selection bias by including in the intervention group doctors or clinics that favour the particular intervention (Grimshaw et al., 2000, Stephenson & Imrie, 1998). The control group design considers other factors influencing the prescribing pattern such as seasonal variations in disease patterns, the introduction of new drugs and changes in treatment policies, the marketing activities of pharmaceutical companies and changes in regulatory policies (Grimshaw et al., 2000). However, due to ethical, practical and methodological reasons, they are seldom possible to apply when evaluating the impact of decision support systems. Therefore, well-designed quasi-experimental studies may be the method of choice.

Alternative research strategies include qualitative research methods to provide a deeper understanding of the subjective aspects of the interaction between healthcare professionals, patients and the electronic tools. The common feature of qualitative studies is that they do not primarily seek to provide quantified answers to research questions. The goal of qualitative research is the development of concepts which can help us to understand social phenomena in natural rather than experimental settings, giving due emphasis to the meanings, experiences, and views of all the participants (Pope & Mays, 1995). Examples of qualitative methods include in-depth interviews, focus group discussions, observations and various consensus methods.

Development and evaluation of a complex system, such as a CDS system and implementing it into the health care organisation require a multiple research approach i.e. method triangulation. The evaluation of the pilot study of the Janus decision support system had primarily a qualitative approach with focus on user satisfaction. Semi-structured qualitative interviews were performed with all users before, during and after the pilot study. By concentrating the evaluation on user satisfaction we gained data both on the technical failures as well as the physicians’ attitudes to medical content and usefulness of the system and acceptance in clinical work. The evaluation and implementation were carried out by a multidisciplinary team within a small scale user clinic in order to be able to easily detect technical and practical obstacles (i.e. integration bugs) and even more serious potential quality problems of the pharmacological sources (i.e. pregnancy and breast-feeding alerts in the Swedish PDR) (Eliasson et al., 2006). Data and support were handled in a rapid way to be able to give direct feed-back to the user. Our experiences confirm that evaluations of small-scale pilot studies for proof of concept are important tools in the design of an optimal intervention that improves health care quality so that resources are used in an optimal way as stated by Harvey & Wensing (2003).

The results of the pilot study even helped us to identify factors, which have major impact on usefulness of the CDS system and user satisfaction and led to a two-part theoretical model for implementation and evaluation (Eliasson et al, 2006). This model considers both system-dependent and system-independent factors (Figure 7). The first part includes system-dependent factors, such as medical content, user friendliness and user support. The second system-independent part includes personal attitudes of the prescribers´ towards computer use as well as the attitude of the organisation towards implementing a CDS system.

Stockholm County Council conducts regular evaluations after pilot studies which we see as a cornerstone for development of successful electronic tools. The effectiveness of Janus toolbar and the frequency of its use, and users’ characteristics are measured by questionnaires. Simultaneously, interviews are carried out to explore doctors’ and other prescribers’ experiences and perceptions of Janus toolbar.

Figure 7.

A two-part theoretical model for evaluation of the CDS system taking system dependent (e.g. medical content, usability, support) and system independent factors (e.g. personal attitudes, organizational aspects) into account.

Those evaluations were used to decide about the development of a new knowledge base for drug-drug interactions Sfinx, which is described above. The regular follow-ups over years showed results similar to the actual literature especially in terms of satisfaction, acceptance and intention to use (Krash, 2004, Ahearn & Kerr, 2003; Magnus et al., 2002). Physicians generally overrode the interaction warnings and expressed irritation on the irrelevant alerts, which often led to ignore them. Furthermore, physicians were dissatisfied with the usability, information and training of how to use the tool, and complained about technical barriers. Although physicians did not seem to use the tool in every day practice they underlined the clinical value and needs of it, i.e. being reminded of unknown /known drug-drug interactions and getting recommendations about how to avoid them. Prescribers were aware of the fact that the decision support system contributes to safer and more effective treatment of the patients. They were clear about their needs for the system and had good intentions to use it. However, even after thorough analysis of physicians’ needs, we could observe that the system was not fully used after its implementation.

Some contributing factors are changes in expectations and intentions of the users from the initial discussion, to later implementation and the actual use at the work place when the database is integrated into daily work flow. Another influencing factor is that CDS systems integrated into daily practice suddenly offer more complete knowledge about patients’ medications for the physicians, demanding new decisions and work tasks that GPs were not aware of. Physicians have different views on their responsibilities for diagnosis, drug treatment and follow-up of a patient resulting in different actions and variations of handling the information provided. Recently we have highlighted that there is a need for common and understandable rules on prescribing physicians´ responsibility in handling the total patients’ drug lists. These lists are made available to all prescribers through a newly implemented IT-tool ( Rahmner et al., 2009 ). We can conclude that work flow, working environment and processes influence physicians’ behaviour to a greater extent than expected.

Consequently, we still do not know how to design optimal CDS systems which affect and influence physicians’ behaviour in drug prescribing. The challenge for the future development and implementation of a CDS system into health care is to find a method to achieve and maintain expected changes in prescribing behaviour.


8. Summary

Knowledge bases provide the contents for any clinical decision support system. In this review we characterize the life cycle of a knowledge database to be used in drug prescribing. The various phases and the important issues in each phase are summarized in table 1. Knowledge bases need to fulfil and be tailored to the needs of the users. The focus of the content should be on practical use in a clinical environment, rather than covering the whole scientific area of a medical speciality. Standards are needed to be able to use knowledge bases across different electronic health care systems and countries, since clinical expertise is often the bottle neck for any development.

Life cycle phase of a knowledge base Important issues for each life cycle phase
Develop ment Thorough analysis of physicians needs Standardisation of data structure, implementation of classification system, optimal linkage to drug registries
Quality assurance Control of quality in key fields for linkage Introduction of semi-automated and manual processes for data auditing
Medical management & maintenance Well documented and standardized procedures for knowledge base maintenance Root Cause Analysis for analysis of mistakes and follow ups of the planned actions
Providing knowledge bases at point of care Integration into electronic health records for patient specific alerts Tailored systems with fast data access to avoid overalerting and increase acceptance
Implementation Consider interests of users, organizations and vendors within the implementation plan Education, personal training, attention to process changes are key factors
Evaluation Evaluation of small scale projects as important tools in the design of optimal interventions Regular evaluations necessary to secure optimal use of knowledge base or CDS system

Table 1.

Summary of important messages for each step in the life cycle of a knowledge base.

Integration of knowledge bases into CDS systems implemented into electronic health record system optimises their effectiveness by delivering patient specific reminders and alerts. The linkage between knowledge bases and CDS systems needs to be quality assured. Knowledge bases and CDS systems need to be surveyed through a management and administration system handling incidents and errors due to system or its content. Though many studies have shown the positive influence of CDS systems on physicians’ performance, there is still lack of understanding, when CDS systems improve performance. Outcome studies on patient care are lacking. Implementation of CDS systems has to be accompanied by staff education and training to assure acceptance and effectiveness even throughout the maintenance phase. More studies are needed with focus on actual improvement of patient safety and care instead of investigating physicians change in prescribing drugs.

With that in mind knowledge bases and CDS systems will prove to be helpful tools in the daily decision making process of any busy clinician when instituting and evaluating the drug therapy of a patient.


  1. 1. Abookire S. A. Teich J. M. Sandige H. et al. 2000 Improving allergy alerting in a computerized physician order entry system. Proc AMIA Symp., 2000., 2 6 .
  2. 2. Ahearn M. D. Kerr S. J. 2003 General practitioners’ perceptions of the pharmaceutical decision-support tools in their prescribing software. MJA., 179.,34 37 .
  3. 3. Ammenwerth E. Schnell-Inderst P. Machan C. et al. 2008 The effect of electronic prescribing on medication errors and adverse drug events: a systematic review, JAM Med Inform Assoc., 15., 585 600 .
  4. 4. Ash J. S. Gorma P. N. Lavelle M. et al. 2000 Multiple perspectives on physician order entry Proc Amia Symp. 27 31 .
  5. 5. Ash J. S. Fournier L. Stavri P. Z. et al. 2003 Principles for a successful computerized physician order entry system implementation. Proc Amia Symp., 36 40 .
  6. 6. Ash S. A. Berg M. Coiera E. 2004 Some unintended consequences of information technology in health care: The nature of patient care information system-related Errors, J AM Med Inform Assoc., 11., 104 112 .
  7. 7. Aspden P. Wolcott J. Bootman J. L. et al. 2006 Committee on identifying and preventing medication erros (2006). Preventing medication errors. Washington DC; National Academies Press.
  8. 8. Baorto D. Li L. Cimino J. J. 2009 Practical experience with the maintenance and auditing of a large medical ontology, J Biomed Inform., 42., 494 503 .
  9. 9. Bastholm Rahmner P. 2009 Doctors and drugs- How Swedish emergency and family physicians understand drug prescribing (Thesis). Medical Management Centre, Department of Learning, Informatics, Management and Ethics. Karolinska Institutet, Stockholm Sweden
  10. 10. Bastholm Rahmner P. Andersen-Karlsson E. Arnhjort T. et al. 2004 Physicians´perceptions of possibilities and obstacles prior to implementing a computerised drug prescribing support system, Int J Health Care Qual Assur Inc Leadersh Health Serv., 17., 4-5., 173-179.
  11. 11. Bates D. W. Gawanda A. A. 2003 Improving safety with information technology, N Engl J Med., 348., 25., 2526 2534 .
  12. 12. Bates D. W. Cohen M. S. Leape L. L. et al. 2001 Reducing the frequency of errors in medicine using information technology, J AM Med Inform Assoc., 8., 299 308 .
  13. 13. Bergk V. Haefeli W. E. Gasse C. et al. 2005 Information deficits in the summary of product characteristics preclude an optimal management of drug interactions: a comparison with evidence from the literature, Eur J Clin Pharmacol., 61., 5-6., 327-335.
  14. 14. Blumenthal D. 2009 Stimulating the adoption of health information technology, N Engl J Med, 360., 15., 1477 1479 .
  15. 15. Böttiger Y. Laine K. Andersson M. L. et al. 2009 SFINX- a drug-drug interaction database designed for clinical decision support systems. Eur J Clin Pharmacol., 65., 6., 627 633 .
  16. 16. Callen J. L. Braithwaite J. Westbrook J. I. 2008 Contextual implementation model: a framework for assissting clinical information system implementations. J Am Med Infrom Assoc., 15., 2., 255 262 .
  17. 17. Dawes M. Sampson U. 2003 Knowledge management in clinical practice: a systematic review of information seeking behaviour in physicians, Int J Med Inform., 71., 9 15 .
  18. 18. Eliasson M. Bastholm Forsberg. P. et al. 2006 Janus computerised prescribing system provides pharmacological knowledge at point of care- design, development and proof of concept. Eur J Clin Pharmacol., 62., 251 258 .
  19. 19. Eslami S. de Keizer N. F. Abu-Hanna A. 2008 The impact of computerized physician medication order entry in hospital patients- A systematic review, Int J Med Inform., 77., 365 376 .
  20. 20. EU directive; 2007/47/EC 2007; accessed 2009-07-30.
  21. 21. Garg A. X. Adhikari N. K. J. McDonald H. 2006 Effects of computerised clinical decision support systems on practitioner performance and patient outcomes, JAMA, 293., 10., 1223 1238 .
  22. 22. Gartner 2009 eHealth for a Healthier Europe! accessed 2009-07-27.
  23. 23. Godman B. Wettermark B. Hoffman M. et al. 2009 Multifaceted national and regional drug reforms and initiatives in ambulatory care in Sweden: global relevance. Expert Rev. Pharmacoeconomics Outcomes Res., 9., 1., 65 83 .
  24. 24. Grimshaw J. Campbell M. Eccles M. Steen N. 2000 Experimental and quasi-experimental designs for evaluating guideline implementation strategies. Family Practice;17:S11-S18.
  25. 25. Grol R. Grimshaw J. 2003 From best evidence to best practice: effective implementation of change in patients´care. Lancet, 362., 9391., 1225 1230 .
  26. 26. Grol R. Wensing M. 2004 What drives changes? Barriers to and incentives for achieving evidence-based practice. Med J Aust, 180., S57 60 .
  27. 27. Gruber D. Cummings G. G. Leblanc L. 2009 Factors influencing outcomes of clinical information systems implementation: A systematic review. Comput Inform Nurs, 27., 3., 151 163 . quiz 164-165.
  28. 28. Gustafsson L. L. Widäng K. Hoffmann M. et al. 2003 Computerized decision support in drug prescribing II. A national database to provide up-to-date and unbiased information. In Swedish. Lakartidningen, 100., 15., 1338 1340 .
  29. 29. Han Y. Y. Carcillo J. A. Venkataraman T. et al. 2005 Unexpected increased mortality after implementation of a commercially sold computerized physician order entry system. Pediatrics 116., 6., 1506 1512 .
  30. 30. Harvey G. Wensing M. 2003 Methods for evaluation of small-scale quality improvement projects. Qual Saf Health Care. 12.,210 214.
  31. 31. Iedema R. A. Jorm C. Long D. et al. 2005 Turning the medical gaze in upon itself: Root cause analysis and the investigation of clinical errors. Soc Sci Med., 62., 7., 1605 1615 .
  32. 32. Isaac T. Weissman J. S. Davis R. B. 2009 Overrides of medication alerts in ambulatory care. Arch Intern Med., 169.,3.,305 311 .
  33. 33. Janusinfo accessed 2009-07-30
  34. 34. Khajouei R. Jaspers W. M. 2008 CPOE system and design aspects and their qualitative effect on usability. Stud Health Technol Inform. 136., 309 314 .
  35. 35. Kirkpatrick D. I. 1967 Evaluation of training. In Craig L., Bittel I., eds. Training and development handbook. New York: McGraw-Hill.
  36. 36. Ko Y. Abarca J. Malone D. C. 2007 Practitioners´ view on computerized drug-drug interaction alerts in the VA system. J AM Med Inform Assoc., 14., 56 64 .
  37. 37. Krahenbuhl-Melcher A. Schlienger R. Lampert M. et al. 2007 Drug-related problems in hospitals. Drug Safety, 30., 5., 379 407 .
  38. 38. Krall M. A. Sittig D. F. 2001 Subjective assessment of usefulness and appropriate presentation mode of alerts and reminders in the outpatient setting. Proc Amia Symp., 334 338 .
  39. 39. Krash B. T. 2004 Beyond usability: designing effective technology implementation systems to promote patient safety. Qual Saf Health Care, 13.,388 394 .
  40. 40. Kuperman G. J. Reichley R. M. Bailey T. C. 2006 Using commercial knowledge bases for clinical decision support: opportunities, hurdles, and recommendations JAMIA., 13., 4., 369 371 .
  41. 41. Lucas H. 2008 Information and communication technology for future health systems in developing countries. Soc Sci Med., 66., 10., 2122 2132 .
  42. 42. Luna D. Otero V. Canosa D. et al. 2007 Analysis and redesign of a knowledge database for a drug-drug interactions alert system. Stud Health Technol Inform, 129., 2., 885 889 .
  43. 43. Magnus D. Rodgers S. et al. 2002 GPs’ views on computerized drug interaction alerts: questionnaire survey. J Clin Pharm Ther., 27.,377 382 .
  44. 44. Norby U. Eiermann B. Tornqvist E. et al. 2006 Drugs and birth defects- a Swedish information source on the Internet. Paed Per Drug Ther., 7., 2., 89 112 .
  45. 45. Palen T. E. Raebel M. Lyons E. 2006 Evaluation of laboratory monitoring alerts within a computerized physician order entry system for medication orders. Am J Manag Care, 12., 7.,389 395 .
  46. 46. Paterno M. D. Maviglia S. M. Gorman P. N. et al. 2009 Tiering drug-drug interaction alerts by severity increases compliance rates. J Am Med Inform Assoc., 16.,40 46 .
  47. 47. Pope C. Mays N. 1995 Reaching the parts other methods cannot reach: an introduction to qualitative methods in health and health services research. BMJ ;311 42 5
  48. 48. Rahmner P. Gustafsson L. L. Holmstrom I. 2009 Who´s job is it anyway-Swedish general practitioners´perception of their responsibility for the patient´s drug list. Ann Fam Med; in press.
  49. 49. Rahmner P. B. Gustafsson L. L. Larsson J. et al. 2009 Variations in understanding the drug-prescribing process: a qualitative study among Swedish GPs. Fam Pract., 26., 2., 121 127 .
  50. 50. Revere D. Turner A. M. Madhavn A. et al. 2007 Understanding the information needs of public health practitioners: A literature review to inform design of an interactive digital knowledge management system. J Biomed Inform., 40.,410 421 .
  51. 51. Schiff G. D. Rucker D. 1998 Computerized prescribing. Building the electronic infrastructure for better medication usage. JAMA, 279., 1024 1029 .
  52. 52. Seidling H. M. Al Barmawi A. Kaltschmidt J. et al. 2007 Detection and prevention of prescriptions with excessive doses in electronic prescribing systems. Eur J Clin Pharmacol., 63., 12., 1185 1192 .
  53. 53. Shah N. R. Seger A. C. Seger D. L. et al. 2006 Improving acceptance of computerized prescribing alerts in ambulatory care. J Am Med Inform Assoc., 13., 1., 5 11 .
  54. 54. Shemeikka T. Gustafsson L. L. Korkamz S. 2008 Following a Lex Maria case: safe computer support systems for drug prescribing required. In Swedish. Läkartidningen 105., 3., 3177 3178 .
  55. 55. Sjöborg B. Bäckström T. Arvidsson L. B. et al. 2007 Design and implementation of a point-of-care computerized system for drug therapy in Stockholm metropolitan health region-Bridging the gap between knowledge and practice, Int J Med Info, 76., 7., 497 506 .
  56. 56. Spina J. R. Glassman P. A. Belperio P. et al. 2005 Clinical relevance of automated drug alerts from the perspective of medical providers. Am J Med Qual, 20., 1., 7 14 .
  57. 57. Stephenson J. Imrie J. 1998 Why do we need randomised controlled trials to assess behavioural interventions? BMJ., 316., 611 613 .
  58. 58. Sweidan M. Reeve J. F. Brian J. A. 2009 Quality of drug interaction alerts in prescribing and dispensing software. Med J Aust 190., 5., 251 254 .
  59. 59. Tamblyn R. Huang A. Taylor L. et al. 2008 A randomized trial of the effectiveness of on-demand versus computer-triggered drug decision support in primary care. J Am Med Inform Assoc., 15., 430 438 .
  60. 60. Van Rosse F. Maat B. Rademaker C. M. A. et al. 2009 The effect of computerized physician order entry on medication prescription errors and clinical outcome in pediatric and intensive care: A systematic review, Pediatrics, 123., 1184 1190 .
  61. 61. Weingart S. N. Massagli M. Cyrulik A. et al. 2009 Assessing the value of electronic prescribing in ambulatory care: A focus group study. Int J Med Inform. 78., 9., 571 578 .
  62. 62. Wears R. L. Berg M. 2005 Computer technology and clinical work: still waiting for Godot. Jama 293., 10., 1261 1263 .
  63. 63. Wolfstadt J. I. Gurwitz J. H. Field T. S. 2008 The effect of computerized physician order entry with clinical decision support on the rates of adverse drug events: A systematic review. J Gen Intern Med., 23., 4., 451 458 . Jama., 299., 6., 685-687.
  64. 64. Wu A. W. Lipshutz A. K. M. Pronovost P. J. 2008 Effectiveness and efficiency of root cause analysis in medicine. Jama 299., 6., 685 687 .

Written By

Birgit Eiermann, Pia Bastholm Rahmner, Seher Korkmaz, Carina Landberg, Birgitta Lilja, Tero Shemeikka, Aniko Veg, Björn Wettermark and Lars L Gustafsson

Published: 01 January 2010