Overview of success factors for implementation of advanced CDSS (see text for references)
Ten years ago, the US Institute of Medicine (IOM) called for a massive redesign of the healthcare delivery system.(Committee on Quality of Health Care in America, 2001) Today it is clear that one of the goals, the nationwide use of an electronic medical record (EMR) by 2010, has failed to be reached as the process of adoption has been slow. Some may consider an EMR as a final destination, although in fact it is only the start point of a revolution in healthcare: the implementation of clinical decision support systems (CDSS) that ‘make it easy to do it right’.(James, 2001) These systems are able to address the large, potential additional value of the implementation of an EMR. When an EMR is available, this is already a step in the right direction, to have an easy and structured access to all patient data available for all healthcare professionals that need them. However, this is still a huge amount of data, but one should also have the ability to integrate all these data and use these data in making the right choices in therapy. Practice has shown that despite the availability of an EMR, still many medication errors are made. Therefore, CDSS are designed to aid clinical decision-making by matching patient characteristics to a computerized knowledge or rule base to generate patient-specific recommendations.(Kawamoto et al., 2005) In the trendsetting IOM reports ‘To Err is Human’ (2000) and ‘Crossing the quality chasm’ (2001), a CDSS was endorsed as one of the most powerful tools available for improving patient safety and healthcare quality.(Kohn et al., 2000;Committee on Quality of Health Care in America, 2001)
It is difficult to accept that despite multiple opportunities and promising results, these systems instead of being common practice, still remain ‘next-generation’.(James, 2001)
During the last five years, research gave more insight in the success factors that could accelerate the idle process of CDSS adoption. (Kawamoto et al., 2005; Garg et al., 2005;Nies et al., 2006) The conclusions however are not univocal because the reviews included a wide variety of systems ranging from computerised to non-computerised CDSS as well as from basic to advanced systems. Basic decision support includes checking on drug-drug interactions, duplicate therapy, drug-allergies and generalized drug dosing. Advanced CDSS, used in addition to basic CDSS, includes for example checking on contra-indications (disease and drugs), individualized dosing support during renal impairment or guidance for medication-related laboratory testing.(Kuperman et al., 2007) Basic decision support is nowadays commonly available. In the Netherlands, this provision is used nationwide since 1980, integrated in a drug database (G-standaard, Royal Dutch Association for the Advancement of Pharmacy). Despite this support that is available for every physician and pharmacy, medication errors occur frequently which emphasizes the urge for advanced solutions.(Leendertse et al., 2008)
In our 600-bed university-affiliated hospital, we have implemented an advanced CDSS in practice on a few departments already and are on the eve of hospital-wide expansion. A critical examination of the literature made clear which prerequisites are needed for optimal implementation of advanced CDSS. To accelerate the process of CDSS adoption, we present an overview of success factors and barriers for implementation of advanced CDSS in hospital practice. Subsequently, we present our own experienced success factors and barriers after implementing an advanced decision support system in 2008 in daily hospital practice and compare these results with literature findings.
Medication errors occur distressingly frequent due to deficiencies in the overall system of healthcare delivery despite use of current medication safety systems.(Kohn et al., 2000;Committee on Quality of Health Care in America, 2001;Schiff et al., 2003) Many reports call attention to the gaps between optimal and actual practice. (Kohn et al., 2000;Committee on Quality of Health Care in America, 2001;Aspden et al., 2006) The report ‘To Err is human’ indicated that 44.000 to 98.000 patients die in hospitals each year due to medical errors.(Kohn et al., 2000) The recent IOM report ‘Preventing Medication Errors’(Aspden et al., 2006) showed that in the USA medications harm at least 1.5 million people per year of which at least 400,000 preventable adverse drug events (ADE’s) in hospitals.
The number of patients, dying from medical errors is probably a low estimate and the situation in Europe is not expected to be different. The Dutch statistics are, for example, not encouraging either. From the patients that died in Dutch hospitals in 2007, 10.7% experienced preventable medical complications; resulting in the death of 1735 patients (4.1%). Even more discouraging is the fact that the number of unnecessary deaths tends to increase. For the Netherlands these were 1745 deaths in 2004 and 1960 in 2008; an increase of 11.5%, despite advances in knowledge and IT-systems (Langelaan, 2010) The HARM-report (2006, NL) showed 19.000 preventable drug related admissions a year in Dutch hospitals; 5.6 % of all acute admissions, associated with a total cost of 86 million euros.(Leendertse et al., 2008)
Medication errors occur due to the rapidly increasing complexity of evidence based medicine and error sensitivity of healthcare.(James, 2002) Physicians need to take many drug- and patient specific characteristics into account and literature shows that this is often omitted or not recognized in time.(Levy et al., 1999;Schiff et al., 2003;Denekamp, 2007) Beyond reminders, CDSS can integrate clinical data to support professionals managing an increasingly complex practice environment.(James, 2001) Integration of these specific parameters is necessary to guide patients through the complete clinical pathway from anamnesis to evaluation and fine-tuning of the therapy.
Reviews of Kawamoto and Garg have shown that a CDSS is effective in decreasing medication errors and improving efficiency and quality of care.(Kawamoto et al., 2005;Garg et al., 2005) These reviews found that 64% respectively 68% of the decision support systems significantly improve clinical practice.(Kawamoto et al., 2005;Garg et al., 2005)
In literature, basic and advanced decision support are both called CDSS and exact descriptions of the systems used are scarce, which makes these systems difficult to compare. This review will only include advanced CDSS, defined as a multi-purpose rule based expert system which contains complex electronic guidelines that can integrate data from different domains.(Sucher et al., 2008) Goals of implementing advanced CDSS are to decrease errors and improve patient safety, improve quality through adoption of best practices, increase cost-effectiveness and optimize the management of chronic diseases.(Greenes, 2007) In our hospital, research is performed with clinical decision support since 1998, in which we found that these goals can be achieved though structured development, validation and implementation. The objective of this chapter is to extend a practical overview of success factors and barriers of advanced CDSS found in literature and practice, before widespread hospital implementation, concentrating on these central aspects.
3. Success factors
|Right message||Accurate content|
|Easy and actionable messages|
|Inclusion of references in the message|
|Right time||Save time|
|Integration in workflow|
|High system’s speed|
|Right place||Deliver message at the point of care|
|Active alerting mechanism|
|Right system||Electronic availability of data in the EMR|
|Integration with other systems|
|Maintenance of the system and content|
3.1. The right message
3.1.1. Accurate content
To facilitate the adoption of evidence based medicine in practice, paper guidelines have to be translated into a computer-interpretable format, called
3.1.2. Reliable messages
Current medication safety systems generate masses of irrelevant alerts leading to alert fatigue as a result of the often low specificity and clinical relevance.(van der Sijs et al., 2006) This is especially the case with basic, drug-oriented decision support, as we know in the Netherlands for about 30 years. Many alerts are generated, and the research of van der Sijs showed that more than 90% of alerts are overridden, without the physician even looking at the alert. The underlying conditions are: low alert specificity, information content unclear, low sensitivity, handling was insufficient and unsafe and the workflow was unnecessary disrupted. Also, physicians were not trained on the alerts and trusted the pharmacy in checking their medication orders. In summary, there are many failures of the systems used today, that can be solved with these new advanced decision support systems. Even, these current systems can be dangerous, which pleads for a rapid response. These basic systems increases the risk of overriding a potentially harmful alert.(Bates et al., 2003;van der Sijs et al., 2006;Denekamp, 2007) Even pharmacists have been found to override a third of life-threatening drug-drug interactions.(Bates et al., 2003) Therefore, it is vitally important to validate the clinical rules extensively before bringing these into practice to ensure reliable recommendations.(Dexter et al., 2001;Wetter, 2002;Wessels-Basten et al., 2007;Sucher et al., 2008;Varonen et al., 2008) The above mentioned obstacles need to be taken into account when designing en implementing these newer systems. Also, validation of the alerts is fundamental for success. Scheepers et al (2009) has described a strategy that can be used in various settings to create clinical rules of a high reliability. By measuring the positive and negative predictive value during the development process and afterwards, one can easily monitor if the quality of the clinical rules is high enough before implementation in practice. Since few studies have focused on validation of clinical rules, more research is needed. There is a need for other successful strategies to be described, to help others with these fundamental questions. Also, research may answer the question which minimal notification threshold is needed for effective alerting before bringing the alert into practice.(Osheroff et al., 2005)
3.1.3. Easy and actionable messages
A CDSS is effective when it minimizes the effort required by physicians to receive and act on system recommendations.(Kawamoto et al., 2005;Varonen et al., 2008) This can be accomplished by giving clear advice and a straightforward way to perform the action indicated. (Bates et al., 2003;Osheroff et al., 2005;Varonen et al., 2008;Mollon et al., 2009) Besides that, applications must anticipate physicians needs for gathering and translating data into actionable recommendations.(Bates et al., 2003;Kawamoto et al., 2005;Mollon et al., 2009) For example, a clinical rule is developed to check if a patient needs a gastro-protective drug when using a NSAID or dosage adjustment because of renal insufficiency. (Field et al., 2008) If necessary, an automatic medication order pops-up and only one click on the ‘authorize’ button is needed to prevent the patient from having adverse drug events. This is all in line with the words of James (2001) to ‘Make it easy to do it right’ and emphasises the need of an expert/end users team that is well involved in the development process.
3.1.4. Inclusion of references in the message
A valuable addition to the message is to present the source of the information and the explanation of the rationale for the guideline.(Maviglia et al., 2003;Kawamoto et al., 2005; Goud et al., 2008;Mollon et al., 2009) Many guidelines exist and these are continuously renewed, which makes it difficult to keep up with all recommendations. Although it appears that not all users use this option(Bates et al., 2003), it is a valuable addition that literature citations and web-based evidence are available when desired. (Bates et al., 2003)
3.2. The right time
3.2.1. Save time
Physicians encounter a barrier to invest time in another computer system. All health professionals face time pressure and increasing data overload. The number of therapies is so large that physicians cannot effectively keep track of them all.(Bates et al., 2003) A CDSS can save time by making associations between different data domains, which physicians might miss because of the large amount of data. Time saving can be achieved by making it easy to do it right and is therefore a highly valued success factor.(Wetter, 2002; Mollon et al., 2009) It is important to find the right balance between over- and under reporting in accordance with the wishes of the end-users of the system and to convince users that the added value of the system compensates for the time required to learn and use the system.(Goud et al., 2008)
3.2.2. Integration in clinical workflow
The most important success factor of CDSS is that the system is integrated in the clinical workflow.(Kawamoto et al., 2005;Osheroff et al., 2005;Mollon et al., 2009) A CDSS will otherwise have no beneficial effect and will not be used. Messages should be presented at the moment of decision making, though with as less disturbance for the physician as possible. Therefore different alert mechanisms (pop-up, automatic lab order, prescription order, email, etc) should be developed, suitable for different alerting priorities.(Osheroff et al., 2005) In this particular field, the research is very limited and needs to be expanded soon. Understanding the clinical workflow and user’s wishes thoroughly strongly increases the probability for success.(Bates et al., 2003)
3.2.3. High system’s speed
As with every computer system, speed is a very important parameter for user’s acceptance. (Mollon et al., 2009) As explained above, recommendations should appear exactly at the right time of decision making. When the speed of an application is slow, user satisfaction declines markedly. Bates et al found that this parameter is valued most by users and therefore it should be a priority factor of the CDSS.(Bates et al., 2003)
3.3. The right place
3.3.1. Point of care
An important quality of advanced CDSS is the ability to deliver recommendations at the point of care.(Mollon et al., 2009) The question is who should receive the message. The point of care may vary, as patients therapy is guided by various health professionals in the process. The multidisciplinary expert team can identify the receiver of the recommendations (e.g. physician, pharmacist, nurse etc).(Wessels-Basten et al., 2007)
3.3.2. Active alert mechanism
The performance of systems in which users are automatically prompted to use the system is significantly more effective, compared with studies in which users were required to actively initiate the system.(Dexter et al., 2001; Kawamoto et al., 2005; Garg et al., 2005; van Wyk et al., 2008) Also a larger positive impact is seen when a recommendation is prompting for an action and not ignorable.(Dexter et al., 2001; Bates et al., 2003) Dexter found that relatively small changes in the presentation of alerts made the difference between a significantly increase in preventive measurement rates and no effect at all.(Dexter et al., 2001) So it is important to select the alert mechanism carefully, as the type of alert will greatly affect the impact of the clinical rule.
3.4. The right system
3.4.1. Electronic availability of data
A CDSS acts on electronically available patient data in the EMR (clinical data, pharmacy, laboratory, diagnoses, complications, microbiology etc).(Soman et al., 2008) Progress of implementation and adoption of EMR is slow, which inhibits adoption of CDSS. The most specific and advanced clinical rules can be developed if a CDSS is able to gather all mentioned data from the EMR, but at least a pharmacy-laboratory link is needed. (Schiff et al., 2003; Maviglia et al., 2003; Jha et al., 2008) The ability to integrate outpatient data in the CDSS can give additional improvements for more specific clinical rules.(Goud et al., 2008)
3.4.2. Integration with other systems
Integrated systems (e.g. in CPOE) are significantly more likely to succeed than stand alone systems.(Kawamoto et al., 2005;Garg et al., 2005;Mollon et al., 2009) This provides relevant information to physicians at key times in decision making, enabling to prompt alerts during drug prescription or chart review.(Denekamp, 2007;Varonen et al., 2008)
3.4.3. Maintenance of system and content
Once a CDSS is implemented it is essential for long term success that the system and content remains up-to-date.(Trivedi et al., 2009) Technical maintenance is important to guarantee a flawless link between the CDSS and the EMR. Maintenance concerning content is needed to ensure that the clinical rules remain applicable regarding the latest evidence-based guidelines.(Osheroff et al., 2005) Sometimes corrections are required after implementation when it turns out that the impact or physician’s satisfaction is not as expected.(Trivedi et al., 2002;Bates et al., 2003) Therefore Bates et al(Bates et al., 2003) assigned each area of decision support to an individual. For example, a cardiologist has to evaluate the clinical rules regarding his specialism periodically. Regarding the novelty of complex CDSS, maintenance of the system and content is hardly studied yet.
Certain barriers found in literature may hamper implementation of advanced CDSS. Besides the lack of the described success factors, an often mentioned barrier to implementation is low computer skills among physicians.(Garg et al., 2005;Leslie and Denvir, 2007;Trivedi et al., 2009) This must be carefully taken into account within the design of the alerts. New-generation physicians, like medical students and junior doctors, may bring a higher level of computer literacy to clinical practice and stimulate implementation of a CDSS in practice. (Leslie and Denvir, 2007)
Another barrier, brought forward by Bates(Bates et al., 2003), is the loss of physician’s autonomy with the use of CDSS. However, CDSS are able to present the best evidence-based practice automatically, without requiring extra thought or work. This allows the health professionals to focus on those areas of special need and adjust care to each individual patient.(James, 2002) This not only increases patient safety, but also physician’s safety by reducing the risk on malpractice claims. Even, the system may improve clinical skills through a learn effect of the corrective messages.(Varonen et al., 2008)
5. Own experienced success factors and barriers
In our years of research we evolved a development and validation strategy for clinical rules. This three-step strategy (retrospective technical, retrospective therapeutic and prospective validation step), following a Plan-do-check-act-cycle in combination with an expert team, has led to high success rates.(Wessels-Basten et al., 2007;Helmons et al., 2009) We found that a well considered strategy, that creates rules with a high positive predictive value (PPV) is essential for accurate content and reliable messages. Application of this strategy has led to clinical rules with a reliability (PPV) of at least 94%, which indicates that more than 94% of the alerts are technically correct and clinically relevant as indicated by the expert team and end-users.(Wessels-Basten et al., 2007)
This strategy mentioned is described in detail in another publication (Scheepers, 2009) and tries to solve important barriers described earlier, especially the ones regarding content and reliability. In our hospital we use the advanced CDSS Gaston, a commercially available system described in many publications. (de Clercq, 2004). Gaston is a software system – you could also call it an intelligent agent – that consists of two parts. The first part – the guideline-editor (Fig. 1) – allows the user to build the guidelines as flowcharts in a user-friendly environment. The steps in the flowchart contain the selection definitions based on the parameters that are available in the EHR. The second part, the decision support module (DCS) is the guideline execution engine. This engine translates the guideline from the guideline-editor into a routine that can be executed.
As the expert team is the key quality of the strategy developed in our previous research, we experienced some specific factors that need to be taken into account. These aspects may help others to compose the right team in order to optimised outcomes:
Define the objective clearly. The context of the clinical rule and how it will be applied in practice should be clear for all participants.
Start by making a clear structure in which the clinical rule is developed. Some content covers only one drug, some covers a certain drug class, while others concern a complete (contra-)indication. One should try to classify the content correctly.
Develop the clinical rule in a way that is clear for all participants before start and in a structured way with a pre-defined strategy.
Compose the team of experts with experts from different departments and specialisms. Also, compose the team with people that can be trusted to have an active role in the discussion and reflect the opinion of others well.
Communicate to the experts what effort is expected form them and how much time it will cost them. How will they be rewarded for their input.
Communicate clearly how feedback will be given, in meetings or digital communication.
Determine how decisions are made when not all experts agree. Which experts are identified as having the last word or should there be a majority with the same opinion.
When opinions differ, stay objective as being the chairman of the session. Try to make a clear overview of all pro- and contra arguments and try to literature to create an agreement.
Integration of all computer systems in one environment is also found to be important to ease implementation. In our hospital, CPOE is integrated in the EMR. Only one link is needed between CDSS and EMR/CPOE which facilitates technical implementation, maintenance and costs.(de Clercq and Hasman, 2004) An example: we found that every time the EMR was upgraded to a new version, also adaptations in de CDSS were needed. Besides these two factors, we experienced that the success factors as described above are important for an effective foundation before widespread hospital implementation.
5.1. Barriers experienced in practice
When the dependence on a CDSS is growing for the checking of guidelines in a more consistent and reliable way, it is necessary that the well-functioning of the CDSS is guaranteed in the hospital. During our research we experienced a lot of barriers. For the guarantee of well-functioning, the following ‘commandments’ need to be addressed:
Stability and performance of the CDSS: When the CDSS is increasingly used, it may impact the performance of the system itself as well as other systems. It is important that the position of the CDSS is well known amongst all and that these performance problems are known, controlled and monitored continuously by the ICT-department.
Maintenance of the CDSS: Maintenance can be divided in functional and technical maintenance, with different responsibilities. It should be indicated clearly who is responsible for the content of the system and all the parameters used in these clinical rules (functional). This is usually a task for the medical professional, who builds, tests and releases these clinical guidelines. Between function and technical maintenance is the maintenance of the data and parameters used (data dictionary). When a parameter, location or database structure are changed for the information systems that feed the CDSS with data (e.g. the EHR or laboratory information system), the data dictionary of the CDSS should be changed accordingly. The use of so-called ‘free-text’ in the data feeding information systems should be minimized for as much as possible. Typo-errors or different use of text for the same concept makes it impossible to guarantee the correct functioning of the CDSS. Technical maintenance includes the maintenance of the CDSS application, installing updates and new releases of the CDSS (and EHR) and testing the technical correct functioning of the CDSS. Because a CDSS uses data from other information systems like the EHR, it is important that technical testing of the CDSS is also performed after installing updates or new releases of other information systems like the EHR. This is usually done by the ICT-department. In addition, advanced reporting-systems should be able for signalling and solving errors of the CDSS. When one is not aware of an error or malfunctioning of the CDSS, the patient safety could be at risk without anybody noticing it. The responsibility of the functional and technical maintenance of the CDSS should be clearly defined and time and budget should be available for medical professionals and the hospital IT –department.
Skills and knowledge of personnel: When there are errors concerning the CDSS, one should be able to estimate the possible risks on patient safety and immediately act upon it. This requires effort from all personnel, medical as well as IT personnel. Documentation on the guidelines, possible errors and their risks and an emergency procedure, in case of malfunctioning of the CDSS is an important aspect of the safe use of a CDSS.
A generalised format: Agreements should be made between the hospital departments about the layout, priority and content of warnings of the CDSS, the way they should respond to them and the content of guidelines. The presence of a wide variety of warnings could cause that medical professionals lose interest for them. In that case the warnings of the CDSS are not used and their additional value declines.
Continuity and support of the CDSS vendor: This support is essential for continued technical quality of the system and the validated manner of application in practice. In every setting local adaptations are needed to fulfil the demands of the CDSS customer.
Comprehensive testing protocols: These should be available for the testing of guidelines before implementation and for the testing of guidelines after new releases and updates. Also testing protocols should be available for the technical testing of the CDSS. When the number of guidelines increases in a hospital, it will cost an increasing amount of time to test all the guidelines after a new release or update. One should develop testing strategies, which allow adequate testing within a limited amount of time.
This list is not a complete list and not a completely new list: Most of the issues are also applicable for other information systems like the EHR itself. It is important that the issues on this list are addressed because downtime and malfunctioning of the EHR or the CDSS have severe implications for the hospital and its patients. One should be aware that malfunctioning of the CDSS is not immediately visible for the medical professional because it generally results in the absence of warnings. Warnings are, however, not always present, only in case of treatment differing from guidelines. The absence of warnings will therefore not automatically alarm a medical professional. Continuously monitoring of the CDSS by the IT department is therefore important, to prevent unreported malfunctioning CDSS, which is a potential risk for patient safety.
The primary finding in this chapter is that many factors determine the successful implementation of advanced decision support. Although several publications have addressed the need of detailed description of system design and implementation features, these are continuously poorly reported.(Kawamoto et al., 2005;Garg et al., 2005;Mollon et al., 2009) Also the definition of a CDSS is used confusingly, as they include a wide variety of computerized and non-computerized systems that apply basic and advanced decision support. This complicates to conclude which factors really contribute to success of advanced decision support and seriously hampers widespread implementation of these systems that are so urgently needed.
The implementation of effective clinical decision support is a challenging task and in this complex process there are no easy solutions to guarantee success or to avoid failure. When starting with a CDSS, first certain prerequisites must be met. First, nationwide adoption of an EMR must be accelerated, as this is a prerequisite for CDSS adoption. Secondly, it is important to gather the right data and create a clinical rule with high reliability with technically correct and clinically relevant messages. Essential is that an effective system must minimize the effort required by clinicians to receive and act on systems recommendations.(Kawamoto et al., 2005) Last but not least, the commitment and participation of physicians as most frequent end-users is crucial for success.(Bates et al., 2003) It is important to identify stakeholders that support the implementation and a multidisciplinary team to achieve realisation, distribution and continuation.(Osheroff et al., 2005) When these people address the value and possibilities of these systems, time and money can be made available. It is very time consuming to add knowledge to a CDSS, continuously test and validate the clinical rules and recommendations. Consequently, high costs are associated with the development of clinical rules and the cost-effectiveness of CDSS has rarely been studied yet.(Field et al., 2008)
Nevertheless, a CDSS is a powerful tool that can even fulfil ‘latent needs’. These needs, that are present but not yet realized, can be fulfilled by detecting errors that need prevention but are too time-consuming with current possibilities. An example is the clinical rule for renal function Helmons et al developed.(Helmons et al., 2009) This clinical rule checks when a patient has a decreased renal function and dose adjustments of the medication are necessary. Compared to chart review, this clinical rule decreases time needed for checking and intervention by 80%. This example illustrates that today many gaps exist between optimal and actual practice. Although a pharmacist should check all patients on the combination of drug dosing and decreased renal function, in the current situation it is impossible as time is limited. Our research shows that with every developed clinical rule, remarkable differences between guidelines and practice were found, even though guidelines are well known.(Wessels-Basten et al., 2007;Helmons et al., 2009) Prospective research must demonstrate the effect of these clinical rules in practice.
7. Conclusion and further recommendations
This chapter demonstrates a practical overview of success factors which are crucial for successful implementation of advanced CDSS to facilitate widespread implementation in hospital practice. These success factors are focused on presenting the right message, at the right time, in the right place with the right system. More adequate descriptions of system design features and implementation are needed to translate these factors to effects on patient outcomes.
CDSS are expected to unchain a revolution in healthcare and fill the existing information gap by bringing relevant data and knowledge to the point of care.(Bates et al., 2003) Despite the consciousness that change is indispensable and the years of research and effort worldwide, this has only led to slow adoption of EMR and CDSS. We believe it is now the time to bring these identified success factors in practice to accelerate the adoption of these systems and reap the benefits of this promising next-generation in medication safety.
More research is needed in most areas concerning CDSS (Kuperman et al., 2007;van Wyk et al., 2008) The focus of this research should be which alerting mechanism (list, pop-ups, e-mail, SMS, dedicated quality persons) results in the best adherence of physicians to previously agreed standards of care and how do these methods effect patient outcomes? Eventually this will give an answer to the question: how to optimally implement a CDSS in clinical practice in order to reduce errors and omissions so frequently noted.
Aspden P. Wolcott J. Bootman J. Cronenwett L. 200 2006Aspden, P., J. Wolcott, J. Bootman, and L. Cronenwett Preventing Medication Errors: Quality Chasm Series: Washington, DC, National Academy Press.
Bates D. W. et al. 2003Ten commandments for effective clinical decision support: making the practice of evidence-based medicine a reality: J.Am.Med.Inform.Assoc., 10 6 523 530
Committee on Quality of Health Care in America, 2001Committee on Quality of Health Care in America Crossing the quality chasm: a new health system for the 21st century: Washington, D.C, National Academy Press.
de Clercq P. Hasman A. 2004Experiences with the development, implementation and evaluation of automated decision support systems: Stud.Health Technol.Inform, 107no. Pt 2, 1033 1037
Denekamp Y. 2007Clinical decision support systems for addressing information needs of physicians: Isr.Med.Assoc.J., 9 11 771 776
Dexter P. R. Perkins S. Overhage J. M. Maharry K. Kohler R. B. Mc Donald C. J. 2001A computerized reminder system to increase the use of preventive care for hospitalized patients: The New England Journal of Medicine, 345 13 965 970
Field T. S. Rochon P. Lee M. Gavendo L. Subramanian S. Hoover S. Baril J. Gurwitz J. 2008Costs Associated with Developing and Implementing a Computerized Clinical Decision Support System for Medication Dosing for Patients with Renal Insufficiency in the Long-term Care Setting: Journal of the American Medical Informatics Association, 15 4 466 472
Garg A. X. Adhikari N. K. J. Mc Donald H. Rosas-Arellano M. P. Devereaux P. J. Beyene J. Sam J. Haynes R. B. 2005Effects of Computerized Clinical Decision Support Systems on Practitioner Performance and Patient Outcomes: A Systematic Review: JAMA: The Journal of the American Medical Association, 293 10 1223 1238
Goud R. Hasman A. Peek N. 2008Development of a guideline-based decision support system with explanation facilities for outpatient therapy: Computer Methods and Programs in Biomedicine, 91 2 145 153
Greenes R. A. 2007Greenes, RA. Clinical Decision Support. The Road Ahead: Boston, Elsevier Academic Press.
Helmons P. Grouls R. Roos A. Bindels A. De Clercq P. Wessels-Basten S. Ackerman E. Korsten H. 2009The potential value on medication safety of a clinical decision support system in intensive care patients with renal insufficiency: QSHC, in press.
James B. 2001Making It Easy to Do It Right: The New England Journal of Medicine, 345 13 991 993
James B. 2002Quality improvement opportunities in health care- Making it easy to do it right: J Manag.Care Pharm., 8 5 394 399
Jha A. K. Laguette J. Seger A. Bates D. W. 2008Can Surveillance Systems Identify and Avert Adverse Drug Events? A Prospective Evaluation of a Commercial Application: Journal of the American Medical Informatics Association, 15 5 647 653
Kawamoto K. Houlihan C. A. Balas E. A. Lobach D. F. 2005Improving clinical practice using clinical decision support systems: a systematic review of trials to identify features critical to success: BMJ, 330 7494 765
Kohn L. Corrigan J. Donaldson M. 200 2000Kohn, L., J. Corrigan, and M. Donaldson To Err Is Human: Building a Safer Health System: Washington, DC, National Academy Press.
Kuperman G. J. Bobb A. Payne T. H. Avery A. J. Gandhi T. K. Burns G. Classen D. C. Bates D. W. 2007Medication-related Clinical Decision Support in Computerized Provider Order Entry Systems: A Review: Journal of the American Medical Informatics Association, 14 1 29 40
Leendertse A. J. Egberts A. C. G. Stoker L. J. van den P. M. L. A. Bemt for the. H. A. R. M. Study Group. 2008Frequency of and Risk Factors for Preventable Medication-Related Hospital Admissions in the Netherlands: Archives of Internal Medicine, 168 17 1890 1896
Leslie S. J. Denvir M. A. 2007Clinical decision support software for chronic heart failure: Crit Pathw.Cardiol., 6 3 121 126
Levy M. Azaz-Livshits T. Sadan B. Shalit M. Geisslinger G. Brune K. 1999Computerized surveillance of adverse drug reactions in hospital: implementation: Eur J Clin Pharmacol, 54 11 887 892
Maviglia S. M. Zielstorff R. D. Paterno M. Teich J. M. Bates D. W. Kuperman G. J. 2003Automating Complex Guidelines for Chronic Disease: Lessons Learned: Journal of the American Medical Informatics Association, 10 2 154 165
Mollon B. Chong J. Holbrook A. Sung M. Thabane L. Foster G. 2009Features predicting the success of computerized decision support for prescribing: a systematic review of randomized controlled trials: BMC Medical Informatics and Decision Making, 9 1 11
Nies J. Colombet I. Degoulet P. Durieux P. 2006Determinants of success for computerized clinical decision support systems integrated in CPOE systems: a systematic review: AMIA Annu Symp Proc, 594 598
Osheroff J. Pifer E. Teich J. Sittig D. Jenders R. 200 2005Osheroff, J., E. Pifer, J. Teich, D. Sittig, and R. Jenders Improving outcomes with clinical decision support: an implementer’s guide.: Health Information Management and Systems Society.
Schiff G. D. Klass D. Peterson J. Shah G. Bates D. W. 2003Linking Laboratory and Pharmacy: Opportunities for Reducing Errors and Improving Care: Archives of Internal Medicine, 163 8 893 900
Soman S. Zasuwa G. Yee J. 2008Automation, Decision Support, and Expert Systems in Nephrology: Advances in Chronic Kidney Disease, 15 1 42 55
Sucher J. F. Moore F. A. Todd S. R. Sailors R. M. Mc Kinley B. A. 2008Computerized clinical decision support: a technology to implement and validate evidence based guidelines: J.Trauma, 64 2 520 537
Trivedi M. H. Kern J. K. Marcee A. Grannemann B. Kleiber B. Bettinger T. Altshuler K. Z. Mc Clelland A. 2002Development and implementation of computerized clinical guidelines: barriers and solutions: Methods Inf.Med, 41 5 435 442
Trivedi M. Daly E. Kern J. Grannemann B. Sunderajan P. Claassen C. 2009Barriers to implementation of a computerized decision support system for depression: an observational report on lessons learned in "real world" clinical settings: BMC Medical Informatics and Decision Making, 9 1 6
van der Sijs H. Aarts J. Vulto A. Berg M. 2006Overriding of Drug Safety Alerts in Computerized Physician Order Entry: Journal of the American Medical Informatics Association, 13 2 138 147
van Wyk J. T. van Wijk M. A. M. Sturkenboom M. C. J. M. Mosseveld M. Moorman P. W. van der Lei J. 2008Electronic Alerts Versus On-Demand Decision Support to Improve Dyslipidemia Treatment: A Cluster Randomized Controlled Trial: Circulation, 117 3 371 378
Varonen H. Kortteisto T. Kaila M. for the. E. B. Me Study D. S. Group 2008What may help or hinder the implementation of computerized decision support systems (CDSSs): a focus group study with physicians: Family Practice, 25 3 162 167
Wessels-Basten S. Hoeks A. Grouls R. Helmons P. Ackerman E. Korsten H. 2007Development strategy and potential impact on medication safety for clinical rules: the lithium case: Br J Clin Pharmacol, 63 4 507 508
Wetter T. 2002Lessons learnt from bringing knowledge-based decision support into routine use: Artificial Intelligence in Medicine, 24 3 195 203