Evolution of Healthcare

J.A. Muir Gray [1] summarized the evolution of evidence-based healthcare in three different steps:

1. Doing things cheaper: during the 1970s the increasing financial pressure on the national healthcare systems forced the research for reducing economic cost per case: healthcare was to be delivered in the shortest time, in the least expensive environment, by the least expensive professional, using the cheapest drugs or equipment sufficient to ensure effectiveness and safety. In one word: efficiency was the main relationship between inputs and outcomes. Economic evaluation studies and efficiency indicators begun to appear in the healthcare literature.
2. Doing things right: during the 1980s, besides efficiency, there was a new imperative, the improvement of the healthcare delivery quality. As patients become more informed, more assertive and better organized, their expectation increased. Putting together the two paradigms, doing the things cheaper and doing the things better, fosters the new paradigm of doing things right.
3. Doing the right things right: in the last decades the overall objective in healthcare is to do more good than harm. The key factor is the level of scientific evidence which support a medical action. This level is determined by the quality of the research on which the evidence is based.

Evidence Based Medicine (EBM)

The practice of EBM means integrating individual clinical expertise with the best available external clinical evidence from systematic research. By individual clinical expertise we mean the proficiency and judgment that individual clinicians acquire through clinical experience and clinical practice. Increased expertise is reflected in many ways, but especially in more effective and efficient diagnosis and in the more thoughtful identification and compassionate use of individual patients' predicaments, rights, and preferences in making clinical decisions about their care. By best available external clinical evidence we mean clinically relevant research, often from the basic sciences of medicine, but especially from patient centered clinical research into the accuracy and precision of diagnostic tests (including the clinical examination), the power of prognostic markers, and the efficacy and safety of therapeutic, rehabilitative, and preventive regimens, as determined by clinical trials. External clinical evidence may both invalidate previously accepted diagnostic tests and treatments and replace them with new ones that are more powerful, more accurate, more efficacious, and safer.

Clinical Practice Guidelines

One of the main tools of EBM is represented by the Clinical Practice Guidelines (CPGs). CPGs are a means to improve the quality of delivered health care:

CPGs are one of the key point of the EBM [3] referring to the management of individual patients through individual clinical expertise integrated with the judicious use of current best evidence from clinical care research. CPGs are a good example of KM, where the process of systematic review of the scientific literature is a source of knowledge to be integrated into and complemented by local, practice-based evidence for individual and site-specific clinical decision making.

There are several official organizations publishing CPGs with the aim of diffusing them within the medical community. Most of them offer CPGs on the web in textual format with some useful hyperlinks. One of the most important organization providing CPGs access online is the National Guideline Clearinghouse (NGC). Through the website it provides access to CPGs resources through a list organized by pathology or through searching criteria. The NGC mission is to provide physicians, nurses, and other health professionals, health care providers, health plans, integrated delivery systems, purchasers and others an accessible mechanism for obtaining objective, detailed information on clinical practice guidelines and to further their dissemination, implementation and use.

Key components of NGC include:

• structured abstracts (summaries) about the CPG and its development
• links to full-text CPG, where available, and/or ordering information for print copies
• palm-based PDA downloads of the complete NGC summary for all CPGs represented in the database
• CPG comparison through different approaches and criteria
• an electronic forum, NGC-L for exchanging information on CPGs, their development, implementation and use
• an annotated bibliography database where users can search for citations for publications and resources about CPGs, including guideline development and methodology, structure, evaluation, and implementation

Synthetizing the Evidence

As CPGs are evidence-based, it is fundamental to grade all the recommendations on the basis of the predictive power of the study designs from which recommendations are obtained. In other words, the grading assigned to a recommendation is related to the likelihood that, if that recommendation is implemented, the predicted outcome will be achieved. This is the reason why good CPGs provide at least the “Level of Evidence” for each provided recommendation.

Several grading systems have been proposed by different organizations and institutions. An example is provided by Scottish Intercollegiate Guidelines Network (SIGN ), [4, 5] SIGN formerly used the levels of evidence developed by the US Agency for Health Care Policy and Research [6] (AHCPR, now the US Agency for Health Research and Quality, AHRQ). However as a number of limitations was becoming apparent in that system, a review was carried out and new levels of evidence and associated grades of recommendation were developed. Following extensive consultation and international peer review, the new grading system was introduced in Autumn 2000. The new grading system is shown in the following table:

From the levels of evidence it is possible to derive the more compact and operative classification: “Grades of Recommendation”:

SIGN provides also a way to mark the “Good practice points” which are important practical points that the guideline development group wish to emphasize but for which there is not, nor is there likely to be, any research evidence. These are not an alternative to evidence-based recommendations, and should only be used where there is no other means of highlighting the issue. A similar grading system has been adopted by the National Institute for Clinical Excellence (NICE), the independent organization responsible for providing United Kingdom national guidance on the promotion of good health and the prevention and treatment of diseases. On 1st April 2005 NICE joined the Health Development Agency to become the new National Institute for Health and Clinical Excellence. The grading system adopted is still based on the grading system proposed by the US Agency for Health Care Policy and Research. Which has been adapted from the Agency for Healthcare Policy and Research (AHCPR) system US Department of Health and Human Services, Public Health Service, Agency for Health Care Policy and Research (1992). Acute pain management: operative or medical procedures and trauma. Rockville MD: Agency for Health Care Policy and Research Publications.

Strength of Recommendation

Another interesting CPGs grading is the “strength of recommendation”. The strength (or grade) of a recommendation for clinical practice is based on a body of evidence (typically more than one study). This approach takes into account the level of evidence of individual studies; the type of outcomes measured by these studies (patient-oriented or disease-oriented); the number, consistency, and coherence of the evidence as a whole; and the relationship between benefits, harms, and costs.

An example of strength of recommendations criteria is depicted in the following table. It is an adaptation coming from the Canadian guide to clinical preventive health care [7] and from US preventive services resources [8]:

Traditional Clinical Practice Guidelines Lifecycle

It is now clear that CPGs must be evidence-based. Normally they are the result of a well defined methodology, usually adopted at organization level in order to guide the development of all the CPGs within that organization. The accepted criteria for validity of guidelines have evolved from the 'essential elements of good guidelines' identified by the US Institute of Medicine in 1990 [9]. These recommended 'attributes of good guidelines' included validity, reliability, clinical applicability, clinical flexibility, clarity, multidisciplinary process, scheduled review, and documentation. The recommendations were underpinned by the twin themes of credibility and accountability:

Most of the methods for CPG development, such as the already mentioned SIGN's original Criteria for Appraisal of Clinical Guidelines for National Use [10], and the more recent AGREE (Appraisal of Guidelines, Research and Evaluation for Europe) guideline appraisal instrument are based on these founding principles of guideline development [11].

Without going into the deepest details, the CPG development process or life cycle can be summarized as in figure 2.6 in which it is quite clear that the dissemination of CPGs is under textual form (books or nowadays hypertext documents). An example of a detailed CPG development process consists in the overview of the SIGN guideline development process in figure 3.

A common scenario

Thus, a common scenario is represented by a clinician with several books on the desk or in the some cases with hypertext documents on a computer for a faster retrieval of the needed information and possibly a faster and cheaper dissemination. Unfortunately, traditional textual CPGs may lead to different class of problems in clinical practice:

• The interpretation may be not unique. CPGs can be semantically very complex and to have a clear understanding of health care tasks sequence is not always easy, as paths are distributed in the whole text. The interpretation of CPGs involves the generation of inferences that are needed for correctly implementing the procedure prescribed by the guideline. The fact that interpretation requires making inferences may discover ambiguities of textual CPGs, in other words it often happens that they are prone to different interpretations by different physicians;
• Organisational constraints limit the implementation of the official CPGs. CPGs are distributed as medical models and they don't usually take care about organization characteristics. In order to be effective it is often needed to adapt the generic model to a specific reality. This adaptation process is hardly performed on textual CPGs, than most of the times the adaptation will be performed, implicitly, by the physician himself, during the CPGs usage. Such changes often are not documented explicitly;
• Health care operators often do not fully comply with CPGs for several reasons and it is really important to keep track of the motivations behind non compliances. Performing this process on the paper is quite ineffective over a long time period;
• The health care process must be contextualized on a specific patient. Certainly textual formats cannot change their shape in front of the patient problems and needs.

For these reasons and for producing alerts and remainders or in general patient-tailored suggestions, it is necessary to computerize CPGs.

Computerizale Clinical Practice Guidelines

In the last years, many research groups put efforts in the computerized CPGs field and developed tools for the representation and execution of CPGs [13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24], and recently two comprehensive papers has been published about the comparison of the different formalisms and tools [25, 26]. The computerization of guidelines could lead to several improvements with respect to the traditional textual format: the text disambiguation, the possibility of viewing the guideline at different levels of detail without loosing the entire view, and the possibility of generating patient-tailored suggestions. This could help physicians to better comply with CPGs. However, recent literature shows that these potential benefits are hardly achieved by the CPGs everyday usage [27, 28]. In particular, it has been argued that the CPGs failure to support medical decision making could be due to the poor user's willingness to use them [29]. In turn, poor willingness could derive from a low integration with the existing legacy systems and a low consideration of organizational and human aspects, the so called "socio-technological issues". From a KM point of view, computerized CPGs are models trying to make the knowledge embedded in textual CPGs as explicit as possible. It is in fact a codification that aims to put medical knowledge into a form that makes it accessible to those who need it. Codification literally puts knowledge into a code to make it as organized, explicit, portable and easy to understand as possible. Obviously, whichever method of codification will be used (singly or in combination) within an organization, it will lead to value and in the same time to limitations. Thus, new paradigms and technologies for supporting knowledge codification activities are necessary.

Computerizale Clinical Practice Guidelines Lifecycle

The CPGs, resulting from a complex process based on scientific evidence and experts' consensus, are normally published in unstructured narrative form in textual or electronic (hypertext) documents. The first step for creating a decision support system is their representation into a computational format. Through an authoring tool, these models will be created once, disseminated and adapted locally [30] on the basis of the needs. The enactment systems will implement these models and help the clinicians in their activity, meanwhile keeping track, in a "logs" database, of all the tasks carried out for each individual patient. This controlled loop will help locally tuning the CPG and, at a higher level, suggest modification of the original guideline.

Computerized Clinical Practice Guidelines Systems Review

Two detailed reports about CPGs systems has been published in 2004 on JAMIA [31] and on Artificial Intelligence in Medicine [32]. I introduce the following systems:

1. Asbru [33] is being collaboratively developed at Ben Gurion University and the Vienna University of Technology. It is a time-oriented, intention-based, skeletal-plan specification language that is used to represent clinical protocols [34]. Skeletal plans capture the essence of a procedure, but leave room for execution-time flexibility in the achievement of particular intentions. Asbru’s developers have enriched skeletal plans by (1) characterizing plan attributes such as intentions, conditions, and effects, (2) adding a rich set of ordering of plans, and (3) defining temporal dimensions of states and plans. Uncertainty in temporal scope and parameters can be expressed by bounding intervals.
2. EON [35] was developed at Stanford University and provides a suite of models and software components for creating guideline-based applications. It views the guideline model as the core of an extensible set of models, such as a model for performing temporal abstractions. EON uses a task-based approach to define decision-support services that can be implemented using alternative techniques [36]. Its guideline execution server uses formalized clinical guidelines and patient data to generate situation-specific recommendations. A temporal data mediator supports queries involving temporal abstractions and temporal relationships. A third component provides explanation services for other components [37].
3. GLIF [38] the Guideline Interchange Format version 3, has been collaboratively developed by groups at Columbia, Stanford and Harvard Universities (the InterMed Collaboratory). GLIF stresses the importance of sharing guidelines among different institutions and software systems. GLIF tries to build on the most useful features of other guideline models, and to incorporate standards that are used inhealth care. Its expression language was originally based on the Arden Syntax [39] and its default medical data model is based on the HL7 Reference Information Model (RIM) [40]. A subsequent object-oriented language, GELLO [41], is being refined for consideration as an HL7 standard.
4. PRODIGY [42] was developed at the University of Newcastle upon Tyne. It provides support for chronic disease management in primary care. The PRODIGY project’s aim is to produce the simplest, most readily comprehensible model necessary to represent this class of guidelines. Teams of clinicians have used Protégé’s knowledge engineering environment [43] to encode three complex chronic –disease management guidelines. Over 150 guidelines encoded in PRODIGY’s simpler Release One model have been translated into the current model. Two vendors have integrated identical PRODIGY components into their clinical information systems for general practitioners [44].
5. PROforma [45] was developed at the Advanced Computation Laboratory of Cancer Research, UK. It combines logic programming and object-oriented modeling and is formally grounded in the R2L Language [46]. One aim of the PROforma project is to explore the expressiveness of a deliberately minimal set of modeling constructs. PROforma supports four task types: actions, compound plans, decisions, and enquiries. All types share attributes describing goals, control flow, preconditions, and post-conditions. The simple task ontology should make it easier to demonstrate soundness and to teach the language to encoders.
6. GASTON [47, 48] was developed by a joint effort of the Department of medical Informatics of Maastricht University and the Signal Processing Systems group of the Eindhoven University of Technology. The guideline representation formalism uses a frame-based model as an underlying mechanism. The formalism is non-monolithic, meaning that it can be extended with additional classes to capture new guideline characteristics. Similar to the GLIF, EON and Guide approaches, the Gaston guideline authoring environment represents and visualizes guidelines by temporally sequenced graphs (flowcharts) of frame instances from the guideline model.
7. GLARE [49] was developed by the Dipartimento di Informatica, Università del Piemonte Orientale "Amedeo Avogadro", Alessandria, Italy, in co-operation with the Laboratorio di Informatica Clinica, Azienda Ospedaliera S. Giovanni Battista, Torino, Italy. The formalism consists of a limited, but very focused and clearly understandable set of primitives. It is made up of different types of actions: plans (i.e. composite actions, hierarchically decomposable in their sub-actions) and atomic actions. Atomic actions can be queries, decisions, work actions and conclusions. All actions are linked by control relations (e.g. sequence, alternative, repetition), defining their order of execution.
8. SAGE & Guide will be introduced later on as both enable the integration with the workflow of care process.

Updated news about methods and tools to support the computerization of clinical practice guidelines are constantly published by OpenClinical. OpenClinical website (www.openclinical.org) provides an increasingly comprehensive set of resources on advanced knowledge management methods, technologies and applications for healthcare (methods and tools for representing computerised clinical guidelines). It is a non-profit organisation co-ordinated by an international group from leading research and teaching organisations.

Organizational Knowledge

The goal of knowledge management is not only increasing the performance of individuals within the organization but of the organization as a 'whole'. It implies an organizational view of the problem where individuals cooperate, as much as possible carrying on evidence-based care processes, each behaving according to the role the organization assigned her/him [50]. Thus, only cooperative care processes can attain the goals, in terms of efficiency, effectiveness and quality of care, the organization planned to achieve [51]. A strong contribution in this direction is coming in these years from the workflow area. A Workflow Management System (WfMS) is defined by the Workflow Management Coalition [52], a non-profit, international organization, as "the automation of a business process, in whole or part, during which documents, information or tasks are passed from one participant to another for action, according to a set of procedural rules". Taking an organizational view of evidence-based care processes, Stefanelli proposed the definition of "Careflow Management Systems (CfMS)" to indicate similar support systems in medical domains where guidelines have been developed and disseminated.

Careflows Management Systems

Careflow Management Systems (CfMSs) are WfMSs supporting medical knowledge management in clinical domains. They are case-based, i.e., every piece of work is executed for a specific patient. One can think of a patient care process as a Cf instance. The goal of CfMSs is to handle patients by executing medical tasks in a specific order. A Cf process definition specifies which tasks need to be executed and in what order. A task, which needs to be executed for a specific case, is called a work item. Most items are executed by a resource, either human or technological. A work item executed by a resource is called activity. To facilitate the allocation of work items, resources can be grouped into classes. The resource class based on the capabilities (i.e., functional requirements) of the HCOs members is called organizational agent. If the classification is based on the structure of the HCO, such a resource class is called organizational unit (e.g., team, laboratory, clinic, department, etc.). A CfMS may also contribute to solve the communication problem within HCOs since it is able to manage automatically a great amount of communication acts among organizational agents involved in patient care. Such view was stimulated by the "continuum view" developed by Enrico Coiera [53]: he pointed out that communication and computation tasks are related, but drawn from different parts of a task space. We strongly believe that knowledge management, in general, and CfMS, in particular, may provide an effective approach to overcome the false dichotomy between communication and computation tasks: careflow technology can be used to make communication more efficient by supporting organizational agents in sharing the needed medical and organizational knowledge. Figure 1 shows the basic elements of the organization ontology on which the model of an HCO can be based on. It represents an adaptation of the organization ontology developed within the TOVE project [54]. HCO consists of a set of organizational units (e.g., wards, laboratories, clinical units, rehabilitation units, etc.), a set of organizational agents (members of an organizational unit), a set of roles that the members play in the organization, and a set of organizational goals that they are committed to achieve.

Part of the CfMS methodology consists in the definition of the roles according to the ontology presented in figure 5. A good example of CfMS application is given by the implementation of a system for Stroke patients care developed as a collaboration between our laboratory and the Neurological Hospital "IRCCS C. Mondino" of Pavia [55, 56]. Stroke is the commonest cause of adult disability and the third leading cause of death in most countries. It is crucial therefore that an effective strategy for prevention and treatment of stroke is implemented. Effective care of stroke requires teams of efficient, informed health care professionals, who must work together in carefully planned patterns appropriate to the problems posed by individual patients. The following four main phases can be distinguished:

• Initial management at the symptoms onset
• Early management of acute stroke in the Emergency Department and in Stroke Units or Neurological Departments
• Rehabilitation and follow-up
• Secondary prevention plans

A stroke care management system must match the HCO's resources with generally agreed upon guidelines (evidence-based medicine), for each of the clinical presentations of the disease. The system should also show that its patterns of care are effective and efficient.

As an example of organisational knowledge representation, figure 6 depicts the taxonomy of roles played by organizational agents involved into the post-stroke rehabilitation process within a rehabilitation hospital [55]. Stroke rehabilitation frequently involves the services of several rehabilitation disciplines. The skills required depend on the nature of the patient’s deficits. Medical specialties that are commonly involved include physical medicine and rehabilitation, speech therapy, neurology, geriatrics, internal medicine, psychiatry, and family practice. Consulting physicians from other specialties (for example, cardiology, haematology, etc.), are called on as needed. Figure 6 shows the hierarchy of roles played by the organizational agents directly involved in the rehabilitation process. They operate within either clinical wards or rehabilitation units, but they may also be involved in several multidisciplinary activities, mainly whenever some collective decisions have to be made (e.g., identification of rehabilitation needs, assessment of deficits’ recovery).

The Stroke CfMS project is at the moment one of the most paradigmatic examples of clinical workflow. However, it requires a complete model of the HCO and medical and organizational knowledge have been represented and disseminated through the same formalism (precisely Oracle WorkflowTm). This leads to scalability and maintenance problems:

• every time new scientific evidence updates the CPGs, the model as to be updated as a whole: it is not possible to update only medical knowledge without having a complete idea of organizational issues;
• evidence-based CPGs are usually disseminated at regional or national level and with small adaptations they can be used in every HCOs; thus, HCOs models may contains few recommendations that are not evidence-based but peculiar to a given organization. Thus, the life cycles of the two type of knowledge are different and managed by people with different skills;
• CPGs management systems and Workflow management systems are complex themselves and putting them together leads to an incredibly high complexity.

SAGE and Guide for “workflow awareness”

SAGE and Guide implemented a different approach that can be characterized as “workflow aware”. They both have defined models for integration of guideline-based decision support with the workflow of care process. In both cases medical and organizational issues are separated and interaction is carried out by communication contracts and the connection with the EPR is performed through a Virtual Medical Record [57].

In the SAGE project the guideline decision support system will not be in control of host systems’ workflow management but it will respond to opportunities for decision support in the care process. SAGE needs to model enough of the workflow contexts to recognize appropriate events that should trigger decision support services. Thus, guideline knowledge must support operations in an eventdriven reactive system (unidirectional communication) and it must take into account clinical and organization contexts such as care setting and provider roles. Instead of just creating an electronic version of a clinical practice guideline, SAGE formalizes guideline knowledge to be used in specific scenarios and settings. Therefore, in the SAGE modelling approach, it is not required to model detailed workflow.

In the Guide approach, a Guideline Management System (GLMS) interacts with a given WfMS8 through messages according to specific contracts. Communication is bidirectional, it is possible that the GLMS asks to the WfMS for performing some tasks and vice versa it is possible for the WfMS to ask for medical decision support. WfMS has to be developed by third parties with the desired level of detail and it is also possible to run the GLMS without a WfMS. Ontologies and terminologies together with a set of data-types are wrapped by contracts and represent the common ground, fundamental for communication between the different areas. The strong separation between medical and organizational issues allows improving not only the software product (reducing crosstalk, powering parallelization and promoting integration with legacy systems provided by different suppliers) but also the quality of interaction between the system and the domain experts, who usually have different skills and background (improvement of socio technological issues).

The Knowledge Conversion Process in the Healthcare Domain

The knowledge conversion process, contextualized for a health care organization [58], is based on some assumptions. First of all three main knowledge types are involved:

• Medical knowledge: it manages all medical aspects related to health care activities;
• Organizational knowledge: it manages organizational aspects related to health care activities;
• Formalization knowledge: it manages knowledge representation formalisms, among which computer-based tools (it can be considered a part of the technical domain above defined).

The mentioned knowledge areas are not necessarily managed by three different kind of agents. In fact, it is possible that the an agent is responsible for more than one area (e.g.: a ward responsible that also is able to formalize a CPG is both medical and formalization expert). However, in order to simplify the dissertation we will consider a situation in which the three knowledge fields are managed by different agents: medical expert, organizational expert and formalization expert. Another important issue is represented by formal models. There are three main kinds of formal models:

• Medical, that means based on medical knowledge, e.g. formalized CPGs and protocols;
• Organizational, based on organizational knowledge, e.g. nurses’/physicians’ shifts scheduling and administrative procedures;
• Common, based on common knowledge, e.g. organizational charts (charts defining roles and hierarchies within hospital personnel).

Putting together knowledge areas and formal models, it is possible to describe a contextualized knowledge conversion process, as in Figure 4.1. Related to the left part of the figure (medical area), we can imagine that specific CPGs (medical models) have been implemented in a hospital ward. Of course, physicians will act by mixing the knowledge contained in the models (explicit medical knowledge) with their skills and experience (tacit medical knowledge) and, in special cases, they will not comply with CPGs. Such process (internalization) will lead each expert to be more and more efficient in evaluating the formal models and possibly criticizing them. Therefore, opinions about CPGs will be informally shared among colleagues (socialization). Once an agreement is found, with the help of the knowledge engineer it will be formalized, so updating the initial model (externalization). Combining models and other explicit knowledge, models can be reformulated and new explicit knowledge may be created (combination). If instead of medical CPGs we consider hospital reimbursement rules (organizational models) a similar knowledge transformation process could be individuated for the right part of Figure 7.

The externalization of common models requires the socialization between different experts. Specific skill of the knowledge engineer is to elicit the expert knowledge in order to assure quality, consistency and usability of the model. During this process, the expert (physician or administrator) learns how to organize his knowledge according to the representation power of the available computational tools (socialization between expert and the knowledge engineer). It is clear that domain experts and engineers must have a sufficient common ground to be able to communicate (e.g. about terminology, basic computational tool features etc.).

References

[1] J.A Muir Gray. Evidence-based Healthcare: How to Make Health Policy and Management Decisions. Churchill Livingston, 1997

[2] L Sackett, William MC Rosenberg, JA Muir Gray, R Brian Haynes, W Scott Richardson. Editorial from the British Medical Journal. BMJ 1996; 312: 71-2

[3] Sim I, Gorman P, Greenes R, Haynes RB, Kaplan B, et al. Clinical decision support systems for the practice of evidence-based medicine. Journal of the American Medical Informatics Association 2001; 8(6):527-34

[4] Harbour R, Miller J. A new system for grading recommendations in evidence based guidelines. BMJ 2001;323:334-6

[5] Rockville (MD): US Department of Health and Human Services.Clinical Practice Guideline No.1: acute pain management: operative or medical procedures and trauma. Agency for Health Care Policy and Research; 1993. AHCPR Publication No. 92-0023

[6] Scottish Intercollegiate Guidelines Network (SIGN). Methodology Review Group. Report on the review of the method of grading guideline recommendations. Edinburgh; SIGN: 1999

[7] The Canadian Guide to Clinical Preventive Health Care. Recommendations by Strength of Evidence. Accessed March 12, 2003

[8] US Preventive Services Task Force. Translating evidence into recommendations. Accessed March 6, 2003

[9] Field MJ, Lohr KN (Editors). Clinical practice guidelines: directions for a new program. Committee to Advise the Public Health Service on Clinical Practice Guidelines

[10] Scottish Intercollegiate Guidelines Network (SIGN). Clinical guidelines: criteria for appraisal for national use. Edinburgh; SIGN: 1995

[11] Appraisal of Guidelines, Research and Evaluation for Europe website. http://www.agreecollaboration.org/

[12] SIGN 50:A guideline developers' handbook website. http://www.sign.ac.uk/guidelines/fulltext/50/index.html

[13] Quaglini S, Dazzi L, Gatti L, Stefanelli M, Fassino C, Tondini C. Supporting tools for guideline development and dissemination. Artificial Intelligence in Medicine 1998; 14:119-137

[14] Quaglini S, Stefanelli M, Cavallini A, Micieli G, Fassino C, Mossa C. Guideline-based Careflow Systems. Artificial Intelligence in Medicine 2000; 20(1) 5-22

[15] Ciccarese P, Caffi E, Boiocchi L, Quaglini S, Stefanelli. A Guideline Management System. Medinfo 2004, IOS Press, pag. 28 - 32 (2004)

[16] A. A. Boxwala, M. Peleg, S. W. Tu, O. Ogunyemi, Q. Zeng, D. Wang, V. L. Patel, R. A. Greenes, E. H. Shortliffe. GLIF3: A Representation Format for Sharable Computer-Interpretable Clinical Practice Guidelines. Journal of Biomedical Informatics (JBI), 37 (3): 147-161 2004.

[17] S. W. Tu, M. A. Musen. The EON Model of Intervention Protocols and Guidelines. 1996 AMIA Annual Fall Symposium, Washington, D.C., 587-591. Hanley & Belfus, 1996.

[18] Bury J., Fox J., Sutton D. The PROforma guideline specification language: progress and prospects. Proceedings of the First European Workshop, Computer-based Support for Clinical Guidelines and Protocols (EWGLP 2000), Leipzig 13-14 Nov. 2000.

[19] Sutton DR, Fox J. The Syntax and Semantics of the PROforma guideline modelling language. J Am Med Inform Assoc. 2003 Sep-Oct;10(5):433-43.

[20] Purves IN, Sugden B, Booth N, Sowerby M. The PRODIGY project - the iterative development of the release one model. Proc AMIA Symp. 1999;:359-63.

[21] Johnson P, Tu S, Jones N. Achieving reuse of computable guideline systems. Medinfo. 2001;10(Pt 1):99-103.

[22] Shahar, Y., Miksch, S., and Johnson, P. The Asgaard project: A task-specific framework for the application and critiquing of time-oriented clinical guidelines. Artificial Intelligence in Medicine (14): 29-51, 1998.

[23] J. R. Campbell, S. W. Tu, J. G. Mansfield, J. I. Boyer, J. McClay, C. Parker, P. Ram, S. M. Scheitel, K. McDonald. The SAGE Guideline Model: A Knowledge Representation Framework for Encoding Interoperable Clinical Practice Guidelines. 2003.

[24] P. Ram, D. Berg, S. W. Tu, J. G. Mansfield, Q. Ye, R. Abarbanel. Executing Clinical Practice Guidelines using the SAGE Execution Engine. 2003.

[25] Peleg M, Tu S, Ciccarese P, Kumar A, Quaglini S, Stefanelli M et al. Comparing models of decision and action for guideline-based decision support: a case-study approach. JAMIA 2003; vol. 1 ,n.10, 52-68

[26] P.A. de Clercq, J.A. Blom, H.H. Korsten and A. Hasman. Approaches for creating computer-interpretable guidelines that facilitate decision support. Artif Intell Med. 2004; 31(1): 1-27

[27] Barner E. S., Baker S, Funkhouser E. et al. Do local opinion leaders augment hospital quality improvement efforts? A randomized trial to promote adherence to unstable angina guidelines. Med Care 2003; 41; 420-431.

[28] Eccles M., McColl E. et al. Effect of computerized evidence based guidelines on management of asthma and angina in adults in primary care: cluster randomised controlled trial. BMJ 2002 vol. 325.

[29] Lazarsfeld PF, Merton RK. Friendship as social process: a substantive and methodological analysis. In Berger M, et al. (eds.). Freedom and Control in Modern Society. New York: Octagon, 1964.

[30] D.B. Fridsma ,J.H. Gennari, M.A. Musen. Making generic guidelines site-specific. Proc. AMIA 1996, 597-601.

[31] Peleg M, Tu S, Bury J, Ciccarese P. et al. Comparing models of decision and action for guideline-based decision support: a case-study approach. JAMIA 2003; vol. 1 ,n.10, 52-68

[32] Clercq, P.A. de; Blom, J.A.; Korsten, H.H.M.; Hasman, A. Approaches for creating computer-interpretable guidelines that facilitate decision support. Art. Intell. Med. 31, nr. 1, 2004, pp. 1-27. SPS-40 [06.01]

[33] Shahar, Y., Miksch, S., and Johnson, P. The Asgaard project: A task-specific framework for the application and critiquing of time-oriented clinical guidelines. Artificial Intelligence in Medicine (14): 29-51, 1998.

<

[34] Seyfang A, Kosara R, Miksch S. Asbru's Reference Manual, Version 7.3. Vienna University of Technology, Institute of SoftwareTechnology, Vienna; 2002. Report No.: Asgaard-TR-2002-1.

[35] Tu SW, Musen MA. A Flexible Approach to Guideline Modeling. Proc AMIA Symp. 1999:420-424.

[36] Tu SW, Musen MA. From Guideline Modeling to Guideline Execution: Defining Guideline-Based Decision-Support Services. Proc AMIA Annu Symp. 2000:863-867.

[37] Tu SW, Musen MA. Modeling Data and Knowledge in the EON Guideline Architecture. Proc Medinfo: 280-284.

[38] Peleg M, Boxwala A, Ogunyemi O, et al. GLIF3: The Evolution of a Guideline Representation Format. Proc AMIA Annu Fall Symp. 2000:645-649.

[39] Hripcsak G, Ludemann P, Pryor TA, Wigertz OB, Clayton PD. Rationale for the Arden Syntax. Comput Biomed Res. 1994;27(4):291-324.

[40] Schadow G, Russler DC, Mead CN, McDonald CJ. Integrating Medical Information and Knowledge in the HL7 RIM. Proc AMIA Annu Fall Symp. 2000:764-768.

[41] Ogunyemi O, Zeng Q, Boxwala A. Object-oriented guideline expression language (GELLO) specification. Brigham and Women's Hospital, Harvard Medical School, 2002. Decision Systems Group Technical Report DSG-TR-2002-001.

[42] Johnson PD, Tu SW, Booth N, Sugden B, Purves IN. Using Scenarios in Chronic Disease Management Guidelines for Primary Care. Proc AMIA Annu Fall Symp. 2000:389-393.

[43] Grosso WE, Eriksson H, Fergerson R, Gennari JH, Tu SW, Musen MA. Knowledge Modeling at the Millennium (The Design and Evolution of Protege-2000). Proc 12th Banff Knowledge Acquisition for Knowledge-Based Systems Workshop. Canada; 1999:7-4-1 to 7-4-36.

[44] Johnson P, Tu S, Jones N. Achieving reuse of computable guideline systems. Medinfo. 2001;10(Pt 1):99-103.

[45] Fox J, Johns N, Rahmanzadeh A, Thomson R. PROforma: A method and language for specifying clinical guidelines and protocols. Proc Medical Informatics Europe; Amsterdam; 1996.

[46] Fox J, Das S. Safe and Sound. In: Safe and Sound: AAAI Press; 2000.<(p>

[47] De Clercq PA, Blom JA, Hasman A, Korsten HHM. Design and implementation of a framework to support the development of clinical guidelines. Int J Med Inf 2001;64(2- 3):285-318.

[48] De Clercq PA, Blom JA, Hasman A, Korsten HH. GASTON: an architecture for the acquisition and execution of clinical guideline-application tasks. Med Inform Internet Med. 2000 Oct-Dec;25(4):247-63.<(p>

[49] Terenziani P, Mastromonaco F, Molino G, Torchio M. Executing clinical guidelines: temporal issues. Proc AMIA Symp. 2000;:848-52.

[50] Stefanelli M. The socio-organizational age of artificial intelligence in medicine. Artificial Intelligence in Medicine 2001;23:25–47.

[51] Stefanelli M. Knowledge management to support performance based medicine. Methods of Information in Medicine 2002;41:36–43.

[52] URL: http://www.wfmc.org/

[53] Coiera E. When communication is better than computation. Journal of American Medical Informatics Association 2000;7:277–86.

[54] Fox MS, Gruninger M. Enterprise modeling. AI Magazine 1998;(Fall):109–21.

[55] S. Quaglini, S. Panzarasa, A. Cavallini, G. Micieli, C. Pernice, M. Stefanelli. Smooth Integration of Decision Support into an Existing Electronic Patient Record. Artificial Intelligence in Medicine ,ed Silvia Miksch; Jim Hunter; Elpida Keravnou (Eds.), pag. 89 - 93, (2005)

[56] S. Panzarasa, R. Bellazzi, C.Larizza, M. Stefanelli. A careflow management system for chronic patients. MedInfo 2004 ,ed Fieschi/Coiera e Yu-Chan Jack Li, pag. 773 - 777, (2004).

[57] Johnson PD, Tu SW, et al. A virtual medical record for guideline-based decision support. Proc AMIA Symp. 2001; pp. 294-298.

[58] Paolo Ciccarese, Ezio Caffi, Silvana Quaglini, Mario Stefanelli. Architectures and Tools for innovative Health Information Systems: the Guide Project. International Journal of Medical Informatics, vol. 74 ,n. ,pag. 553-562 ,(2005)

[59] Nonaka I, Takeuchi H. The knowledge-creating company. Oxford, UK: University Press; 1995.