Developing performance indicators for clinical governance in dimensions of risk management and clinical effectiveness

Abstract

Objective

This study has been designed and conducted to develop domestic indicators for evaluating the performance of clinical governance in dimensions of risk management and clinical effectiveness.

Design

This study implemented a 5-stage process including conducting a comprehensive literature review, expert panel (∼1000 h per person per session, 11 experts), semi-structured interviews, a 2-round Delphi study (33 experts were in attendance) and a final expert panel (8 experts were in attendance).

Setting

East Azerbaijan-Iran Province.

Participants

Fifty-six specialists and experts in different fields of medical sciences.

Main Outcome Measures

Importance and applicability of indicators.

Results

Using a thorough literature review, 361 indicators (129 risk management indicators in 4 dimensions and 232 clinical effectiveness indicators in 18 dimensions) were found. After conducting expert panels and interviews, the number of indicators decreased to 168 cases (65 risk management indicators in 4 dimensions and 103 clinical effectiveness indicators in 12 dimensions). Two rounds of Delphi identified four indicators that were omitted. The members of the final expert panel agreed on 113 indicators (43 risk management indicators in 4 dimensions and 70 clinical effectiveness indicators in 11 dimensions).

Conclusion

In this study, indicators for assessing clinical governance in domains of risk management and clinical effectiveness were designed that can be used by policy-makers and other authorities for improving the quality of services and evaluating the performance of clinical governance. Those indicators can be used with slight modifications in other countries having healthcare systems similar to that of Iran.

Background

The quality of care delivery is an increasingly important issue in healthcare sector throughout the world. Healthcare professionals aim to increase patient benefits and apply many methods to improve patient care [1, 2]. The quality of care can be defined as ‘the degree to which health services for individuals and populations increase the likelihood of desired health outcomes and are consistent with current professional knowledge’ [3]. Improving quality of healthcare requires performance measurement and feedback [4–7]. The assessment of the quality of care has become increasingly important to providers, patients, regulators and purchasers [8–12].

Using performance indicators and outcome measures is one way to measure and monitor the quality of care and services. Performance indicators and outcome measures allow the quality of care and services to be measured [13–16]. Developing and reporting indicators have become widely accepted methods for improving the quality in most countries, and several studies have been conducted to design and implement these indicators in healthcare systems [1, 17–22]. ‘Indicators usually provide a quantitative (i.e. numerical) measure that can be used as a guide to monitor and evaluate the quality of important patient and support service activities’ [23]. A literature review examining the development and implementation of indicators shows that the most of these studies have been conducted in high-income countries (HICs).

In Iran and many other low- and middle-income countries, the use of indicators in the healthcare sector has not been considered sufficiently. In Iran, due to the increased pressure on the health system, the Ministry of Health and Medical Education (MOHME) has recently implemented models and strategies to improve the quality of healthcare, such as European Foundation for Quality Management model. Since such attempts did not make a major impact, the MOHME decided to apply the NHS ‘clinical governance’ model with seven pillars including risk management, clinical effectiveness, patient and public involvement, clinical audit, education and training, staff and staff management, and use of information [24, 25].

To evaluate the performance of clinical governance, specific and appropriate indicators commensurate with the Iranian health system context are required. This study aims to develop and select indicators to evaluate the clinical governance model in the domains of risk management and clinical effectiveness.

Methods

After getting approval from the medical science board of Tabriz University, a multiphase study was designed to develop and evaluate performance indicators of the clinical governance model in hospitals in East Azerbaijan, Iran, during 2013 and 2014. The first phase of this study aimed to identify the appropriate indicators and included an extensive literature review, expert panels, semi-structured interviews and Delphi survey [26].

Literature review

A comprehensive literature review was conducted to identify the performance indicators for clinical governance, risk management and clinical effectiveness. Through the literature review, the following databases were searched for relevant studies: PubMed, Ovid Medline, Science Direct, Google Scholar, Medline, SID (a scientific information database developed to index articles published in Iranian journals) and relevant websites The keywords used included: clinical governance, indicator*, risk management*, clinical effectiveness*, hospital performance*, quality indicator*, clinical indicator*, hospital accreditation indicator*, as well as the Persian equivalents of these keywords for the time period from 1975 to 2012. All articles and reports published in English and Persian languages were also searched. Manual searches were also conducted through the index listings and peer-reviewed medical journals. Additional searches were conducted using Cochrane Collaboration and Evidence-Based Medicine, as well as clinical practice guidelines, consensus reports and best practice reports. Criteria for selecting relevant studies included indicators in hospitals (not in primary care facilities), the provision of adequate descriptive information about indicators, and extracted, summarized and reported results in appropriate extraction tables.

Expert panel

In the second phase of the study, five expert panels were held to review and adapt the indicators to the Iranian health system or hospital context. The expert panels were comprised of stakeholders, project partners and external experts.

Panel members were composed of the following individuals: Expert group members were selected according to the following criteria: The session lasted 2 h and was organized as a group discussion. Discussions were primarily carried out about the feasibility and importance of collected indicators.

• four clinician specialists (one urologist, one infectious disease specialist, one neurologist and one emergency medicine specialist),

• three nurses,

• one epidemiologist,

• one PhD with expertise in health services management,

• one director of the clinical governance department and

• one expert in hospital evaluation.

• practical knowledge of risk management and/or clinical effectiveness,

• knowledge and experience about performance indicators,

• employed in health system and

• interested in and availability to participate in the study.

Interviews

In some specific domains, due to the lack of expert knowledge, interviews were conducted with experts in intended domains. Semi-structured interviews were conducted with the following experts: The duration of the interviews varied from 30 to 90 min. Experts' comments were applied on forms provided for modifying indicators.

• physiotherapist for physiotherapy indicators,

• epidemiologist for mortality indicators,

• medical records expert (responsible for medical records and Hospital Information System (HIS)) and

• expert in hospital statistics.

Delphi survey

After conducting the expert panel and interviews, selected indicators were entered into the Delphi survey phase. Indicators were categorized into the following areas: The Delphi questionnaire/form was designed based on the extensive literature review and experts' comments. Questionnaires were sent to experts from different medical specialties and basic sciences (epidemiologist (n = 2), PhD in Health Services Management (n = 2), Urologist (n = 2), community medicine specialist (n = 2), emergency medicine specialist (n = 1), infectious disease specialist (n = 1), mental disease specialist (n = 1), medical informatics expert (n = 1), pathologist (n = 1), PhD in environmental health (n = 1), PhD in public health (n = 2), Medical Equipment expert (n = 1), nurse (n = 2), Rehabilitation specialist (n = 2), director of hospital quality improvement (n = 1) and director of hospital infection control (n = 1)). The panelists rated each indicator in terms of ‘importance’ and ‘applicability’. Respondents indicated whether they ‘agreed’, ‘had no idea’ and ‘disagreed’. Agree and disagree responses were also divided into a four-point Likert-type rating scale that ranged from ‘strongly agree/disagree’ [1] to partially agree/disagree [4] (see Supplementary Appendix 1).

• risk management including indicators of patient safety, staff safety, equipment safety and environment safety and

• clinical effectiveness including indicators of infection control, surgery, gynecology, ICU, pediatrics, pathology, oncology, ophthalmology, radiology and physiotherapy (indicators of mortality and emergency medicine were not entered into the Delphi survey phase due to the existence of the appropriate national indicators).

After calculating the mean of responses, indicators with a mean of equal to or less than 4 were omitted, indicators with the mean of 4 to 7 entered to the second Delphi phase and indicators with mean of greater than or equal to 7 were definitely accepted. To assess the validity of the questionnaire, the Content Validity Index (CVI) and modified KAPPA were assessed based upon expert perspectives. A minimum cumulative modified KAPPA score of 0.76 is expected to be achieved for the content validation [27].

Final expert panel

A final expert panel was held with participants including one epidemiologist, three clinician specialists (one urologist, one infectious disease specialist one neurologist) and four nurses. This study was approved by the ethic committee in Tabriz University of Medical Sciences. Individuals were asked to participate on a voluntary basis, and informed consent was obtained from them. Confidentiality of the study processes was assured.

Results

Through the literature review 361 indicators were identified. Five expert panels, four semi-structured interviews, two rounds of Delphi surveys and a final expert panel were conducted while panelists examined the indicators from aspects of ‘importance’ and ‘applicability’. Ultimately, several indicators were excluded, merged, changed in order to be well suited to Iranian hospital settings and accepted. Results of this process are summarized in Table 1.

Through the literature review, 129 indicators were found for risk management in four dimensions and clinical 232 indicators were found for effectiveness in 18 dimensions. While the highest number of indicators was related to patient safety, the lowest number of indicators was associated with pediatric and radiology. Internal medicine and hospital-wide indicators were merged into other indicators, anesthesia and endoscopy indicators were excluded due to the lack of sources and home care indicators were excluded because these services are not provided in Iranian hospitals. The members of the final expert panel decided to exclude indicators of emergency medicine because these indicators are currently collected by Iranian hospitals. The main reasons for excluding other indicators are as follows: The process of study is shown in Figure 1.

• the lack of data,

• weaknesses or lack of information systems such as HIS, management information system (MIS) and the lack of medical records,

• the lack of sources such as insufficient number of staff, the lack of medical equipment and the lack of time,

• not provided for intended services,

• the lack of cooperation from authorities and hospital personnel for data collection

• social and cultural issues

• the difference between Iranian hospitals administration and management systems and those in HICs and

• different pattern of diseases.

Discussion

In order to evaluate the performance of the clinical governance model in hospitals, the use of functional indicators to determine the rate of success and improvement is inevitable. In order to achieve greater transparency, better accountability and improved quality of services, functional indicators may play a key role. These indicators may be also useful to evaluate and measure the performance and the degree to which predetermined goals have been reached [28]. In this study, 113 indicators in domains of risk management and clinical effectiveness, based on the socio-economic situation and also hospital settings in Iran, were designed using different methods and stages (structured literature review, interviews, expert panels and Delphi technique).

In previous studies, researchers [29] extracted evaluation indicators of clinical governance in domains of risk management and clinical effectiveness using literature review and focus group discussion with staff working in all hospitals of East Azerbaijan province. This study tried to design indicators of clinical governance in these two domains, which finally led to the design of 204 indicators. Researchers in the mentioned study concluded that the weakness of the information system and the lack of preparation of hospitals have been the main reasons for excluding several indicators. Therefore, supporting the information systems of hospitals, the proper completion of medical records and an emphasis given by hospitals' top managers to collect and use indicators could be effective in using indicators in hospitals.

The regional office of the World Health Organization in Europe in a project titled ‘Performance assessment tool for quality improvement in hospitals (PATHS)’ and by using the expert panel and systematic review of literature and also with the cooperation with 20 European countries and piloted indicators in 8 countries developed 100 performance indicators in 6 dimensions including clinical effectiveness, safety, patient centeredness, staff orientation, efficiency of services and responsive governance [30]. The number of prepared indicators in this study is more than that of the study mentioned earlier, and the reason could be due to the accuracy and wide investigation of the studied domains in the latter study. While the latter study may be useful and effective, it can also be useless in situation where there is a lack of time. So, proper policy-making should be considered while using indicators of this study.

One of the main domains in the dimension of risk management is patient safety. Given the importance of this issue, a great attention has been paid to patient safety in many studies and the development of indicators of patient safety has been an ongoing issue. One of the most important studies is the study conducted by the Agency for Healthcare Research and Quality. This organization developed patient safety indicators in a study published in 2007 and introduced 20 indicators as the main and vital indicators for evaluating patient safety in healthcare centers [31]. Another important study in this field is the study conducted by the Organization for Economic Cooperation and Development in 2004 in which 21 indicators of patient safety in 5 major domains including nosocomial infections, surgical and post-surgical complications, sensitive and important events, obstetric indicators and other indicators of unwanted events related to patient safety were extracted and developed [32]. Conducting several studies in the domain of patient safety and great investments in conducting these studies indicate the importance of patient safety and the role of indicators in its improvement. Therefore, using local patient safety indicators developed in this study could be used for improving patient safety in Iranian hospitals and, with a slight modification, could be used in hospitals in countries similar to Iran.

In the dimension of clinical effectiveness, the infection control indicators have been considered as important indicators in this study. In the field of infection control, several studies have been conducted on infection control indicators [33–36]. In one of the most important studies, conducted by Blais et al. (2009) [37] in Canada, the authors introduced 97 key indicators in 22 domains of infection control using a literature review, expert panel, management consulting and data collection from 6 hospitals. The Australian Council on Healthcare Standards [38] and the Joint Commission on Accreditation of Healthcare Organizations [39] also designed and used indicators of infection control. Considering complications and unwanted consequences of hospital infections and the effect of using indicators on the control and decrease these consequences, using indicators identified in the current study and indicators available in references may be effective in this area.

A five-stage process including conducting a comprehensive literature review, expert panels, interviews, Delphi technique and a final expert panel was used in this study. By investigating those studies on developing indicators, it can be concluded that similar processes have been used to develop indicators [1, 11, 40]. In this study, after a literature review and the expert panels were conducted, key informants were interviewed. The indicators obtained from the literature review were modified and improved. So, there were few problems in the Delphi stage. As it has been mentioned in the results section of the Delphi phase, only 4 indicators out of the total of 168 indicators had low scores and were omitted from the study. Therefore, the process used in this study can be used as a pattern for developing indicators in further studies.

One of the weaknesses of the current study is the exclusion of some domains (such as home care and endoscopy). These domains were excluded from the study due to the lack of sufficient provision of these services in Iranian hospitals, the lack of adequate information and the lack of required equipment in this field. Furthermore, there was the risk of bias in this study, because members might select certain indicators for inclusion during the panel sessions. However, such bias would be minimized because of the multidisciplinary nature of the panel (i.e. the participation of a diverse range of participants including healthcare providers such as doctors and nurses, health policy makers and methodologists). During the panel sessions, the panel members were allowed to freely suggest, discuss and add further quality measures to the initial list.

Conclusion

Since clinical governance (7-column model) has been implemented for the first time in Iranian hospitals and also given that this model has been firstly designed in NHS of England (which is very different from Iranian healthcare system), reliable and valid local indicators are required in order to assess this model. In this study, using various processes and ideas of specialists and experts in different fields of medical sciences, local indicators for assessing clinical governance model in domains of risk management and clinical effectiveness were designed that can be used by policy makers, managers and other authorities for improving the quality of services and evaluating the performance of clinical governance model in Iranian hospitals. The results of this study can be used with slight modifications in other countries having healthcare systems similar to that of Iran.

References