Quality tools and resources to support organisational improvement integral to high-quality primary care: a systematic review of published and grey literature

There is growing awareness of the need to improve quality in health care, including in primary care.1-4 In Australia, this is witnessed by the National Primary Health Care Strategy, with its focus on the importance of quality as a foundation and driver of change.2 There is also an ongoing push for the development of new indicators for performance improvement, quality and safety benchmarking, and change management approaches and strategies for quality improvement (QI) in primary care.2

There are diverse terms and definitions used for QI,5 and varying QI strategies involving structured processes that include assessment, refinement, evaluation and adoption of processes used by individuals, teams, an organisation or a health system, with the aim to enhance some aspect of quality and achieve measurable improvements.6,7 These can include simple tools, such as flow charts and checklists; more complex multiple-method tools, such as re-engineering; and frameworks, such as the Plan, Do, Study, Act (PDSA) and audit cycles.8 These strategies have yielded modest change and are often not sustained over time.8,9 There is increasing evidence that QI initiatives that are locally owned and delivered, team-focused, formative and flexible and involve interorganisational collaboration and networking are more sustainable and yield better outcomes.10,11 The primary care practice team has a responsibility for QI as part of clinical and organisational governance, and team members are encouraged to collaboratively engage in QI activities in areas that will improve the safety or quality of patient health care.12-16 Primary care practices that embrace a QI culture and support QI initiatives are likely to have better health outcomes, better care delivery and better professional development.1,7,17,18

There is currently no single tool available to Australian general practices that combines traditional areas of clinical governance and less widely used aspects of organisational performance.19 In response, the Primary Care Practice Improvement Tool (PC-PIT)11 was co-created by a range of stakeholders using various engagement platforms, including ongoing cyclical feedback from partners and end users. The result is an organisational performance tool tailored to Australian primary care.11 The PC-PIT includes seven key elements integral to high-quality practice performance: patient-centred and community-focused care; leadership; governance; communication; change management; a culture of performance; and information and information technology. Results from the pilot study and the trial of the PC-PIT indicate that this tool offers an appropriate and acceptable approach to internal QI in general practice.11,20 The findings also showed that additional QI tools and resources are necessary to support the seven elements in the PC-PIT.20 Therefore, we aimed to undertake a systematic review of the international published and grey literature to identify existing primary care QI tools and resources that could support organisational improvement related to the seven elements in the PC-PIT. The identified tools and resources were then included in the next phase of study, which used a Delphi approach to assess the relevance and utility of these tools and resources for use in Australian primary care and to complement the PC-PIT.21

Methods

We undertook a systematic review of published and grey literature to identify existing online QI tools and resources to be included in a Delphi study assessing their relevance and utility in Australian general practice.21

Search strategy

In March 2014, we searched the electronic databases CINAHL, Embase and PubMed for articles published between January 2004 and December 2013, using the search strategy outlined in Table 1 of Appendix 1. We searched for articles where the search terms appeared in the title, abstract or subject headings, and limited results to those published in the English language. All searches were designed and conducted in collaboration with an experienced search librarian. We imposed no restrictions on the type or method of QI tool or resource and included any simple tools, multiple-method tools or frameworks that can be used by an individual in the practice, teams in the practice or the whole organisation to improve any aspect of organisational quality related to any of the seven elements in the PC-PIT.

In March–April 2014, we also conducted a comprehensive search of grey literature for documents dated between 1992 and 2012.22,23 This included an iterative manual search of the electronic database GreyNet International (http://www.greynet.org) and relevant government and non-government websites (Appendix 2). We consulted experts in primary care and QI to ensure key electronic databases, organisation websites and online repositories were included in the search. Searches were also conducted using Google Advanced Search (http://www.google.com/advanced_search) and repositories such as OpenGrey (http://www.opengrey.eu), WorldCat (http://www.worldcat.org) and OpenDOAR (http://www.opendoar.org).

For all relevant tools and resources identified through the grey literature search, we also searched in the research databases CINAHL, Embase and PubMed, as well as Google Scholar, for evidence of their use in practice. Search terms used in the grey literature search are shown in Table 2 of Appendix 1.

Finally, we reviewed the bibliographies of all identified relevant studies, reports, websites, databases, tools and resources to identify any additional QI tools and resources for inclusion. All additional tools and resources identified through this snowballing process underwent the screening and assessment process.

Selection of studies, tools and resources

All citations were imported into a bibliographic database (EndNote, version X7). To be included in the review, identified citations, tools and resources had to meet the following eligibility criteria: (1) purpose of the tool or resource is QI; (2) tool or resource is used in the primary care setting or has potential for use in primary care; (3) tool or resource addresses at least one of the seven elements integral to high-quality primary care practice; and (4) tool or resource is available and in the English language.

The initial screening process involved two reviewers (S U and T J) screening the titles and abstracts of published citations and any articles, reports, tools or resources identified through the grey literature, and categorising them as “relevant” or “not relevant” according to the review objective. The full texts of all tools and resources deemed relevant were sought and reviewed by two independent reviewers with expertise in primary care QI (S U and T J) to further assess their relevance according to the eligibility criteria.

There is no single well established assessment or scoring instrument suited for QI tools and resources that covers the broad range of tools and resources included in this review. Therefore, we developed a four-criteria appraisal framework from common sets of criteria proposed for assessing a range of QI tools, resources and initiatives, such as guidelines, instruments, programs and web-based resources (Box 1).24-30 All identified tools and resources that met the eligibility criteria were evaluated for their accessibility (ie, able to be accessed online and at no cost), relevance, utility and comprehensiveness using this four-criteria appraisal framework. Two reviewers (S U and L C) independently gave each tool or resource a score out of 8 using the criteria. Tools or resources with a score of 7–8 were rated as the “best” and passed on to the Delphi study21 for further assessment. Tools and resources rated less than 7 were rejected and not included in further assessment. The reviewers compared their ratings, and any discrepancies were resolved through discussion.

Data extraction and synthesis

We created a data extraction template using Microsoft Excel to assist in systematically extracting information about the tools and resources that met the eligibility criteria. A content analysis approach was used to explore each tool or resource to collate the following information: name of the tool or resource, year and country of development, author, name of the organisation that provided access to the tool or resource and its URL, accessibility information or problems, a brief overview of each tool or resource, the QI element(s) it addresses and any supporting evidence (published or unpublished data). If accessible, a copy of the tool or resource was downloaded into the bibliographic database. Any supporting evidence (studies, reports and any other data) on the use of the tool or resource in primary care was also added to the bibliographic database.

Results

The database search yielded 1900 citations after duplicate records were removed (Box 2). After reviewing the titles and abstracts for relevance to the review objective, the total was reduced to 249 articles. Of these, 140 did not meet eligibility criteria and were excluded, leaving 109 articles. Most excluded citations did not meet the eligibility criteria because the tools or resources were not used in primary care settings. From the 109 citations, 76 QI tools or resources were identified (Appendix 3).

The level of empirical evidence for each tool or resource varied substantially — some, such as the PDSA, had numerous studies supporting their use in primary care,31-35 whereas others, such as the Organisational Capability Questionnaire,36 had only been taken to pilot stage. Of the 76 tools and resources identified in the published literature, 37 were rejected because of accessibility problems. Of the remaining 39 tools and resources, 19 scored less than 7 on the four-criteria appraisal and were rejected due to problems related to utility (n = 10), relevance (n = 3) and comprehensiveness (n = 6). This left 20 that were classified as the best tools and resources (Appendix 4).

Through the grey literature search, we identified 186 tools or resources that met the eligibility criteria (Appendix 5). Of these, 12 were rejected because of accessibility problems. A further ten tools or resources were duplicates and also excluded. Of the remaining 164 tools and resources, 131 scored less than 7 on the four-criteria appraisal and were rejected due to problems related to comprehensiveness (n = 99), utility (n = 16) and relevance (n = 16). This left 33 tools or resources identified as the best from the grey literature (Appendix 4).

Of the total 53 best tools and resources identified through published and grey literature, 13 were from Australia and the remainder were from the United Kingdom (n = 14), United States (n = 14), Canada (n = 4), New Zealand (n = 4) and Europe (n = 4). There was significant overlap of the PC-PIT elements covered by the best tools and resources, with most tools relevant to two or more elements integral to high-performing practices. Of the 53 identified tools and resources, 34 predominantly addressed performance, 20 governance, 19 patient-centred care, 15 change management, nine leadership, nine communication, and six information and information technology (Appendix 4).

Discussion

In an effort to strengthen primary care practices, and thereby strengthen the broader health care system, many providers, delivery systems and other organisations are supporting the use of QI initiatives to improve the performance of practices.37 There are currently no published data regarding the available QI tools and resources for Australian primary care. In this review, we identified and synthesised existing primary care QI tools and resources from the international published and grey literature that are relevant to the seven elements integral to high-quality primary care practice,19 which are specifically covered by the PC-PIT.11 Our findings provide data on QI tools and resources that can be used to support QI initiatives in primary care, including complementing and optimising the value of the PC-PIT.

Given the complexity of health care, developing, implementing and assessing QI initiatives is a dynamic, evolving and challenging area.38 This review illustrates the wide range of primary care QI tools and resources that are available. There is substantial variability in the accessibility, comprehensiveness and utility of tools and resources for primary care, as well as the evidence for their use. Many tools and resources require extensive (and often costly) external facilitation, which adds further complexity and limitations to their application in general practice settings.

Variability in evidence

There is a gap in the published literature on QI tools and resources in primary care settings, and the available literature is of varying quality.39,40 This is partly due to the complexities involved in reviewing a heterogeneous set of interventions that are applied in a varying set of contexts.41 This lack of scientific literature has somewhat inhibited the acceptance of QI methods in health care.38 With new approaches, tools and resources being introduced at a rapid pace and disseminated through the World Wide Web, there is some debate about the most effective QI tools and resources for use in the health care setting.7 Although new studies are emerging,38 there is a need for more rigorous evaluations of different QI tools and resources in primary care settings.39,42

Comprehensiveness of tools and resources

There are many approaches and strategies that can be used to improve the quality of primary care practices. These improvement strategies are generally divided into two types: improvement focusing on clinical areas and improvement focusing on quality from a management perspective.6 Although the two may share common themes, they are often seen as discrete parallel activities. For example, the NPS MedicineWise Clinical e-Audits are used to facilitate clinical QI by assisting GPs to review their prescribing practices,43 while the Advanced Access and Efficiency Workbook for Primary Care focuses on improving the organisational quality of the practice to enable patients to see their doctor when they need to.44 Some tools and resources, such as Lean,45 Six Sigma46 and the Manchester Patient Safety Framework,47 are based on theoretical frameworks, whereas others, such as the Canning Data Extraction Tools48 or the eCHAT (electronic case-finding and help assessment tool), are more pragmatic.49 Some tools and resources, such as the PDSA, Six Sigma and Significant Event Analysis, are well known.6,50 Other less well recognised tools and resources range from the simple, such as the Organisational Capability Questionnaire,36 to the more comprehensive, incorporating a range of other supporting tools, such as the UK’s National Health Service (NHS) clinical engagement resources51 and the NHS Scotland Quality Improvement Hub.52

Due to the complexity of primary care practice and the dynamic process of QI, several QI tools and resources could be used in conjunction with each other, or one after another, to yield successful outcomes; for instance, beginning with root-cause analysis, then using either Six Sigma or PDSA to implement a change in processes.38 Another example is the use of tools and resources for improving chronic illness care, such as using the Primary Care Resources and Supports for Chronic Disease Self Management53 in conjunction with the Assessment of Chronic Illness Care,54 with the former focused on self-management support and the latter on improved patient and staff competency in self-management processes.

Accessibility and utility of tools and resources

It can be challenging to engage practices in QI initiatives because primary care clinicians and staff often feel intense time pressures; have competing priorities; lack a culture and leadership that support change; lack resources, capability and capacity; and may fear the perceived costs of undertaking QI.7,17,18,55 Therefore, ease of access and utility are important factors in optimising the acceptance and adoption of QI initiatives in primary care practices.7,18 In line with the literature, the main reasons tools and resources were rejected in this review were that they rated poorly with regard to their comprehensiveness (42%), accessibility (19%), utility (10%) (ie, too complicated, contained difficult language, too time-consuming or required extensive facilitation) and relevance to primary care (8%).

QI efforts need to be substantially more efficient and easy to access and must reduce the burden on practices to maximise their adoption in primary care settings.17 Recognising this need, some health care organisations provide comprehensive online libraries of quality and service improvement tools and resources that are readily accessible and free of charge.56-58 Nonetheless, it is often difficult for busy practitioners to navigate through multiple websites to obtain the right tools or resources for QI. Therefore, a better option would be a suite of QI tools and resources that is embedded into existing quality frameworks.

Support and incentives for quality improvement

Practices need to be supported and incentivised to adopt a QI culture and engage in continuous QI initiatives.7,18 Even the most determined practices are likely to require help in developing their QI capacity, such as skills to identify areas for improvement, knowledge and understanding of QI approaches, how to use data for QI, planning and making changes, and tracking performance over time.37 This demands the commitment of practice leadership and staff to dedicate time and resources to QI activities.37,38 Practices will also require external support, such as technical assistance, learning activities and tools and resources provided by organisations to assist practices undertaking QI initiatives.37

Public and private health care sectors around the world are now linking service quality with provider payment. Both the UK and the US provide financial incentives to some health care providers for adopting improved quality practices. Using a “pay for performance” system can drive and support practices to adopt QI initiatives to improve the quality of their practice and patient outcomes.59 In Australia, the Primary Health Care Advisory Group recently considered new payment mechanisms to better support the primary care system to drive safe and high-quality care.60

Limitations

Our review has several limitations. First, the exclusion of non-English-language literature may have omitted some relevant tools and resources. However, non-English tools and resources could not have been used in Australian primary care without being translated, which was not feasible within the scope of the study. Second, QI initiatives (including tools and resources) are poorly indexed in bibliographic databases.39 As such, we employed broad search strategies that used free text and Medical Subject Headings (MeSH) to optimise our search strategy. While we also included grey literature to capture tools and resources, an exhaustive search was not undertaken due to time constraints. Other studies have reported similar challenges.61,62 In response to this, we consulted with experts in the area to ensure that the key relevant electronic databases, organisation websites and online repositories were not missed in the search. Finally, the four-criteria appraisal framework and the method of rating the tools and resources was subjective and potentially biased, and we did not perform a sensitivity analysis against the robustness of the assumptions. Hence, caution is required when interpreting the classification and rating of each tool or resource. To address these limitations and increase reliability, the two reviewers who assigned the ratings discussed, checked and agreed on the scoring.

Conclusions

The necessity for QI initiatives permeates health care37,59,63 and presents opportunities to fundamentally improve health in Australia. Engaging primary care practices in QI and practice redesign activities allows them to work toward achieving improved quality, better health and improved patient and provider experiences, as well as reducing the ongoing costs of care.37,64 To ensure these efforts have a positive impact, there is a need to build and sustain the ability of primary care practices to engage in QI initiatives in a continuous and effective way. To foster QI capacity in Australian health care, we have identified tools and resources that can potentially be provided as part of a suite of tools and resources to support primary care practices in improving the quality of their practice, to achieve improved health outcomes. Following this review, a Delphi study was undertaken to evaluate the 53 best tools and resources to assess their relevance and utility in Australian general practice; the results are published elsewhere in this supplement.21

Box 2 – Flow diagram outlining selection process for tools and resources