Development and pilot study of the Primary Care Practice Improvement Tool (PC-PIT): an innovative approach

There has been increased focus on the importance of primary health care across Australia, the United States and the United Kingdom.1-4 Australia's first National Primary Health Care Strategy was developed in 2010, with a refocus on the importance of quality primary health care via a range of approaches, including regional integration and a focus on information and technology including e-health; ensuring a skilled workforce; improving infrastructure; and a focus on financing and system performance.4 It also heralded the development of new indicators, including performance improvement; teamwork; patient-centric approaches to care; an emphasis on quality and safety benchmarking; a combined approach to organisational and clinical governance and strategies for change management.4

While much has been written about the development and implementation of approaches to quality improvement in tertiary care or approaches adapted to primary health care, there is a paucity of research on approaches designed specifically for primary health care settings.

There is currently no single tool available to general practices combining the traditional areas of practice organisation (eg, clinical governance and the use of information technology) with more contemporary and, as yet, less widely used elements (eg, change management and leadership) in an internally facilitated approach. Consequently, there is a need for quality improvement tools and approaches “bespoke” to primary health care; sensitive to the wide variations in primary health care contexts; which take a whole-of-practice approach. These can be based on organisational assessment approaches that are sensitive to, and inclusive of, the concerns of clinical management.5-7

In a systematic review published in this supplement, we identified 13 key elements integral to high-performing practices.8 Building on this research, we developed the Primary Care Practice Improvement Tool (PC-PIT).

The PC-PIT seeks to link tangible aspects of quality improvement (eg, the presence of defined processes of care, formalised meetings, data collection and review) and less tangible domains (eg, communication, change management and the creation of a culture of performance). Although it was not designed to replace existing accreditation processes, it can highlight specific areas across clinical and organisational cultural aspects in which a practice may wish to focus its improvement efforts.

The PC-PIT aims to (1) be adaptable to variable and dynamic individual service settings; (2) address clinical governance and the impact of organisational management as part of an ongoing quality improvement cycle; (3) be led by practice managers as an internal quality improvement process based on a whole-of-practice approach; and (4) be delivered online and at low cost.

In this article, we describe the development of the PC-PIT and subsequent validation in a pilot study. The pilot study had three objectives: to determine the readability of the PC-PIT; to establish its content validity; and to explore staff perceptions of its use in general practice.

Methods

Development of the PC-PIT

Based on the findings of our systematic review (Box 1),8 the PC-PIT was co-created18 using cyclical feedback from a range of key national stakeholders, including the Australian Association of Practice Managers (AAPM), the Royal Australian College of General Practitioners (RACGP), the Improvement Foundation (Australia), the Australian Primary Health Care Nurses Association and the Australian Commission on Safety and Quality in Health Care. This process provided ongoing input into the elements used, and the structure and mode of delivery of the tool.

The PC-PIT was designed as an online instrument that could be completed confidentially by practice staff, facilitated by the practice manager. Staff receive a personal link to the online tool, provide basic demographic information, and rate their perception of each of the 13 elements on a five-point Likert scale. To maintain the privacy and confidentiality of the respondents, a report is generated from the completed tools. The report provides average scores for each of the 13 elements for the entire practice. Results are presented to the practice manager in a spider diagram, separating the lower scoring elements (those that score an average of 1–3) from those that scored higher (an average of 4–5). Lower scoring elements indicate where the practice may wish to make some improvement. If the practice has more than 15 staff members, practice managers can request aggregated scores for each employee group (administration; management; clinical; allied health) and differentiate scores provided by permanent versus contract staff.

After reviewing their practice report and discussing the scores with practice staff, practice managers then undertake the Plan–Do–Study–Act cycle. This approach guides practice managers through a process to identify area(s) for improvement, plan and monitor strategies to achieve the improvement, identify staff to undertake the improvement, and identify clear indicators to measure when the improvement has been achieved.

Pilot study

The pilot study was based on a formative assessment framework and mixed-methods research design. Assessment of the PC-PIT was conducted with a purposive sample of six general practices in Brisbane, Australia. Critical case sampling19 was used to select the practices from which the most detailed, and information-rich data could be obtained on this topic because all practices had experience in the use of quality improvement processes and their integration into the general practice setting.

Each practice provided detailed quantitative and qualitative assessment. In addition, two practice managers were experienced practice accreditation assessors.

All staff across the six practices were provided with a personal link to access the PC-PIT and asked to complete it within a week. They were also provided with a hard copy questionnaire, which elicited quantitative and qualitative data.

The study was conducted between 1 February and 30 July 2013. Ethics approval was granted by the University of Queensland Ethics Committee.

Quantitative data collection

Readability was assessed using the Flesch–Kincaid Readability Formula and Gunning–Fog Index in a combined online test.20 The Flesch–Kincaid grade level indicates a reading age based on the US education reading assessment system. The Gunning–Fog Index score is based on the number of words, and additional complex words (words containing three or more syllables) in the selected text. A limitation of the Index is that not all complex words are difficult to understand.

Participants completed a series of Likert scales that specifically asked for ratings of the following content:

• Readability. How easy was it to understand the PC-PIT, were there any words or phrases you were unfamiliar with; were there any words or phrases you were unsure of?
• Content validity. Relevance to general practice; relevance to the role and position of practice staff. Wording and understanding: where did you get stuck; why did you get stuck (layout versus content); what does this element mean to you/how would you describe this element?
• Process validity. Usability of the tool — ease of use online; layout of questions; problems or issues completing the online PC-PIT; suggested changes to layout and process of completion.

Qualitative data collection

Participants were asked to respond to a series of open-ended questions about their experiences of completing the PC-PIT and their perceptions of the relevance and usefulness of the tool to general practice. Semi-structured interviews were conducted with practice managers to gain feedback on perceptions of the content of the PC-PIT, its usefulness as a primary health care improvement tool and the process of using the PC-PIT.

Likert scale data for each practice were analysed using Microsoft Excel 2010 (Microsoft Corporation) to enable basic descriptive statistics (frequencies). Semi-structured interviews and open-ended qualitative data were transcribed and imported into NVivo 10 (QSR International). An inductive thematic analysis was undertaken to identify common themes.21

Results

Six practices were enrolled in the pilot. Four practices completed the pilot and provided complete datasets. Two practices did not complete all data collection due to staff absences or building renovations during the study period. These practices were not included in the final data analysis. In total, 28 staff completed the pilot — 10 GPs, six practice or community nurses and 12 administrators (including four practice managers; one business manager and eight reception or general administrative staff).

Readability of the PC-PIT

The PC-PIT required a high reading age of greater than 20 years. The definitions of organisational and clinical governance, along with those related to information and information technology, were scored as highly complex text. These scores were consistent with the qualitative feedback from several administrative practice staff members, who assessed these element descriptions as difficult to understand. Appendix 1 provides a comparison of the readability scores for each of the PC-PIT elements and their corresponding definitions.

Many administrative and reception staff (with the exception of practice managers) found elements of the PC-PIT difficult to understand. Two GPs also provided low ratings (1–3) for the element relating to education and training. This was due to confusion about how the element of education and training related to requirements for continuing professional development available to GPs in practice. Details are shown in Appendix 2.

Two key difficulties were identified by staff in the qualitative feedback: complicated wording and difficulties understanding the terms governance and performance (Box 2). A range of staff (nurses, allied health professionals and administrative staff) felt that the wording of the PC-PIT definitions were long and complicated. Nursing staff also made suggestions to change the term governance to management, to clarify the meaning for all staff.

Acceptability and relevance of the PC-PIT to general practice

Nineteen of the 28 participants rated the PC-PIT as a useful tool for assessing key elements of practice organisation and function. Participants emphasised the relevance of the PC-PIT to everyday practice work and planning, and the role of the PC-PIT in allowing all staff to be involved in the identification of areas for improvement (Box 3).The PC-PIT was seen as highly usable, with 21 participants rating it as easy and preferable to complete online than a paper-based form.

However, eight participants did not think it would be usable as a future assessment tool based on its current format. This group included administrative staff, who generally found the PC-PIT elements difficult understand, and two GPs, who perceived that the PC-PIT covered areas that were predominantly outside clinical management processes. Four participants did not respond to the question.

Discussion

Overall, participants assessed the PC-PIT as an acceptable and easy-to-complete quality improvement tool that offers a new approach to improving practice performance in areas that are not routinely addressed. Although inclusion of all staff was noted as particularly useful, some sectors of senior administration felt that not all staff needed to be aware of areas of practice function, such as performance measures and aspects of organisational and clinical governance. Such attitudes may compromise the future effective use of the PC-PIT, particularly as many staff roles and responsibilities are highly interdependent, and improved teamwork is based on an understanding and appreciation of the complementary aspects of these roles.

Based on findings from the pilot, we have reworded and simplified PC-PIT element definitions. Based on the attitudes expressed by business managers, an online prerecorded presentation provides an introduction to the purpose of the PC-PIT and the whole-of-practice approach. This approach aims to clarify the purpose of the PC-PIT and ensure all staff have a basic understanding of how the elements relate to practice performance and their own roles and responsibilities.

A national trial of the PC-PIT is now underway. The trial will assess process and construct validity with a range of general practices and primary health care business models in different settings, including small private practices and large corporate service models across Australia. It seeks further and more in-depth information about the means by which individual staff members perceive and score the 13 elements of the PC-PIT, the utility and effectiveness of the tool across the range of practice models and the role of context (eg, inner urban, regional and rural settings) in the adaption and use of the tool. Participation in the trial allows staff to gain quality improvement and continuing professional development points under the RACGP 2014–2016 triennium and points under the AAPM professional development program.

The PC-PIT provides an innovative approach to address the complexity of organisational improvement in general practice and primary health care, matched to key elements identified in international evidence as being integral to high-functioning practices. Our study indicates that the PC-PIT has content and process validity offering a comprehensive, practice-led approach to organisational aspects of continuous quality improvement. Further research to refine the tool has now commenced with primary health care services across Australia.