Trial of the Primary Care Practice Improvement Tool: building organisational performance in Australian general practice and primary health care

As the cornerstone of health care, primary care should be effective, efficient and responsive to the needs of patients, families and communities.1 The Australian primary care system currently has significant opportunities to co-create approaches to quality improvement (QI) and practice redesign in ways that could fundamentally improve health care in Australia. To ensure these efforts are successful, there is a need to build and sustain the ability of primary care practices to engage in QI activities in a systematic, continuous and effective way.2 Primary Health Networks (PHNs) are a central component of the government’s primary health care reforms and have a number of roles in improving the quality of primary care. They will work closely with general practices and other primary care services in planning and supporting primary care teams to adopt QI initiatives, with the overall aim of improving care delivery and health outcomes.3,4

The Australian Commission on Safety and Quality in Health Care has recently developed a national set of practice-level indicators of safety and quality for primary health care. These indicators are designed for voluntary inclusion in QI strategies at the local practice or service level and are intended for local use by organisations and individuals providing primary health care services.5 The Australian Safety and Quality Framework for Health Care was also developed, which sets out the actions needed to achieve safe and high-quality care for all Australians.6 To respond effectively to these, primary care practices will need to be equipped with both organisational development and change management approaches. In response, the Primary Care Practice Improvement Tool (PC-PIT), an organisational performance improvement tool, was co-created for Australian primary care.7

The co-creation methods involved combining results from a systematic literature review8 and pilot study9 with cyclical feedback from partners and end users (general practices), using a variety of engagement platforms (interviews, face-to-face meetings, formal presentations, workshops and webinar discussions).10 The systematic literature review identified 13 elements integral to high-quality practice performance, which was defined as “systems, structures and processes aimed to enable the delivery of good quality patient care” but which do not necessarily include clinical processes.11 During discussion sessions, stakeholders and end users identified a list of desired attributes for a performance improvement process; namely, that it should be simple, accessible online, enable a “whole-of-practice” approach, be an internal process facilitated by practice managers or nurses without the need for extensive external facilitation, have additional support resources, be no or low cost and fit with existing QI or practice support programs.

The tool resulting from this combination of processes, the PC-PIT, was initially piloted with six high-functioning practices.9 In this study, we trialled the refined PC-PIT approach nationwide with three objectives: (i) to document and describe the use of the online PC-PIT in practice; (ii) to validate the PC-PIT independent practice visit objective indicators; and (iii) to identify the perceived needs (eg, resources, professional development) to support practice managers as leaders of organisational improvement in general practice.

Methods

We conducted the national PC-PIT trial from March to December 2015, with volunteer general practices from a range of Australian primary care settings. Practices were invited to participate using newsletter information sheets distributed by partner and stakeholder organisations and through national webinars and conference workshops. We used a mixed-methods approach, collecting both quantitative and qualitative data. A full description of the trial protocol is available in Appendix 1. Ethics approval for the study was granted by the University of Queensland Behavioural and Social Sciences Ethical Review Committee (approval number 201000924).

The PC-PIT trial consisted of two parts. In Part 1, practice staff at all participating practices completed the online PC-PIT, with each practice staff member giving a subjective assessment of how they perceived their practice met (or did not meet) the best practice definition of each of the 13 PC-PIT elements, using a 1–5 Likert rating scale.

For Part 2, we selected a purposeful sample of the primary care services to represent a range of practice sizes, business models and geographic locations. Two external raters conducted an independent practice visit to each selected practice, during which they assessed the subjective practice assessment from Part 1 against objective indicators of the same 13 PC-PIT elements, as supported by documented practice evidence. Each rater completed a separate evidence assessment form and ranked a set of objective indicators for each PC-PIT element (using a 1–5 Likert scale) by reviewing the documented practice evidence; working with both practice managers and practice nurses to identify and cite the evidence. Box 1 provides a summary of the Likert ratings and how they are interpreted in the context of the PC-PIT elements.

Data were gathered from a variety of sources during the independent practice visits, including interviews with practice managers and practice nurses, background materials and documented evidence such as meeting minutes, policy and procedure manuals, and communications books. Well documented strategies to enhance trustworthiness and rigour were incorporated in the qualitative phase of the study design; namely, that the two raters cited the sources, which allowed the triangulation of information and the review and confirmation of findings.12,13 The in-depth, semi-structured interviews with practice managers and practice nurses explored their involvement in QI and also their perceptions of resources and support needed to facilitate their role in performance improvement. A proforma was used to guide the interview discussions and all interviews were recorded, either manually or using a digital recorder. Interviews were then transcribed, and participants were provided with the opportunity to review and edit their responses.

The staff ratings for the PC-PIT elements from Part 1 were aggregated to a median practice score for each element and then compared with the ratings from the objective indicators in Part 2 and displayed in two side-by-side spider diagrams. These data were presented to each practice in individual PC-PIT reports, thus presenting a profile of practice performance against each of the 13 elements, using staff perceptions compared with documented practice evidence. Appendix 2 provides two de-identified examples of completed PC-PIT reports, including spider diagrams, as provided to a higher- and a lower-scoring practice. Practice managers used their PC-PIT reports to lead staff discussions about the identification of specific improvements to be made, strategies to achieve them, a time frame, measures of success and the staff member(s) responsible. This was then formalised using the Plan, Do, Study, Act (PDSA) approach.

Data analysis

Quantitative data were entered into a Microsoft Excel 2013 spreadsheet, then imported into SPSS version 21.0 (SPSS); data were analysed using SPSS and Excel. A key outcome measure was the degree of concordance between Rater 1 and Rater 2 during the independent practice visits, measured using concordance and κ statistics. Standard integer weights were used, as described by Fleiss.14

A total of 32 in-depth, semi-structured interviews were held (with 19 practice managers and 13 practice nurses). Transcribed interviews were coded by one member of the research team (L C) using NVivo 10 (QSR International). Codes were reviewed for duplication and clarity. We used thematic analysis to identify and classify recurrent patterns and themes. Interviews focused on aspects such as the background training and current role of the practice manager and practice nurse, QI experience of the individual interviewees and the most recent QI undertaken in the practice.

Results

A total of 45 general practices participated in Part 1 of the PC-PIT trial. At the time of writing, complete datasets were available for 34 of the 45 practices. These represented a range of geographic locations (urban, regional and rural areas), although most were urban and regional practices. It also included a range of practice sizes (< 2, 2–< 5, 5–< 10, 10+ full-time-equivalent general practitioners) and represented a range of business models (privately owned, partnerships and corporate business models). Ten of the 34 practices described both undertaking internal QI activities, such as PDSA cycles, and involvement in externally run QI activities, such as Medicare Local programs, National Prescribing Service activities and the Australian Primary Care Collaboratives. The remaining practices described either internal QI activities or involvement in external improvement programs. One practice was newly established and had not undertaken any improvement activities within the past 12 months. The practice managers came from a variety of backgrounds, including business management, nursing and allied health. Appendix 3 details the characteristics of the 34 participating practices.

Of these trial practices, 20 were selected for the independent practice visits and qualitative interviews in Part 2, and complete datasets were available for 19 practices. One practice was excluded due to competing commitments of the practice manager, which made it impossible for the independent practice visit to be conducted by the raters during the study timeframe.

Assessment of practice performance against the PC-PIT

A total of 310 online PC-PIT forms were completed in Part 1 by practice staff, comprising 19 practice managers, 95 GPs, 56 practice nurses, 109 administration and reception staff, 25 allied health (including pharmacy) staff and six “other” staff (primarily in business development, finance or information management roles). Using the combined online PC-PIT element ratings, the independent practice visit ratings and interviews with practice managers and practice nurses, three specific practice types were identified among the 19 practices from Part 2, each with a distinct way of using the PC-PIT. Rather than being discrete, these three types represented key points along a continuum of organisational performance, from lower-scoring to higher-scoring practices.

First practice type

The three lowest-scoring practices appeared to have separate and uncoordinated clinical and practice management processes. This was evidenced by uncoordinated clinical governance and organisational management activities and the incomplete translation of clinical and management processes into formalised policies and protocols that were clearly known and understood by all staff (both clinical and administrative).

Second practice type

A further three practices had a primary focus on clinical governance, with organisational management as a supporting basis. In this model, practice managers had limited or no autonomy in relation to organisational changes within the practice. This was illustrated by a lack of cited documented evidence (and therefore lower scores) on the PC-PIT element of organisational management, including key indicators such as evidence of staff role descriptions, performance appraisals, internal QI activities and the use of information such as data reports, formal meetings and discussion to improve the internal function of the practice. The practice manager generally worked in a supporting role to the GP(s), but there was limited evidence of communication and coordination between clinical and organisational management.

Third practice type

The five highest-scoring practices recognised the equal importance of organisational and clinical management in supporting the ongoing operation of the practice as a whole, demonstrated by high ratings in both the independent practice visit and staff online PC-PIT forms. Documented evidence of meeting minutes and previous PDSA processes and outcomes showed that management processes were constantly reviewed in a combined approach by clinical and administrative–management staff and readjusted to facilitate patient care. These practices demonstrated close communication and shared decision making in relation to continuous QI, championed by an autonomous practice manager who worked closely with a defined clinical leader. They were also more likely to have a history of involvement in a range of external continuous QI programs.

The remaining eight practices fell along the continuum, with most toward the lower-scoring end. These practices were generally characterised by positive staff perceptions of the 13 PC-PIT elements but a lack of documented supporting evidence, particularly on the use of practice data in making ongoing improvements to their organisational processes and in reviewing and using performance results. Box 2 provides examples of the three practice types, the median PC-PIT element scores given by the staff and the raters, illustrative interview quotes and the evidence cited during the independent evidence assessments.

Agreement between raters: independent practice visits

Overall, there was complete agreement between the two raters in 11 of the 19 general practices. Rater 1 scored higher for 11 PC-PIT elements and lower for one. The mean difference was 0.10. Box 3 presents the agreement between raters and the κ statistic for each element. The element with the lowest κ (0.43) was team-based care. For this element, the two raters agreed in 11 of the 19 practices; Rater 1 scored higher than Rater 2 in seven practices and lower in one. If we excluded this element from the overall κ coefficient, the χ² test for homogeneity was 14.66 (P = 0.20).

To identify reasons for key discrepancies by practice and by element, the raters reviewed their evidence-based assessment forms and discussed possible reasons for the discrepancies. The discrepancy in Practice 10 was due to circumstances that required the raters to interview different informants and cite separate documentation in relation to the PC-PIT elements. Poor concordance between the ratings for the element of team-based care reflected a lack of formally documented policies and procedures available to practice managers, while additional undocumented information could be provided by practice nurses. Rater 1 scored this verbal (but undocumented) information, while Rater 2 did not (Box 3).

In terms of practices, Rater 1 scored more elements higher than Rater 2 in six practices, especially for Practice 10, where there was agreement on only one element (Appendix 4).

Resource and support needs of practice managers

During the in-depth interviews, four key themes were identified in relation to practice managers’ perceived resource and support needs (Appendix 5). Most practice managers were not familiar with internal organisational development tools other than those previously developed by the former Divisions of General Practice. Most of these tools were neither trialled nor validated in general practice settings. Only one practice manager described the use of a formalised approach to organisational development (Six Sigma), which was recently adapted for use in general practice but required extensive external facilitation to complete. Practice managers perceived the benefits of having additional supporting tools relating to elements in the PC-PIT, and also identified strategies such as online forums or email updates, based on the PC-PIT elements of high-performing practices, which might focus on sharing key problems and solutions for organisational performance improvement.

Discussion

This national trial of the PC-PIT determined that it can be a useful organisational performance tool in various general practice settings.

As health care delivery becomes more complex and technology-driven, the organisational context in which QI initiatives take place becomes increasingly recognised as a crucial determinant of their effectiveness.16,17 Contextual elements have been described as the “adaptive reserves” of a practice; that is, those features that represent a practice’s internal capability.18 They include features such as culture, leadership, collaboration and teamwork, data and information tools, improvement skills, incentives and time allocation, which general practices should address to support a context of continuing QI.19

The establishment of the PHNs and the release of the consultation paper for the review of the Performance and Accountability Framework indicators20 illustrate the integration of aspects of QI across the health reform strategy. Although as yet incomplete, the national Performance and Accountability Framework objectives relating to quality focus on outputs related to safety, responsiveness (based on measures of patient experience), capability and capacity.20,21 Following this, the proposed national PHN evaluation framework lists continuous QI activities, outputs and outcomes related to provision of practice support and the identification of high-priority practices (those practices requiring targeted support to build their capacity to engage in QI), such as accreditation and the use of data for practice improvement.

The need to develop and strengthen managers’ skills also involves the development of management processes for motivation, supervision control and action, and support at an organisational level.22 Practice managers may be responsible for large and fluctuating numbers of staff, high yearly financial turnovers and the ongoing facilitation and management of change; many do so with limited timely access to appropriate ongoing training and validated support resources. While there is an undeniable need to focus on the role of GPs in QI, it is also worth noting that the elements relating to organisational improvement are also the domain of practice of operational managers.

Although the independent practice visit was conducted by two external raters in this trial, we anticipate that this aspect will become part of the PC-PIT as a wholly internal assessment process. However, future testing of the PC-PIT will seek to further address the discrepancies relating to the involvement of different practice staff members (ie, practice managers v practice nurses) in the use of the evidence-based assessment forms. The objective indicators for the element of team-based care have also been refined and clarified to include additional meeting minutes and documentation accessible to either practice managers or practice nurses, to ensure that all available evidence is taken into account during the assessment.

Two of our partner organisations, the Royal Australian College of General Practitioners and the PHNs, identified a key benefit of the PC-PIT as the ability to identify the lower-scoring practices and more effectively engage them in organisational improvement activities, allowing for more targeted interventions that are relevant to the capacities of the individual practices. Thus, there are two aspects to the future sustainability of the PC-PIT: (i) embedding the PC-PIT approach into existing QI frameworks, and (ii) further research into the role of the PC-PIT in supporting performance improvement in primary health care. The PC-PIT approach will be further developed to include a suite of high-quality, validated and free-to-access resources that complement the use of the tool.23,24

Limitations

Although this trial was conducted with volunteer practices, every effort was made to ensure a range of geographic locations and practice sizes were incorporated in the inter-rater comparison. However, there was an over-representation of urban and regional practices. Many rural practices were unable to commit due to perceptions of the time required to complete improvement activities. However, this is an area that may be of interest to the newly established PHNs, given their formal role as facilitators and supporters of practice engagement in QI. We also acknowledge the lack of consumer involvement during the trial phase of the PC-PIT. Further work to refine and embed the PC-PIT in existing QI programs will seek to involve the Consumer Health Forum of Australia as a key partner in the process, with emphasis on the role of consumer feedback as an embedded feature of external validation.

In relation to the inter-rater comparison, the calculation of the aggregate value of κ over the 13 PC-PIT elements assumes that the κ values are independent, which is unlikely. The lack of independence, however, is unlikely to affect the aggregate value but might increase the standard error to a small degree.

Conclusion

With the refocus on the importance of organisational aspects of practice in relation to quality care delivery, the time is now right to focus on a standardised, internally led approach to improving practice performance, designed for the dynamic context of primary care. Work with our key partners and end users is ongoing, with the aim of further trialling and embedding the PC-PIT within existing QI initiatives.