Australian clinical practice guidelines — a national study

Evidence-based clinical practice guidelines translate findings from health research into recommendations for clinical practice1 and, when implemented, can improve health outcomes.2-4 Currently, Australian clinical practice guidelines are produced by disparate groups, including government agencies and professional societies. Without information on the number and topics of current Australian guidelines, and documentation of evidence search and appraisal processes on which they are based, it is not possible to assess the degree to which clinical practice in Australia matches best practice recommendations in specific areas, or to identify priorities for guideline implementation.

A 1993 study of national and state clinical practice guidelines identified 34 clinical practice guidelines produced by 32 organisations.5 None of these guidelines fully described their methods for finding and synthesising evidence, and 29 of the 34 guidelines gave no description of the methods used to identify evidence.

In 2006, the National Institute of Clinical Studies (NICS) initiated a study of national and state Australian clinical practice guidelines to identify priorities for future implementation programs. The study continued after NICS became an institute of the National Health and Medical Research Council (NHMRC) in 2007. Here, we report the study findings, including the extent to which clinical practice guidelines produced between 2003 and 2007 documented their evidence search and appraisal processes.

Of the 1014 documents received from national and state health-related organisations or identified through website searches, 313 met the inclusion criteria for clinical practice guidelines produced or reviewed between 2003 and 2007. The reduction from 1014 to 313 documents resulted from document duplications and documents that were not related to clinical practice or were public health guidelines. Ninety-one of the 313 clinical practice guidelines (29%) were classified as evidence-documented, and 60 of these described their evidence search and appraisal process in sufficient detail for it to be replicated. Box 1 shows the numbers of guidelines, with and without evidence documentation, by key health area.

Over 80% of the evidence-documented guidelines (76/91) cited one or more of the NHMRC “guidelines for guidelines” series of publications.1,7-12 Twenty-three percent of the guidelines without evidence documentation (50/222) also cited one or more of the publications from this series.

Box 2 shows numbers of guidelines, with and without evidence documentation, by category of guideline producer. More producers than guidelines were identified, because some guidelines had multiple producers. Eighty-two separate organisations or one-off collaborations were involved in producing guidelines.

More than half of the guidelines (173/313) did not include a statement about funding sources. Box 3 shows the numbers of guidelines, with and without evidence documentation, by key health areas and federal or state government agency support. Federal or state government agencies produced or funded 53% of the guidelines (167/313). Twenty-eight percent of government-supported guidelines (46/167) were categorised as evidence-documented. Almost all of the guidelines on drugs and alcohol received some government support. In contrast, most of the guidelines on cardiovascular and renal disease were funded by non-government sources. Pharmaceutical companies were identified as funders of 27 guidelines, 20 of which were evidence-documented.

Box 4 shows the numbers of evidence-documented guidelines produced or reviewed between 2003 and 2007 that would expire or require review if expiry dates of 5 years and 3 years were applied. Guideline producers specified a review date for 52% (47/91) of the evidence-documented guidelines in our study. For 14 of these guidelines, the guideline producer specified that a review should happen in 2007 or earlier but the guideline had not been reviewed by the specified date.

Most of the guidelines made no mention of possible competing interests (Box 5), and specific information on how any competing interests were managed during guideline development (eg, whether individuals with financial ties to specific companies participated in formulating recommendations for use of these companies’ products) was not published in any of the guidelines. Of 22 organisations that produced or co-produced five or more guidelines during the study period, competing interest policies specific to guideline development were available on the websites of four, while eight had general competing interest policies applying to committees, authorship or relationships with pharmaceutical companies. No information about competing interest policies was published on the websites of the remaining 10 organisations.

We identified 313 Australian clinical practice guidelines, including 91 evidence-documented guidelines. About half of the guidelines were funded or produced by government agencies, of which 28% were evidence-documented. Most of the guidelines made no mention of possible competing interests.

There has been a substantial growth in the number of Australian clinical practice guidelines in the past 15 years. Our study identified nine times more guidelines than identified in 1993,5 although our search was more extensive and not confined to guidelines for medical practitioners. In 1993, 15% of guidelines described the methods used to identify evidence and none gave full descriptions of these methods.5 In our study, nearly a third of guidelines provided specific details of the search and appraisal process, and almost 20% fully described the process in the guideline or an accompanying document. Given the extent to which NHMRC guidelines for guidelines1,7-12 (which have a strong focus on review and use of evidence) were cited by the guidelines identified in our study, it is reasonable to conclude that the NHMRC has been influential in improving this aspect of the quality of Australian clinical practice guidelines.

A 1997 study of 286 Australian general practitioners showed that 88% of GPs thought it was extremely or very important that a guideline be evidence-based.13 Documentation of the evidence search and appraisal process provides prospective users with information about the evidence basis for guideline recommendations. In our study, if there was a guideline in a clinical area where there was no evidence to guide practice, the guideline would still have been classified as evidence-documented if some details had been given about a literature search, even if the search proved fruitless. A majority of guidelines produced in several key health areas — such as asthma, cardiovascular disease, drugs and alcohol, and infectious diseases — did not document the evidence review process. This may be because the guidelines are based on consensus rather than evidence, because a non-systematic approach to evidence review was used, or because the guideline producers feel it is unnecessary to provide basic details about the process to prospective guideline users. Although there is little point in producing voluminous information about guideline development methods, if the information needed to guide clinical practice is not easily identifiable or accessible, it is important that guideline users are able to reach informed judgements about the basis for recommendations. Transparency and conciseness are not incompatible — many of the guidelines classified as evidence-documented in our study described their evidence-appraisal process in less than a page or provided information about the review in a separate, publicly available document.

Our study had some limitations. Assessors did not check whether the evidence search and appraisal process described for guidelines categorised as evidence-documented was the most appropriate for the topic under review, so some guidelines included in this category may be based on inappropriate or poor-quality reviews. On the other hand, some guidelines that were based on structured literature reviews may have been categorised as not evidence-documented because no details of the process were provided in a publicly available document. Similarly, data on funding and competing interests were derived from information provided in the guidelines and accompanying material. Also, despite extensive efforts to identify all national and state guidelines produced during the study period, some documents may have been missed.

The need for transparency and appropriate management of competing interests has been highlighted recently in Australia and overseas.14-18 It is important that guidelines are not biased, or perceived as biased, because of financial or other competing interests of individuals or groups involved in the guideline development process. Competing interests of guideline developers were poorly recorded in the guidelines that we assessed — they were infrequently reported within the guidelines, and policies on management of competing interests were not clearly stated by many organisations that produced multiple guidelines. This highlights the importance of clear local guidance and the need for revised NHMRC guidelines in this area.

The effort and resources needed to produce clinical practice guidelines are largely wasted if guidelines are not easily available to their target audience beyond the initial publication date. We found that there was no easy way to find locally produced guidelines on specific topics. For example, clinical guidelines on websites are hard to locate, particularly if the documents do not have the term “clinical guidelines” in their title or content. To help improve access to current guidelines, the NHMRC launched a Clinical Practice Guidelines Portal in February 2010 (http://www.clinicalguidelines.gov.au/) which provides information on current Australian guidelines, with links to the websites of agencies that publish guidelines. This should also help identify gaps and duplications in the pool of currently available guidelines.

Government agencies produced or funded just over half the clinical practice guidelines identified in our study. Despite this, most health departments had no central point of reference for guidelines and were unable to provide comprehensive information about the guidelines they produced, commissioned or helped fund. This probably reflects the “bottom-up” way in which guideline activity has grown in Australia, with government support for guideline development occurring as one of a range of activities within particular departmental health program areas rather than as a specifically identified function of government. The benefits of a bottom-up approach are that government health departments can be locally responsive to needs expressed by clinicians, and that there may be less clinical concern about bureaucratic involvement in clinical practice issues. However, the lack of coordination associated with this approach leads to gaps and duplications, and an inconsistent approach to ensuring that guidelines are of high methodological quality.

One key rationale for development of clinical practice guidelines is the rapid growth in research knowledge in some clinical areas, which means that recommendations for best practice may change as the results of important studies or new treatments become available. The value of evidence-based guidelines therefore depends on the extent to which guideline recommendations reflect current knowledge. However, a sporadic one-off approach to guideline funding means that there is no ongoing commitment to ensuring that guidelines in clinically important areas are kept up to date. Also, little research on when and how guidelines should be updated has been carried out. The only empirical study on how quickly good-quality guidelines become outdated that has been published to date showed that half of such guidelines were outdated within 5.8 years of being produced.19 The authors recommended that, as a rule, guidelines should be reassessed for validity every 3 years. If a scheduled review date of 3 years was applied to the group of evidence-documented guidelines in our study, 60 of the 91 guidelines would have required reassessment by the end of 2008; if a scheduled review date of 5 years was applied, 32 would have required review by the end of 2009. Within the study period, 15% (14/91) of the evidence-documented guidelines had already passed the review dates recommended by their authors.

High-quality clinical practice guidelines provide an opportunity to close the gaps between current clinical practice and best available evidence. Given the large number of clinical practice guidelines being produced in Australia, a substantial amount of money and effort is being spent on guideline production, with federal and state governments supporting much of this activity. However, there are deficiencies in the documentation of basic information that could help users assess guideline quality, such as information about literature searches and review processes and management of competing interests. A requirement that government-funded guidelines undertake systematic literature reviews and document these processes may result in the production of fewer high-quality guidelines, but would help assure potential users that guideline recommendations reflect the best available research knowledge. There would be similar value in a requirement for a transparent approach to identifying and managing competing interests. In addition, although a large number of the guidelines that we identified were government-funded, some health areas with heavy disease burden do not appear to have received government support for guideline production.

A coordinated national approach to funding, producing and updating high-quality guidelines in priority health areas may provide opportunities to gain greater returns on the current investment in clinical practice guidelines.