Quality of Australian clinical guidelines and relevance to the care of older people with multiple comorbid conditions

Clinical guidelines have been developed to improve health care by increasing the uptake of evidence-based treatments and reducing the use of unnecessary, ineffective or harmful interventions.1 However, several recent studies have found that the methodological quality of clinical guidelines was highly variable.2-5 Guidelines developed by professional societies were of poorer quality than guidelines produced by governmental agencies.6-8 Researchers and commentators have noted that most guidelines for chronic diseases do not modify or discuss the applicability of their recommendations to older patients with multiple comorbid conditions, and that they provide limited guidance on combined use of treatments for different diseases.9,10

In Australia, the National Health and Medical Research Council (NHMRC) has adopted standards with which guideline developers need to comply if they want NHMRC approval. NHMRC standards require strong methods, the involvement of a multidisciplinary working panel and a public consultation process.11,12 In practice, many Australian clinical guidelines are produced by professional societies and charities without seeking NHMRC approval.

We aimed to assess the quality of selected Australian national clinical guidelines and their relevance to older people with multiple illnesses. We also compared the quality of NHMRC-approved guidelines with those that were not NHMRC-approved.

Seventeen national guidelines were selected, including five for cardiovascular health, four for cancer, four for mental health, two for respiratory health, one for musculoskeletal conditions and one for diabetes mellitus (Box 3).17-33 No national Australian guideline for the management of arthritis was identified. All guidelines were developed by professional or disease-oriented charity organisations. Only two received governmental funding. Six had been approved by the NHMRC, including four for cancer, one for cardiovascular health and one for diabetes.

This study showed that the quality of Australian guidelines for chronic conditions listed as National Health Priority Areas was average (around 50%) in three domains, low in two domains and good in one domain. The mean quality of guidelines not approved by the NHMRC was below 50% in all domains except clarity and presentation. Only a few guidelines not approved by the NHMRC described the methods used for selecting the evidence or formulating the recommendations. Our findings are consistent with the results of other studies which showed that quality of guidelines produced by specialist societies was low,6 and that high-quality guidelines were more likely to be produced by government-funded agencies or developed within a structured and coordinated program to produce clinical practice guidelines.5,7,34

Compared with an international study that assessed the quality of 86 guidelines in 10 European countries and Canada,5 mean scores achieved by Australian guidelines in our study were higher for stakeholder involvement (45.8% v 33.6%), rigour of development (40.9% v 36.9%), clarity and presentation (71.3% v 57.2%), and lower for scope and purpose (54.3% v 66.1%), applicability (21.6% v 31.3%) and editorial independence (11.8% v 47.8%). This was mainly owing to the high mean scores for NHMRC-approved guidelines, which were higher than mean scores observed in all other international studies that have used the AGREE instrument, in all domains except scope and purpose, and editorial independence.2-5,34-39

All the Australian guidelines we studied, regardless of whether they were approved by the NHMRC, performed poorly in the domain of editorial independence. Disclosure of conflicts of interest of guideline development members was not a requirement of the NHMRC until a recent update of its standards in October 2007.11

The applicability domain evaluates issues that are pertinent to guideline implementation, such as organisational barriers, cost implications and monitoring criteria. The mean score for the Australian guidelines was low, suggesting implementation barriers were not sufficiently overcome in Australian guidelines. However, it should be noted that the AGREE instrument does not assess the actual implementation of guidelines.

Low-quality scores in the rigour of development domain raise concern about guideline validity. This needs to be addressed as the promotion of invalid guidelines may result in ineffective or unsafe treatments, or inefficient use of resources. The costs involved with the development of rigorous guidelines based on systematic reviews of the evidence may be a barrier for a single Australian professional society or charity. There is a limited amount of public funding for guideline development. Only two guidelines in our sample were funded by state or federal government departments. Increased resources for developing of guidelines may be required to overcome this barrier.

Another reason for not following the NHMRC process is that several professional organisations have chosen to adapt existing international guidelines for use in Australia rather than develop their own. This option avoids duplicating efforts when high-quality international guidelines are already available on similar topics. However, there is no validated process for adapting guidelines produced in one cultural and organisational setting for use in another.40 NHMRC guidance in this area would be welcome to support the efforts of professional societies.

Only half of the guidelines studied covered treatment for older patients and for patients with a single comorbid condition. Only one discussed issues of older people with multiple conditions. These results are consistent with a previous American study.9 The 2004–05 Australian National Health Survey reported that 80% of Australians aged 65 years or older had three or more chronic conditions,41,42 and the absence of guidance on how to manage older patients with multiple chronic conditions means that treatment decisions are left with individual practitioners. It has been proposed that meta-guidelines should be developed for the most common patterns of chronic conditions, which would define how to prioritise the many interventions that could be undertaken in people with multiple conditions.43

The AGREE instrument and the instrument used to assess relevance examined different aspects of patient-centred care. A minority of guidelines reported having involved at least one consumer in part of the guideline development (35%), discussed patients’ preferences (47%) and considered the burden of comprehensive treatment on patients or caregivers (29%). The revised NHMRC standards for externally developed guidelines require that consumers should be involved early in the development process.11 However, “effective” patient involvement in the development process is still being debated.44 Research on how patient preferences can be integrated into an evidence-based model of care is also still in its infancy.44

Our study has several limitations. The selection criteria allowed the inclusion of a wide range of important health conditions and of guideline producers. However, our results may not be generalisable to all Australian guidelines on chronic diseases or to guideline producers. We used the generally agreed definition of a guideline as “a set of systematically developed statements to assist practitioner and patient decisions about appropriate health care for specific clinical circumstances”.45 Because of the definition adopted and the format of the AGREE instrument, we did not include documents produced by the Royal Australian College of General Practitioners (RACGP) or Therapeutic Guidelines Limited in our analysis.

Poor quality of guidelines could also be the result of poor reporting of the methods used during guideline development rather than actual poor process. However, guideline readers should be able to assess the reliability of guidelines for themselves. Clear documentation of how guidelines were developed may also enhance their acceptance by clinicians.

The AGREE criteria provide a well validated assessment instrument, and the inter-rater agreement was excellent or good for most items in our study. However, ratings can be influenced by the reviewer’s background knowledge of the guideline topic or the method of guideline development. In our study, one reviewer had significant experience in guideline development, while the other had not been involved in guideline development. The high κ scores suggest the background knowledge of the reviewers did not significantly influence the results.

Our results show that the quality of Australian clinical guidelines is low when they have not been approved by the NHMRC. Professional societies and charities should be encouraged and supported to adopt NHMRC standards and procedures for developing guidelines. Future Australian guidelines should also provide more guidance on the management of older people with multiple comorbid conditions.

3 Individual standardised AGREE domain scores for the 17 guidelines studied