eMJA: Hirst & Ward, Clinical practice guidelines: reality bites

Evidence-based Medicine

Clinical practice guidelines: reality bites

Geoffrey H L Hirst and Jeanette E Ward

MJA 2000; 172: 287-291

→ Other articles have cited this article

Abstract - What is LUTS? - Why develop? - Searching for evidence - Prostate cancer screening - The consultation process - Derivative guidelines - Implementation - Updating guidelines - Conclusions - References - Authors' details
Make a comment - Register to be notified of new articles by e-mail - Current contents list - More articles on Administration and health services

Abstract

In 1995, the National Health and Medical Research Council (NHMRC) announced its commitment to developing evidence-based clinical practice guidelines "to promote best practice linked to outcomes and effective cost management". To date, resources for dissemination and implementation have been identified for only two guidelines developed by the NHMRC as part of that commitment. Clinical practice guidelines for the management of men with lower urinary tract symptoms (LUTS) were launched in 1997. We were members of the working party that developed these guidelines, and here we give our personal account, offering insights into the tensions and contradictions impeding the translation of political commitment to evidence-based medicine into policy and practice.

Introduction

For at least a decade, there has been vocal support in Australia for the concept of evidence-based medicine and its most obvious progeny, clinical practice guidelines (CPGs). Yet, we perceive a "healthy cynicism" is emerging. Perhaps too much has been inferred from weak evidence. CPGs have been perceived as "cook book" medicine. Health and medical research might be criticised for focusing more on "academic" outputs than on generating evidence relevant to the needs of practising doctors. As participants in an initiative to develop evidence-based CPGs -- Clinical practice guidelines for the management of uncomplicated lower urinary tract infections in men¹ -- under the auspices of the National Health and Medical Research Council (NHMRC), we describe our experiences in the light of these concerns.

What is LUTS?

Lower urinary tract symptoms (LUTS) in men are very common. Nearly 40% of men aged over 45 years admit to having some lower urinary tract symptoms, and this prevalence increases with age.² For many years, the public has been encouraged to equate symptoms with disease. Given media interest in prostate cancer and the consequent community awareness, it is also not surprising that Australian men worry that LUTS might signal malignancy.^3,4

LUTS was previously known as BPH (benign prostatic hyperplasia or benign prostatic hypertrophy) or prostatism, implying that the underlying cause of the symptoms was a disorder of the prostate.⁵ However, as the prevalence of LUTS is the same for men and women in age-matched cohorts, it is plausible that LUTS may partly be explained by ageing. Invasive and expensive investigations, medical treatments and surgical procedures may have inadvertently reinforced the perception that LUTS is potentially life-threatening.^3,4

Why develop evidence-based CPGs in Australia about LUTS, and how?

In 1994, the United States Agency for Health Care Policy and Research (AHCPR) published CPGs entitled Benign prostatic hyperplasia: diagnosis and treatment.⁵ This evidence-based document heralded changes in fundamental concepts about LUTS in men and questioned accepted practice. As Australian guidelines seemed timely, the NHMRC saw merit in addressing this issue.

In 1995, a 44-page document entitled Guidelines for the development and implementation of clinical practice guidelines was published by the NHMRC to help those developing CPGs.⁶ It advised that CPG working parties should be comprised of representatives of all relevant stakeholders, but gave no direction on how to select them. Instead, the NHMRC relied on the perspicacity of the stakeholder organisations to understand the complexity of guideline development and the likely controversies. The multidisciplinary members of the LUTS working party worked cohesively, even though their selection was more accidental than planned.

The principal medical organisations with a legitimate interest in the management of men with LUTS are the Royal Australian College of General Practitioners (RACGP) and the Urological Society of Australasia (USA). The Consumers' Health Forum was also asked to nominate a representative.

Evidence-based medicine emphasises the importance of consumer participation in health decision-making at all levels.⁷ Articulate, well-informed and effective consumer representation is crucial in the development of CPGs. Yet such representation raises many confronting questions. In our case, ought the consumer representative have LUTS? Could one individual reflect the vast and diverse needs and views of the affected consumer group? Might consumers self-nominate or be selected because they have an "axe to grind"? How might consumer representatives react to discrepancies between the evidence they access during the deliberations of the working party compared with previous treatment they may themselves have received?

We also were concerned about conflict of interest. While the CPG development process should focus exclusively on the evidence, every member of the LUTS working party (including ourselves) potentially had a conflict of interest. Representatives of professional groups were expected to convey the concerns and preferences of their constituency. Yet the influence of pecuniary interests might be more subtle. For example, specialists may benefit financially from an aggressive approach to referral, investigations and treatments. Bureaucrats might put cost savings or short term political gain before outcomes.

Searching for evidence to answer clinical questions

An immediately apparent and perennial problem for the LUTS working party was the impoverished nature of the evidence available. There was very little evidence from well-designed randomised-controlled trials (level II evidence) or from meta-analyses of such trials (level I evidence). Having to consider evidence predominantly at lower levels of the NHMRC evidence taxonomy, the LUTS working party soon found this taxonomy too coarse to classify the nature of the evidence about LUTS treatments. At that time (1995), the NHMRC included within one broad band of evidence (classified as level IV) those "opinions of respected authorities, based on clinical experience, descriptive studies, or reports of expert committees".⁶ Of necessity, we devised a more detailed taxonomy whereby three subcategories of level IV evidence could be distinguished as follows:

IV-1 Evidence from descriptive studies including case series, case reports and cross-sectional studies.

IV-2 Published policies, recommendations or opinions of recognised experts, organisations or learned colleges, including endorsement of IV-3 evidence by recognised Australian bodies such as the RACGP and the Urological Society of Australasia.

IV-3 Consensus opinion of the working party not endorsed formally by recognised bodies.¹

The NHMRC has since excluded anecdote and expert opinion entirely from its more recent classification of evidence,⁸ under which level IV evidence now only includes case series (either post-test only or pretest-post-test). Furthermore, the NHMRC taxonomy had been designed to classify the evidence for treatments. Neither its original nor its revised taxonomy is appropriate for the appraisal of the evidence for diagnostic investigations and prognostic factors.

Even the desirability of level I or II evidence can be challenged if it fails to pass the test of "clinical relevance". For example, randomised-controlled trials of the use of the drug finasteride for LUTS showed a statistically significant improvement in symptom score points from 2.5 to 2.8 in men treated with this drug compared with controls. As changes of at least three such points are required for men to experience a subjective sense of benefit in their quality of life, this evidence failed the "test of clinical relevance".¹ To quote Gertrude Stein, "for a difference to be a difference it has to make a difference"!

Another difficult task for the working party was making recommendations when published evidence was non-existent or inconclusive. Guided by Davidoff,⁹ the working party recognised two typical reactions in response. An interventionist approach assumes that patients are best served by providing a service for which benefit is as yet speculative, on the basis that treatment is better than no treatment. By contrast, a conservative approach demands a greater certainty of benefit, assuming there is always a risk of iatrogenic harm from intervention in the absence of definitive evidence of benefit. The LUTS working party soon realised that the clinical management of LUTS often represented a choice between conservative and interventionist interpretations of inadequate and circumstantial evidence. The annual healthcare cost of treating men with LUTS was estimated to be $177 million.¹⁰ Adopting a conservative rather than an interventionist approach in response to an impoverished evidence base would readily save $27 million per year, without compromising patient outcomes.⁵

Prostate cancer screening ... again

At the time of the LUTS working party's deliberations, the Australian Health Technology Advisory Committee had already established a lack of compelling evidence for prostate cancer screening for the symptomless.¹¹ However, it vacillated about the issue of prostate-specific antigen (PSA) testing of men with LUTS. The AHCPR guidelines included seven pages about this issue, but made no recommendation.⁵

The LUTS working party "bit the bullet". It found no evidence that men with LUTS were at any greater risk of prostate cancer than men with no symptoms,¹ and advised against PSA testing in men presenting with LUTS. It added the caveat that, if a doctor or his patient chose to disregard this recommendation, such testing should only be done with properly informed consent.¹

Because of the somewhat entrenched positions about PSA testing by various groups, the working party correctly anticipated this might be a controversial recommendation, but we did not anticipate the longevity of this controversy. As recently as September 1999, the working party was polled about proposed changes to its recommendation as a result of ongoing "significant concerns" conveyed to the NHMRC. However, the evidence for an association between LUTS and early prostate cancer remains dubious.¹²

The consultation process

The NHMRC requires all draft guidelines be submitted for public consultation.⁶ To fulfil this requirement, it is sufficient to place an advertisement inviting interested parties to request a copy of the draft document. The LUTS working party decided to mail copies of the draft guidelines to all urologists in Australia, all identifiable general practitioner organisations, a selection of consumers and to six international urological referees.

Responses from the Australian urological community varied widely. Some were very supportive: many were negative about CPGs in general and the recommendations of the LUTS draft in particular. Urologists also expressed anxiety about the purpose and outcome of the economic analysis of implementing the CPGs. By contrast, the international urological referees were uniformly supportive and complimentary (see Box 1).

Our recollection, however, was that this consultation process largely failed to bring to the attention of the working party any objective evidence of which it was unaware. Perhaps it was a predominantly political process. On reflection, we recommend the development of a standard proforma for submissions in the mandatory CPG consultation process that reinforces rather than undermines its evidence-based principles.

Derivative guidelines for GPs and consumers

The LUTS working party attempted to establish the most appropriate form of information for general practitioners by conducting a focus group with GPs from diverse backgrounds. Opinions were divided. Some preferred an "in-depth" reference, while others preferred a single-sided laminated sheet listing key messages as bullet points. As there is no body of empirically derived insights regarding optimal formats for derivative GP guidelines, we had nothing but our creativity and experience to guide our development of the GP version of the guidelines.¹³

Similar frustrations beset the development of derivative information for consumers. An author experienced in writing for a lay readership was needed, but such a person was not present from the beginning of the CPG development process to understand fully the nuances of the many complex issues. The NHMRC's guideline for guidelines⁶ did not provide detailed and empirically based advice for "transforming" information from a technical CPG into appropriate consumer information. We did our best,¹⁴ attempting a humorous approach to appeal to the target group in response to findings of qualitative research.¹⁵ The NHMRC subsequently realised the importance of more practical advice, commissioning a consumer information "toolkit".¹⁶

Implementation: who really cares?

The cost of developing the LUTS CPGs was approximately $160 000 (C Lutton, NHMRC, personal communication). Original correspondence between the NHMRC and the chair of the LUTS working party suggested a shared and genuine commitment to implementation.¹⁷ The NHMRC commissioned research to establish baselines against which dissemination and implementation could be evaluated.¹ Accordingly, the working party developed a list of performance indicators with which to evaluate the effectiveness of such efforts (Box 2).

Not unreasonably, the working party expected that wide dissemination and implementation of the derivative documents -- for general practitioners and consumers -- would be similar to that for the NHMRC early breast cancer guidelines.¹⁸ Instead, we were advised that distribution should be revenue-neutral, or possibly even revenue-generating. All versions of the LUTS CPGs had to be purchased from the Australian Government Publishing Service. As only 14% of GPs currently have access to the Internet in their practices,¹⁹ Internet availability would not have compensated for this restricted distribution. In any case, only the consumer version of the CPGs can be downloaded from the NHMRC publications website (<http://www.health.gov.au/nhmrc/ publicat/cp-home.htm>, accessed January 2000).

Between the initiation of the LUTS working party and the launch of the LUTS CPGs in April 1997, both the federal government and the membership of the NHMRC changed. The NHMRC newsletter included a cover story about the LUTS CPGs two years after the launch in 1997,²⁰ and, to our knowledge, this was the only prominent promotion of the guidelines by the NHMRC. Awareness among GPs of this newsletter and the proportion prompted to obtain the LUTS CPGs in response are unknown. Recently, it was claimed that the LUTS CPGs "have not been widely adopted" by urologists, generating consensus among the Australian Prostate Cancer Collaboration against active promotion of a "LUTS education message in the community".²¹ Yet an international publishing house readily agreed to publish an evidence-based book to address men's questions.²² An application by one of us (G H L H) to the Strategic Research Development Committee of the NHMRC in response to its call for proposals in relation to its Evidence-Based Clinical Practice Research Program (EBCPRP) was unsuccessful in obtaining funds for an implementation trial. Of the seven indicators for dissemination and implementation proposed in the guidelines (Box 2), none has been measured since their release.

Updating guidelines: the need for commissioned research

CPG working parties inevitably identify important issues that require research. Arguably, organisations sponsoring guideline development ought to support mechanisms to "fast track" priority research. The LUTS working party identified 10 research issues that it considered would add significantly to currently available evidence.¹ To our knowledge, this research has not been pursued. The guideline development process also called for "regular" review of guidelines to ensure that their recommendations remain current.⁶ The LUTS guidelines are due to be updated. Yet our reflections on their development suggest this could also be a disappointing exercise. With the benefit of hindsight, we recommend that all working parties developing guidelines regularly compare their experiences against a comprehensive checklist (Box 3). If, at any point, a specific mismatch is identified, steps could be taken quickly to rectify the situation. This did not happen with the LUTS guidelines, and it appears their time has passed. We remain disappointed that the genuine effort of every member of the LUTS working party to produce sound and influential guidelines has largely "come to naught".¹⁷ We have no evidence to suggest practice has improved.

Conclusions

An effective organisation is needed in Australia for the synthesis, propagation and generation of evidence in response to priority health issues. As suggested elsewhere, it could be a reformed NHMRC,²³ or the proposed National Institute for Clinical Studies.²⁴ Until there is such an organisation, better outcomes promised through the development and implementation of evidence-based CPGs will not be realised.

References

Clinical practice guidelines for the management of uncomplicated lower urinary tract symptoms in men. Canberra: National Health and Medical Research Council, 1997.
Ward JE, Sladden M. Urinary symptoms in older men, their investigation and management: is there an epidemic of undetected morbidity in the waiting room? Family Practice 1994; 11: 251-259.
Pinnock CB, Marshall VR. Troublesome lower urinary tract symptoms in the community: a prevalence study. Med J Aust 1997; 167: 72-75.
Ward JE, Hughes A-M, Hirst GHL, Winchester L. Men's estimate of prostate cancer risk and self-reported rates of screening. Med J Aust 1997; 167: 250-253.
Benign prostatic hyperplasia: diagnosis and treatment. Clinical practice guideline no. 8. Agency for Health Care Policy and Research, US Department of Health and Human Services, 1994 (AHCPR Publication No. 94-0582).
Guidelines for the development and implementation of clinical practice guidelines. Canberra: National Health and Medical Research Council, 1995.
McDonald J. Evidence-based health care from the consumer perspective. Aust Health Consumer 2000; 1 (Summer): 8-10.
A guide to the development, implementation and evaluation of clinical practice guidelines. Canberra: National Health and Medical Research Council, 1999.
Davidoff F. Evangelists and snails redux: the case of cholesterol screening. Ann Int Med 1996; 124: 513-514.
Butler J. Economic aspects of lower urinary tract symptoms in men and their management. Working paper 37. Canberra: National Centre for Epidemiology and Population Health, Australian National University, 1996.
AHTAC. Prostate cancer screening. Canberra: AGPS, 1996.
Young J, Muscatello D, Ward J. Are men with lower urinary tract symptoms at increased risk of prostate cancer? A systematic review and critique of the available evidence. Br J Urol Internat 2000. In press.
"Is it my prostate Doc?": a guide for general practitioners. Canberra: National Health and Medical Research Council, 1997.
To pee . . . or not to pee . . . A guide for men about their urinary symptoms. Canberra: National Health and Medical Research Council, 1997.
Pinnock C, O'Brien B, Marshall V. Older men's concerns about their urological health: a qualitative study. Aust N Z J Public Health 1998; 22: 368-373.
How to present the evidence for consumers: preparation of consumer guidelines. Canberra: National Health and Medical Research Council, 2000.
Hirst G. Clinical practice guidelines: to what end? Med J Aust 1997; 167: 288.
Clinical practice guidelines: the management of early breast cancer. Canberra: National Health and Medical Research Council, 1995.
Young J, Ward J. General practitioners' use of evidence databases. Med J Aust 1999; 170: 56-58.
Hirst G. Men and their urinary symptoms: guidelines support a conservative approach to the management of many men. NHMRC News. 1999; 1(2): 3-5.
Pinnock C. Report from the first national meeting of the Australian Prostate Cancer Collaboration Education Group. Cancer Forum 1999; 23: 165-167.
Hirst G, Wilde S. Your prostate, your choices: the news may be more reassuring than you think. Sydney: Bantam Books, 1999.
Health and Medical Research Review Strategic Review. The virtuous cycle: working together for health and medical research. Canberra: AGPS, 1998.
Australian Department of Health and Aged Care. The 1999-2000 Health Budget in detail. Major boost to medical research takes Australia into the century of healing. Budget media release, 11 May, 1999. <http://www.health.gov.au/pubs/ budget99/media/mrmw991.htm> (accessed 15 February, 2000).

Authors' details

Department of Urology, Mater Hospitals, Brisbane, QLD.
Geoffrey H L Hirst, MB BS, FRACS, Urologist.

Needs Assessment & Health Outcomes Unit, Central Sydney Area Health Service, Sydney, NSW.
Jeanette E Ward, PhD, FAFPHM, Director, and Clinical Associate Professor, Department of Public Health & Community Medicine, University of Sydney.

Reprints will not be available from the authors.
Correspondence: Dr G H L Hirst, Taylor Medical Centre, 40 Annerly Road, Woolloongabba, QLD 4102.
drhirstATgil.com.au

Other articles have cited this article: