Use of systematic reviews of randomised trials by Australian neonatologists and obstetricians

Abstract

Introduction

Reviews address this problem, but conventional review methodology is unscientific^4,5 and open to selection biases inherent in the publication process.^5,6 This has led to the development of systematic reviews. A properly conducted systematic review begins with an exhaustive search for published and unpublished randomised trials addressing a well defined research question. The second step is to identify trials of adequate quality to contribute to decision-making. The results of included trials are then pooled, or "meta-analysed", to arrive at a quantitative estimate of the benefits and harms of treatment. Systematic reviews published by the Cochrane Collaboration are also continuously updated by specialist review groups.⁷

Systematic reviews in pregnancy and childbirth have been available for some years. In 1988 a comprehensive trial register became commercially available in database form as the Oxford database of perinatal trials.⁸ Effective care in pregnancy and childbirth, ⁹ a textbook based on this research synthesis, was published the following year, with a guide for non- specialist readers.¹⁰ A companion textbook, Effective care of the newborn infant,¹¹ appeared in 1992. From 1993, systematic reviews relating to pregnancy and childbirth were available electronically in the Cochrane pregnancy and childbirth database.¹² All these reviews are currently being updated for The Cochrane library.¹³

These new publications represent prototypes for future publications in other fields of practice. However, their impact on clinical practice remains uncertain, despite several surveys to ascertain the extent of their dissemination and uptake.^14-18 To make a practical difference, systematic reviews must be readily available to clinicians who are aware of them, who use them, and who implement their findings. To ascertain whether this is occurring in Australia, we conducted a survey of neonatologists and obstetricians with the aims:

• To gauge awareness of, access to, and use of the Oxford database, Cochrane database, and Effective care textbooks;

• To place these resources in the context of other information sources that inform practice in obstetrics and neonatology; and

• To ascertain how often and why these resources are consulted, predictors of their use, and whether their use has led to reported changes in clinical practice.

Methods

Participants

Eligible clinicians were sent an introductory letter, telephoned, and invited to schedule a confidential, 10-minute telephone interview about information sources used in clinical practice. Systematic reviews were not mentioned until the interview was under way.  

Interviews

Respondents were then asked about their access to and use of computers, and asked directly whether they had heard of, had access to, and used the Cochrane database, Oxford database, and Effective care of the newborn infant (neonatologists) or Effective care in pregnancy and childbirth (obstetricians).

Those who reported using any of these resources were asked what a systematic review was, how often they consulted that resource, what they used it for, whether they thought using it had made any difference to their clinical practice and, if so, whether they could name a treatment policy they had altered in response to evidence from a systematic review. To count as users of systematic reviews, respondents had to mention randomised trials or meta-analysis in their description of a systematic review.

The final questions sought demographic information on clinical experience, place of education, attendance at professional meetings, research publications, academic appointments, and qualifications. A research degree was defined as a doctorate, relevant master's degree, or bachelor's degree of science in medicine, held in addition to basic medical and specialty qualifications. Reports of authorship were verified by searching MEDLINE and by screening abstracts.

For neonatologists it was noted whether they held full- or part-time positions, whether they worked in a hospital with an obstetric unit, whether they headed a neonatal unit, and in which unit they worked. For obstetricians, it was noted whether they held a position as a full-time staff specialist in a public hospital. They were asked how many deliveries they attended each year, and whether they worked at any hospital with an accredited RACOG training post.

The interview schedule was modified after a pilot study with seven neonatologists in New Zealand. The main survey was conducted between August and December 1995, with ethics approval from the ANZNN, RACOG and the University of Sydney. All interviews were conducted by the first author (C F C J).  

Analysis

Results

For the purpose of keeping up with new clinical developments, respondents favoured journals, conferences and meetings, colleagues and MEDLINE (in that order) over systematic reviews. For the purpose of clinical problem-solving, colleagues, MEDLINE, journals, and other printed medical literature were favoured over systematic reviews as sources of information. Although the percentages favouring each source varied between the two specialties and according to whether the respondent was simply nominating the resource or nominating it as the most important resource, the rank order of the resources remained consistent.

In each specialty, 21% of respondents mentioned systematic reviews without prompting from the interviewer when naming the information sources they used, or when describing their computer use. When questioned directly about systematic reviews, 71 neonatologists (72%) and 55 obstetricians (44%; 95% confidence interval, 36%-51%) reported using them in either electronic database or textbook form. The confidence interval indicates the reliability of the sample prevalence as an estimate of the population prevalence. The Figure shows reported levels of awareness of, access to and use of systematic reviews by specialty.

Predictors of
use of systematic
reviews

Among obstetricians, four factors sig nificantly and independently predicted use of systematic reviews: familiarity with computers, mode of practice (full-time staff specialists were more likely to use systematic reviews than others), clinical experience (recent graduates were more likely to use systematic reviews), and number of deliveries per year (likelihood of using systematic reviews increased with this number).  

Descriptors of
use of systematic
reviews

Of the 71 neonatologists who reported using systematic reviews, 58% said that this had changed their clinical practice in some way, and 44% gave at least one example of a treatment policy they had altered in response to a systematic review. The most common examples were treatment of respiratory disorders, and use of steroids and indomethacin. Corresponding percentages for obstetricians who reported using systematic reviews were 80% (reported a change in practice) and 71% (gave at least one example). The most common example was use of steroids in management of preterm rupture of membranes.  

Discussion

The findings of this survey are based on self-reporting. Although the survey was designed to minimise over-reporting, it was still subject to imprecision in respondents' recall about (for example) the frequency with which they used systematic reviews, and to difficulties in attributing practice changes to their use. Nevertheless, this was the first study of the use of systematic reviews by Australian clinicians, and the findings have immediate and practical relevance for organisations such as universities, government agencies and specialty colleges that are trying to improve access to the best available evidence and to promote its use.^20-23

The association between use of systematic reviews and attendance at Australian Perinatal Society meetings among neonatologists could reflect greater receptiveness to innovations among clinicians who attend professional meetings. However, it also supports the perception (reported elsewhere¹³ ) that professional organisations play an important role in disseminating research findings. Future dissemination efforts could usefully concentrate on these organisations.

In both specialties, familiarity with computers predicted use of systematic reviews. Better access to digital information technology and training in its use is therefore likely to enhance uptake of these reviews. This might include improved access to The Cochrane library and the Internet, and training workshops for searching specialist databases. The remaining predictors of use of systematic reviews, as well as the overall contrast between neonatology and obstetrics, suggest that specialists practising primarily in public hospitals are more likely to use systematic reviews than those working primarily in private practice. Special efforts are therefore needed to reach the latter. As systematic reviews do not appear to rank highly among the information sources used in clinical practice, it is important to stress their relative advantages. Uptake of this innovation appears more likely among recent graduates, with senior clinician researchers acting as "product champions".²⁴

Although debate persists as to whether practising clinicians accept the innovations of evidence-based medicine,²⁵ our survey found evidence that Australian neonatologists and obstetricians use systematic reviews, and appear to modify their practice accordingly. Efforts are needed to enhance the use of these reviews, and to conduct further evaluations of their influence on clinical practice. In attempting to improve practice standards, it is important to pursue methods which have been shown to be effective.²⁶  

Acknowledgements

References

1. Antman EM, Lao J, Kupelnick B, et al. A comparison of results of meta-analyses of randomized control trials and recommendations of clinical experts: treatments for myocardial infarction. JAMA 1992; 268: 240-248.
2. Stross JK, Harlan WR. The dissemination of new medical information. JAMA 1979; 241: 2622-2624.
3. Warren KS. From papyrus to parchment to paper to pixels: information technology and the future of biomedical publishing. In: Lock S, editor. The future of medical journals. London: BMJ, 1991: 127-146.
4. Mulrow CD. The medical review article: state of the science. Ann Intern Med 1987; 106: 485-488.
5. Light RJ, Pillemer DB. Summing up: the science of reviewing research. Cambridge: Harvard University Press, 1984.
6. Dickersin K, Berlin JA. Meta-analysis: state-of-the-science. Epidemiol Rev 1992; 14: 154-176.
7. Sackett DL. The Cochrane Collaboration. ACP J Club 1994; 120 Suppl 3: A-11.
8. Chalmers I, editor. Oxford database of perinatal trials. Version 1.2, disk issue 7 (Spring). Oxford: Oxford University Press, 1992.
9. Chalmers I, Enkin M, Keirse MJNC. Effective care in pregnancy and childbirth. Oxford: Oxford University Press, 1989.
10. Enkin M, Keirse MJNC, Chalmers I. A guide to effective care in pregnancy and childbirth. Oxford: Oxford University Press, 1989.
11. Sinclair JC, Bracken MB. Effective care of the newborn infant. Oxford: Oxford University Press, 1992.
12. Enkin MW, Keirse MJNC, Renfrew MJ, Neilson JP, editors. Cochrane pregnancy and childbirth database. Oxford: Update Software, 1993.
13. The Cochrane Library [database on disk and CD-ROM]. The Cochrane Collaboration. Oxford: Update Software; 1996. Updated quarterly.
14. Lomas J. Retailing research: increasing the role of evidence in clinical services for childbirth. Milbank Q 1993; 71: 439-475.
15. Stocking B. Implementing the findings of effective care in pregnancy and childbirth in the United Kingdom. Milbank Q 1993; 71: 497-523.
16. Paterson-Brown S, Fisk NM, Wyatt JC. Uptake of meta-analytical overviews of effective care in English obstetric units. Br J Obstet Gynaecol 1995; 102: 297-301.
17. Paterson-Brown S, Wyatt JC, Fisk NM. Are clinicians interested in up to date reviews of effective care? BMJ 1993; 307: 1464.
18. Hyde C. Who uses the Cochrane pregnancy and childbirth database? BMJ 1995; 310: 1140-1141.
19. Zeger SL, Liang K-Y. Longtitudinal data analysis for discrete and continuous outcomes. Biometrics 1986; 42: 121-130.
20. Rychetnik L. Evidence-based medicine in the GMP (Graduate Medical Program). Focus: graduate medical degree news and events. Sydney: 1995: 1-2.
21. Liddle J, Williamson M, Irwig L. Method for evaluating research and guideline evidence. Sydney: NSW Department of Health, 1996.
22. National Health and Medical Research Council. Clinical practice guidelines: the management of early breast cancer. Canberra: NHMRC, 1995.
23. Commonwealth Department of Human Services and Health. Guidelines for the development and implementation of clinical practice guidelines. Canberra: the Department, 1995.
24. Rogers EM. Diffusion of innovations. New York: MacMillan, 1983.
25. Miles A, Bentley P, Polychronis A, Grey J. Evidence-based medicine: why all the fuss? J Evaluation Clin Pract 1997; 2: 83-85.
26. Grimshaw JM, Russell IT. Effect of clinical guidelines on medical practice: a systematic review of rigorous evaluations. Lancet 1993; 342: 1317-1322.