In recent years, systematic reviews have been published on the effectiveness of interventions for depression and anxiety in primary and community settings,1-6 in Australian schools,7 over the Internet,8 and using the telephone.9 These reviews focused on published research papers. However, other promising programs might have been implemented in community and general practice settings, but not formally described in peer-reviewed medical publications. These programs could potentially provide unique data on effectiveness, feasibility and implementation, which could inform the development of better service models for the delivery of health care in the future.
Information about such intervention programs for depression and anxiety might have been published in “grey literature”. Grey literature has been defined as “that which is produced on all levels of government, academic, business and industry in print and electronic formats, but which is not controlled by commercial publishers”,10 and it includes “conference abstracts, research reports, book chapters, unpublished data, dissertations, policy documents and personal correspondence”.11 Increasingly, grey literature includes content published on the Internet, and the distinction between grey and non-grey literature is consequently eroding.12 As grey literature frequently includes outcomes that might also be subsequently published in the formal “black” literature, a review of grey literature may capture information that is published in both formal scientific journals and grey sources.
There is evidence that grey literature is often commissioned and used in policy making by government, and contributes to decisions made by health practitioners.13 Outcomes reported in grey literature may differ from those in published reports. Indeed, the inclusion of grey literature in systematic reviews may reduce bias. In a recent meta-analysis, outcomes in the grey literature were reported to be less strong for a range of interventions than outcomes reported in the published literature.11
Here, we report a systematic review of the grey literature to identify primary care programs in Australia that aim to improve depression and anxiety outcomes in service users. We aimed to identify and describe programs that report effectiveness data, and we comment on the value of grey literature as a means of identifying useful improvements to primary care.
Our search strategy was formulated to locate reports on Australian projects concerned with interventions targeting the reduction of depression and anxiety in primary mental health care patients.
We initially searched two comprehensive archives of relevant Australian literature: the Primary Mental Health Care Australian Resource Centre (PARC; http://www. parc.net.au) and the Primary Health Care Research & Information Service (PHCRIS; http://www.phcris.org.au). PARC was systematically searched using the PARC subject heading “Depression”, and then using the terms “depression” OR “anxiety” to search all data on the site. Similarly, PHCRIS was searched using the terms “depression” OR “anxiety” in three sections of the site (General Practice and Primary Health Care Research Conferences, 2002–2006; Roadmap Of Australian primary health care Research [ROAR]; and Activities of Divisions).
Websites of agencies under the Better Outcomes Implementation Advisory Group and other relevant organisations (listed in Box 1) were hand-searched. Academic websites, particularly for universities with primary mental health care units as part of their medical or psychiatry schools (eg, University of Adelaide, Monash University, University of Western Australia, Flinders University, and the Australian National University), were also searched.
Submissions to the Senate Select Committee on Mental Health were searched, with the aim of identifying any projects not captured by other databases or websites. In particular, interventions described as promising in the Committee’s first report14 were examined in detail, and followed up through examination of relevant submissions.
Confidential evaluation reports on the Better Outcomes in Mental Health Care (BOiMHC) initiative were made available by the Australian General Practice Network.
To be included in the review, reports had to contain outcome data (or descriptions of outcome data) for patients or consumers, and be published between 1995 and 2006. For reports identified by these criteria, further details were sought by locating formally published or unpublished research trials or by contacting individual researchers. However, any programs with formally published reports were excluded if they were captured by our simultaneous systematic search of the international and Australian black published literature, which described intervention trials for anxiety and depression in primary care.15
Some of the material for this article has been extracted from a more extensive report of the project.16
A total of 642 reports describing projects or programs potentially meeting the inclusion criteria were identified (Box 2). Although we did not systematically code their content, the bulk of these reports focused on general practice education initiatives, including dissemination of guidelines for youth depression, programs to increase general practitioner skills, and materials for consumers. There were also reports describing programs to aid GP networking or peer support, models of general practice, and information about programs in schools, antenatal care, refugee settings, emergency departments, families and minority communities.
Many of the reports did not clearly describe a research program that addressed the research question (depression, n = 209; anxiety, n = 39), while others did not provide or describe outcome data (depression, n = 86; anxiety, n = 29) or were excluded for other reasons (depression, n = 160, anxiety, n = 12), usually because they focused on practitioner outcomes rather than patient or consumer outcomes. The remaining 107 reports were reassessed in greater detail by two researchers (H C and A G). Of these, 59 were excluded because closer examination revealed that they failed to report outcome data, and another five were excluded because they consisted of a case study, reported on a sample of children, were an examination of comorbidity, or reported data from overseas.
The remaining 43 reports form the basis of this review. Nine of these reported data that were also reported in the formal literature, but which had not been captured by our review of the published literature.15
Thirty reports of general practice programs met the inclusion criteria. Most (27) were evaluations of the BOiMHC initiative; the major report from these programs was a set of evaluations (here considered collectively) of the Access to Allied Psychological Services (ATAPS) component.17 Outcomes of ATAPS projects were reported to be positive, although there was little focus on patient outcomes. The Sixth Interim Evaluation Report contained the most comprehensive data on patient outcomes, reporting 1047 before-and-after scores, calculated as change scores.18 Of these change scores, 88% indicated that patients improved, in 5% they remained the same, and in 7% they worsened. However, these data were provided by fewer than 5% of patients, and some outcome data were counted twice.
Findings from the 26 BOiMHC evaluation reports provided by the Australian General Practice Network were universally positive. However, the number of people referred to the service, the numbers starting and completing the sessions, and the number returning to the GP for “exit” interviews were rarely recorded. Based on the 26 reports, we estimated that about 66% of the patients referred by GPs took up the service, 22% of all referred patients completed a full course with an allied health provider, and less than 20% returned to the GP for the exit interview.
Three other reports of general practice programs were located. One described the outcome of a study (the data of which were also published in the Medical Journal of Australia) that compared patients provided with six sessions of focused psychological therapies with other clinic attendees.19 Depression and anxiety symptoms improved in the treated group relative to the comparison group. However, the design of the study was weak, because the comparison group was not selected from the same pool of patients as the group provided with psychological intervention.
Another report investigated four models for delivering psychological services within general practice: a visiting Division-employed psychologist; a voucher system for a private psychologist; a clinical psychology registrar in the practice; and referral for post-natal depression.20 However, due to data collection problems, no comparisons could be made between the different models. Major methodological problems (eg, lack of control group) limited this evaluation. The preferred model based on the qualitative data collected was the psychologist resident in the practice.
Finally, the Bendigo Health Care Group described a program that introduced psychiatric nurses or mental health workers into general practices in the Bendigo and Murray Plains areas of Victoria.21 The outcome data from participants was not based on pre–post tests, but rather on retrospective ratings of improvement. There was strong agreement that the therapy improved function, but the weak design (no control group or baseline measure) limits conclusions from the study.
Five programs in residential care or the community were identified, with three of these also reported in the black literature.
Three of the programs were community based, with one of these finding that mental health first aid training improved mental health knowledge and decreased symptoms of distress.22 The physical activity program “Get up and go: walking away from depression” reported reduced depression scores (measured by the Zung Self-Rating Depression Scale) for the majority of participants who maintained involvement with the project.23 Finally, the South East Anxiety Disorder Project provided anxiety support services over a 9-week period. Researchers reported that: “Qualitative and quantitative measures showed that participants attending the anxiety management course benefited greatly.”24
The two residential care programs identified involved either screening or early intervention in residential settings. One reported improved outcomes on the Geriatric Depression Scale for an intervention group relative to a comparison group.25 However, this study had a relatively weak design, because the intervention and comparison groups were not examined simultaneously. The other study examined the effect of a screening procedure as a means of improving mental health outcomes for people entering residential care facilities. It found that screening and early referral to a psychogeriatric service did not improve mental health outcomes.26
There were three reports of programs located in schools: the “Adolescents Coping with Emotions” (ACE) program,27 the MindMatters program,28 and specific anxiety prevention programs.29 Two of these reports were also found in the black literature. Outcomes for the ACE program were better in the intervention group,27 while the MindMatters program reported an improved willingness to discuss mental health issues and increased help-seeking behaviour.28 Although not strictly “school programs”, the third paper reported positive outcomes for specific anxiety prevention programs in adolescents.29
Five Internet or computer-based programs were identified, each of which was reported to be associated with a reduction in depression or anxiety symptoms. The Internet programs reported were MoodGYM and BluePages,30 Panic Online,31 and Rural Carers Online,32 the latter involving a pilot program of the effectiveness of an information technology intervention on reducing depression symptoms and social isolation in older carers. The fifth program involved computer-assisted group therapy in a general practice in Adelaide.33
This systematic review clearly indicates that a large amount of activity in mental health services across a range of settings in Australia is described in the grey literature. However, the bulk of the identified literature did not report on mental health outcomes for patients using the services described. For example, many reports that focused on government-funded initiatives such as the BOiMHC program, or large school initiatives, reported information about the content of the program or the training required to undertake it, but failed to report the mental health outcomes of the service recipients.
Of the 43 programs with data on outcomes, nine had reports also published in the black literature. These included Internet, community, and school-based programs, all of which were associated with positive outcomes. However, only one formally published report was identified for the 30 general practice programs described in the grey literature. This clearly indicates that the grey literature contains unique data that will not be found in the black literature. These unpublished reports therefore have the potential to provide information about the usefulness of important general practice programs, such as the BOiMHC initiative.
Our findings raise questions about the role and value of grey literature. It appears that grey literature may serve a vital role in disseminating information about the existence of new programs and policies, the realities of service provision, and the difficulties of implementation. However, our review clearly indicates that it provides little information to practitioners about the effectiveness of new programs. The reasons for this low level of reporting of target outcomes are unclear, but it may reflect the low priority given to patient outcomes by the authors of these reports; inadequate funding, time or available expertise to collect data; or some other factor.
Our review also indicates that many publicly funded programs in Australia are either not being evaluated in terms of the effect they have on outcomes for primary care or community groups, or these evaluations are not being made publicly available. In addition, we found that the quality of current evaluations of outcomes is relatively poor. Most evaluation designs simply measured post-test, without baseline measures, so the value of these evaluations for improving best practice is questionable. Such reports would certainly fail to meet criteria for inclusion in research databases that provide the evidence base for published clinical practice guidelines.
Finally, based on this and our review of the black literature,15 it is clear that, despite efforts such as those of the Australian Primary Health Care Research Institute (which aims to develop a robust primary health care research environment), research describing general practice outcomes for anxiety and depression is rarely published in formal medical or health journals in Australia, unlike in countries such as the United Kingdom.34
Some limitations of our review need to be acknowledged. First, potentially effective programs were excluded if no details of outcomes were provided in the grey literature we reviewed. Second, broader search terms such as “mental health reform” or “mental health” might have yielded further data. However, despite these limitations, our findings are relatively straightforward.
A core finding was that the grey informal sources and black published, peer-reviewed literature may operate in relatively separate spheres, with unique information available in each. To bridge the gap between these spheres, there may be a role for an information “clearinghouse” of evidence-supported programs in mental health, which would allow dissemination, education and collaboration among researchers, health practitioners, consumers and policymakers. Additionally, innovative schemes to encourage the collection of outcome data in evaluation studies, to promote effective programs, and to recognise the importance of data on patient outcomes are needed.35
1 Organisations whose websites were hand-searched
Better Outcomes Implementation Advisory Group agencies:
Australian General Practice Network
beyondblue: the national depression initiative
Rural Doctors Association of Australia
Australian Psychological Society
Australian Medical Association
Royal Australian and New Zealand College of Psychiatrists
Royal Australian College of General Practitioners
Australian Government Department of Health and Ageing
Mental Health Council of Australia
VicHealth Centre for the Promotion of Mental Health and Community Wellbeing
Australian Infant, Child, Adolescent and Family Mental Health Association
Australian Network for Promotion, Prevention and Early Intervention for Mental Health (Auseinet)
National Youth Affairs Research Scheme
Youth Studies Australia
Australian Research Alliance for Children and Youth
General Practice Evaluation Program
Australian Primary Health Care Research Institute
Australian Institute of Health and Welfare
- Helen Christensen1
- Kathleen M Griffiths2
- Amelia Gulliver3
- Centre for Mental Health Research, Australian National University, Canberra, ACT.
The systematic review on which this article is based was conducted by Helen Christensen, Kathleen Griffiths, Amelia Gulliver, Dannielle Clack, Marjan Kljakovic and Leanne Wells. It was performed with funding from the Australian Primary Health Care Research Institute, and the Australian Government Department of Health and Ageing under the Primary Health Care Research, Evaluation and Development Strategy. Helen Christensen is a recipient of an NHMRC Fellowship No. 366781.
Kathleen Griffiths and Helen Christensen have developed online psychoeducational and automatic psychological skills training programs, including MoodGYM and BluePages.
- 1. Gilbody S, Whitty P, Grimshaw J, Thomas R. Educational and organizational interventions to improve the management of depression in primary care: a systematic review. JAMA 2003; 289: 3145-3151.
- 2. Bower P, Gilbody S, Richards D, et al. Collaborative care for depression in primary care. Making sense of a complex intervention: systematic review and meta-regression. Br J Psychiatry 2006; 189: 484-493.
- 3. Gilbody S. What is the evidence on effectiveness of capacity building of primary health care professionals in the detection, management and outcome of depression? Health Evidence Network report. Copenhagen: World Health Organization Regional Office for Europe, 2004.
- 4. Neumeyer-Gromen A, Lampert T, Stark K, Kallischnigg G. Disease management programs for depression: a systematic review and meta-analysis of randomized controlled trials. Med Care 2004; 42: 1211-1221.
- 5. Tsai AC, Morton SC, Mangione CM, Keeler EB. A meta-analysis of interventions to improve care for chronic illnesses. Am J Manag Care 2005; 11: 478-488.
- 6. Weingarten SR, Henning JM, Badamgarav E, et al. Interventions used in disease management programmes for patients with chronic illness — which ones work? Meta-analysis of published reports. BMJ 2002; 325: 925.
- 7. Neil AL, Christensen H. Australian school-based prevention and early intervention programs for anxiety and depression: a systematic review. Med J Aust 2007; 186: 305-308. <MJA full text>
- 8. Griffiths K, Farrer L, Christensen H. Clickety-click: e-mental health train on track. Australas Psychiatry 2007; 15: 100-108.
- 9. Leach LS, Christensen H. A systematic review of telephone-based interventions for mental disorders. J Telemed Telecare 2006; 12: 122-129.
- 10. Auger CP. Information sources in grey literature. 4th ed. London: Bowker-Saur, 1998.
- 11. Hopewell S, McDonald S, Clarke MJ, Egger M. Grey literature in meta-analyses of randomized trials of health care interventions (Cochrane review). The Cochrane Library, Issue 2, 2007. Chichester, UK: John Wiley & Sons, Ltd.
- 12. Banks MA. Towards a continuum of scholarship: the eventual collapse of the distinction between grey and non-grey literature. In: Farace D, editor. Proceedings of GL7: Seventh International Conference on Grey Literature; 2005 Dec 5–6; Nancy, France. http://dlist.sir.arizona.edu/1079/01/GL7Paper%5FFinal.pdf (accessed May 2008).
- 13. LaPelle NR, Luckmann R, Simpson EH, Martin ER. Identifying strategies to improve access to credible and relevant information for public health professionals: a qualitative study. BMC Public Health 2006; 6: 89.
- 14. Parliament of Australia, Senate Select Committee on Mental Health. A national approach to mental health — from crisis to community. First report. Canberra: Senate Printing Unit, 2006: 19-25. http://www.aph.gov.au/senate/committee/mentalhealth_ctte/report/index.htm (accessed May 2008).
- 15. Christensen H, Griffiths KM, Gulliver A, et al. Models in the delivery of depression care: a systematic review of randomised and controlled intervention trials. BMC Fam Pract 2008; 9: 25.
- 16. Christensen H, Griffiths K, Wells L, Kljakovic M. Models of mental health delivery: efficacy, support and policy. Canberra: Australian Primary Health Care Research Institute, 2006. http://www.anu.edu.au/aphcri/Domain/MentalHealth/approved_final_25_christensen.pdf (accessed May 2008).
- 17. Better Outcomes in Mental Health Care. Access to Allied Psychological Services (ATAPS). Project reports. http://boimhc.org/bin/view/Main/ATAPSReports (accessed May 2008).
- 18. Kohn F, Morley B, Pirkis J, et al. Evaluating the Access to Allied Psychological Services component of the Better Outcomes in Mental Health Care program. Sixth interim evaluation report. Melbourne: Program Evaluation Unit, 2005.
- 19. Vines R. Clinical psychology in general practice: a collaborative model of mental health service delivery. PARC Update 2003; (7). http://www.parc.net.au/PARCUpdateSept03.htm#Clinicalpsych (accessed May 2008).
- 20. Tyson G, Roufeil L, Kiernan M. Evaluation of the Central West Division of General Practice Better Outcomes in Mental Health Care Initiative Pilot Program. Bathurst: School of Social Sciences, Charles Sturt University, 2004.
- 21. Oerlemans M. Better Outcomes in Mental Health Care Initiative: Access to Allied Health Services Pilot Project: evaluation. Bendigo: Bendigo Health Care Group, 2004.
- 22. Kitchener BA, Jorm AF. Mental health first aid training in a workplace setting: a randomized controlled trial. BMC Psychiatry 2004; 4: 23.
- 23. Norton J. Get up and go: walking away from depression. J Aust N Z Stud Serv Assoc 2006; 27: 44-55.
- 24. Dugdale S, Braunack-Mayer A, Moss J, et al. “I haven’t looked back”: evaluation of the South East Anxiety Disorder Project (ProSEAD). Final report. Adelaide: University of Adelaide, 1999.
- 25. Llewellyn-Jones RH, Baikie KA, Smithers H, et al. Multifaceted shared care intervention for late life depression in residential care: randomised controlled trial. BMJ 1999; 319: 676-682.
- 26. Kotynia-English R, McGowan H, Almeida OP. A randomized trial of early psychiatric intervention in residential care: impact on health outcomes. Int Psychogeriatr 2005; 17: 475-485.
- 27. Kowalenko N, Rapee RM, Simmons J, et al. Short-term effectiveness of a school-based early intervention program for adolescent depression. Clin Child Psychol Psychiatry 2005; 10: 493-507.
- 28. Hazell T, Vincent K, Greenhalgh S, et al. Evaluation of MindMatters: eighth interim report. Newcastle: Hunter Institute of Mental Health, 2005.
- 29. Dadds M, Seinen A, Roth J, Harnett P. Early intervention for anxiety disorders in children and adolescents. Vol. 2. In: Kosky R, O’Hanlon A, Martin G, Davis C, series editors. Clinical approaches to early intervention in child and adolescent mental health. Adelaide: Australian Early Intervention Network for Mental Health in Young People, 2000: 27-33.
- 30. Christensen H, Griffiths KM, Jorm AF. Delivering interventions for depression by using the internet: randomised controlled trial. BMJ 2004; 328: 265.
- 31. Klein B, Richards JC, Austin DW. Efficacy of internet therapy for panic disorder. J Behav Ther Exp Psychiatry 2006; 37: 213-238.
- 32. Dow B, Black K. Rural carers online: a feasibility study. Final report. Melbourne: National Ageing Research Institute, 2006.
- 33. Mitchell J, Howell C, Turnbull D, Murphy M. Computer-assisted group therapy for the treatment of depression and anxiety in general practice. Prim Care Mental Health 2005; 3: 27-40.
- 34. Seddon ME, Marshall MN, Campbell SM, Roland MO. Systematic review of studies of quality of clinical care in general practice in the UK, Australia and New Zealand. Qual Health Care 2001; 10: 152-158.
- 35. Dobbins M, Rosenbaum P, Plews N, et al. Information transfer: what do decision makers want and need from researchers? Implement Sci 2007; 2: 20.
Abstract
Objective: To identify reports in the “grey literature” of programs conducted in Australian primary care to improve depression and anxiety outcomes, and to examine these reports for evidence of effectiveness.
Methods: A systematic search was undertaken for grey literature reports using primary health care research databases, community and professional websites, clearinghouse sources, government reports, and reports from the Australian General Practice Network. Reports were included if they related to programs targeting depression or anxiety, contained qualitative or quantitative effectiveness data, and were published during 1995–2006.
Results: In total, 642 reports were identified, of which 43 met inclusion criteria. Of the 43 programs described, 30 were delivered in general practice, five in the community or residential care, three in schools, and five were Internet or computer based. Nine programs were also reported in the formal “black” literature, but most, including the Better Outcomes in Mental Health Care initiative, were not. Limited data on effectiveness or patient outcomes were available in the grey literature.
Conclusions: There is currently no single service that identifies, describes and catalogues the range and effectiveness of mental health initiatives in Australia. There may be a role for a mental health information “clearinghouse” to facilitate dissemination and education, and to promote collaboration among researchers, practitioners, consumers and policymakers. Innovative schemes to disseminate evidence-based models and to encourage the collection of data on patient outcomes in such programs are needed.