It has been estimated that 6.4 million Australian adults — almost half the adult population — accessed the Internet during 2000.5 Reportedly, more Internet users search the Web for information on depression than any other health condition.6 This is not surprising given the high level of disability associated with depression in the community7 and the fact that the Web provides a convenient, anonymous means of obtaining information about the problem. However, much of the depression information on the Web is of low quality8-10 and originates in the United States. If Australian consumers are to benefit from Web-based media, they need to know which sites are of high quality and appropriate to local needs.
Because consumers are more likely to trust information on a website that is recommended by their doctor,11 general practitioners are uniquely placed to advise people about which depression websites to visit.11 However, there are currently no systematic guides to Australian depression websites on which GPs can base their advice.
Another way that consumers or non-specialist healthcare providers might judge the quality of sites is by indirect indicators that do not rely on specialist knowledge. Although many such indicators have been proposed, their validity as markers of content quality has not been established.8,12 Recently, the DISCERN scale13 has generated considerable interest as a potential indicator of website quality. In particular, the scale is intended to assist people without content expertise assess the quality of written health treatment information by systematically rating a number of attributes of a publication, such as the extent to which it describes treatment alternatives and their risks and benefits; the degree to which the information appears balanced and unbiased; and whether the publication documents areas of uncertainty. To our knowledge, there have been no studies of the validity of DISCERN as an indicator of the content accuracy of Web-based or other written health material.
Here we intend to:
provide information for consumers and healthcare providers about Australian depression sites and the quality (accuracy and comprehensiveness) of their content;
explore the validity of the DISCERN system as an indicator of the quality of treatment content; and
determine the accessibility of Australian depression websites.
The methodology used in this study is an extension of that described in our previous study of the quality of "popular" international depression sites.8
Potentially relevant Australian depression sites were identified (November to December 2001) by entering the query term "depression" into each of 17 Australian search engines and six major search and metasearch engines that permit searches restricted to the Australian domain.14,15 Only sites appearing in the top 200 results of a search engine list and containing at least eight internal Web pages focusing on depression were included. Three additional, newly released sites not returned by the search engines were also included.
Twenty-seven sites were found. Of these, 12 (clearing house, bipolar disorder, postnatal depression, web-based cognitive therapy) were excluded from formal analysis, leaving 15 sites (Appendix).
Each author independently rated the sites in terms of site characteristics and coverage, quality of content and potential quality indicators using a standard protocol sheet. Disagreements between item ratings were resolved through discussion (except for the DISCERN and global subjective judgements).
Site characteristics (eg, ownership structure, scope, privacy policy, registration required) and coverage of symptoms/diagnosis, self-assessment screening, treatment, prevention, risk factors, prevalence, resources, and specific groups were rated for each site.
The total guideline score was the number of items (maximum, 20) for which site information agreed with depression guidelines produced by the Centre for Evidence Based Mental Health (CEBMH) at Oxford.16 Items covered antidepressants and associated management issues, St John's wort, psychotherapy, counselling, bibliotherapy, and exercise.
The number of evidence-based treatments (evidence+) (maximum, 8) and the number of non-evidence-based treatments (evidence−) (maximum, 26) (Box 1) that were recommended as effective were calculated for each site. Evidence-based treatments were defined as those interventions that are supported by a systematic review of the evidence as effective.17 Non-evidence-based interventions were those which, on systematic review of the available evidence, do not seem to be effective or which have been the subject of little or no appropriate research.17
Interventions were typically not accompanied by a reference to scientific evidence. We therefore classified each site according to whether it always, or almost always, cited scientific evidence to support claims (rating, 3); provided scientific evidence to support more than three but not the majority of claims (rating, 2); provided evidence to support between one and three of the claims (rating, 1); or never cited supporting scientific evidence (rating, 0).
Each judge independently rated the sites using the 15-item, five-point DISCERN scale.13 A DISCERN score was calculated by summing the 15 scores across the scale for a site. An additional item referred to the overall quality of the treatment information and was analysed separately.
The site characteristics are summarised in Box 2. All sites provided information for consumers. Significantly, only a third of the sites had an editorial board. All sites published some treatment information and most included information about symptoms and diagnosis (14 sites), prevalence (13 sites), stigma reduction (11 sites), risk factors (11 sites), and resources (10 sites). Half the sites provided a self-assessment screening test on the site. Only four sites included information on prevention and only four had information on specific groups. Four sites included a bulletin board and two had chatrooms, but none provided online counselling or psychotherapy.
On average, sites contained correct information for less than half of the guideline items (mean, 7.5 out of 20; SD, 4.8; range, 0–14). This was often due to omission of relevant material rather than inclusion of inaccurate information. With respect to the guideline items, most sites acknowledged that antidepressants are effective (14 sites) and non-addictive (11 sites). However, only three sites clearly specified that a trial of six weeks is required before discontinuing a particular antidepressant, and only six indicated that antidepressants should be continued for at least four to six months after improvement or that a discontinuation syndrome can occur with abrupt cessation of antidepressants. Few sites (two) indicated that problem solving might be an effective treatment for depression. Finally, of the 11 sites that mentioned counselling, only two warned that counselling is not by itself an effective intervention.
On average, sites recommended more than half of the eight evidence-based treatments (mean, 4.7 out of 8; SD, 2.4; range, 0–8) but few of the non-evidence-based treatments (mean, 1.4 out of 26; SD, 0.8; range, 0–3). Antidepressants were recommended most often (14 sites), followed by ECT (11 sites), exercise (10 sites), interpersonal therapy (9 sites), and cognitive behaviour therapy (8 sites). Of the non-evidence-based treatments, supportive counselling was most often recommended (9 sites).
The average issues score was 11.8 out of 22 (SD, 4.0; range, 4–18). Positive features were that all sites promoted speaking to healthcare professionals and stated the risk of suicide. Most indicated that depression can (14 sites) and should (13 sites) be treated, and provided details of contact organisations (12 sites). Most sites also acknowledged that treatments other than antidepressants are effective and that psychological therapy can be effective in its own right. On the other hand, 13 sites failed to indicate the relative effectiveness of different treatment options compared with antidepressants. Only six sites indicated that psychological therapy (cognitive behaviour therapy or interpersonal therapy) is as effective as antidepressants for mild to moderate depression. About half the sites failed to mention side effects, dangers and contraindications of antidepressants, and few (4 sites) mentioned the drawbacks of non-antidepressant treatments (for example, lack of availability of individual cognitive behaviour therapy).
As there was a high correlation between ratings of the two judges (r = 0.92; P < 0.001), the scores for the two judges were averaged. The mean global score was 4.6 out of 10 (SD, 2.5). Individual ratings ranged from 1 to 8.5, suggesting that the quality of the treatment information was variable but that some sites were of high quality.
The mean scientific evidence score was 0.9 out of 3 (SD, 0.9; range, 0–3). Only one site consistently referred to the level of scientific evidence in support of claims and only two other sites referred to evidence for three or more recommended treatments. Of the remaining sites, half did not refer to any scientific evidence.
The total DISCERN scores for the two judges were significantly correlated (r = 0.88; P < 0.001), as were the "overall rating" scores (r = 0.65; P < 0.01). The judges' ratings were therefore averaged for each measure. The mean DISCERN score was 44.1 out of 60 (SD, 12.1; range, 26.5–62). The mean score on the overall rating DISCERN item was 2.9 out of 5 (SD, 1.0; range, 1–4.5), which suggests that on average the sites had significant shortcomings.
On average, search engines returned 1.7 (SD, 1.2; range, 0–4) sites in the top 10 results list, and 4 (SD, 2.5; range, 0–10) sites in the top 200 list. DepressioNet was the most accessible site, being returned first and in the top 10 results by 11 and 17 search engines, respectively. No other site was returned in top place. beyondblue and Climbing the Cliffs of Depression were the next most frequently returned top 10 and top 200 sites. The well-known Reach Out! site received few hits.
Box 3 shows the intercorrelations between each of the content and potential quality indicator scores. With one exception (evidence–), all content scores (guideline, issues, evidence+) were significantly correlated with each other and with the DISCERN "overall rating" score. There was also a significant correlation between the extent to which sites mentioned scientific evidence in support of treatments (scientific score) and scores on all content measures except evidence–.
There was no relationship between the accessibility of a site (as judged by the number of search engines retrieving it in the top 10 or the top 200 results) and the quality of the site (as judged by any of the content or other measures and excluding the new sites InfraPsych and BluePages, which would not have been indexed by public search engines at the time of the study).
We have systematically identified Australian websites that provide information about depression. On average, scores on measures of content quality were relatively low, a result that is broadly consistent with the conclusions of previous studies that, overall, the quality of depression information on the Web is not high.8-10
Nevertheless, there was considerable variability in the comprehensiveness and the quality of sites, and all sites had strengths as well as weaknesses. The consistently best-scoring sites included two university-based sites (BluePages and CRUfAD), the site of the National Depression Initiative (beyondblue) and the privately owned site InfraPsych (Box 4). These sites had the best average ranks across the four main content measures (guideline, issues, evidence+, global), and achieved top scores on the evidence-based guideline scale and top ratings on at least three of the content measures. Other specific advantages of these sites are summarised in Box 4.
Several other sites were notable in providing useful information with respect to particular aspects of depression. These were Depression doctor.com (offers extensive information about depression management and methods of coping with antidepressant side effects), myDr (includes a searchable MIMS database), and DepressioNet (provides consumer support and networking, extensive lists of sources of help). Dark Side of the Mood stands out for its accessible and concise presentation of above-average quality of content.
Despite their limitations, all sites provided useful information. The fact that all or most sites indicated that depression can be treated, encouraged patients to seek expert advice, attempted to destigmatise depression and indicated that antidepressants are effective and non-addictive treatments is likely to encourage help-seeking and facilitate treatment compliance. Providing information about symptoms, as most sites did, might facilitate correct diagnosis, as patients who self-label as depressed are more likely to receive an appropriate diagnosis.18
On the other hand, there is a clear need to improve the coverage and the accuracy of content in a number of areas, including details of the recommended duration of antidepressants to avoid relapse, the time needed to trial an antidepressant and the importance of ceasing antidepressants slowly to avoid discontinuation effects. This information could improve compliance, lessen the likelihood of relapse and premature abandonment of effective treatments, and, by avoiding discontinuation symptoms, improve the likelihood that patients will be willing to take antidepressants in the future. Sites also need to provide more information about the relative effectiveness of different treatments (eg, that cognitive behaviour therapy is as effective as antidepressants for mild to moderate depression). Currently, many sites recommend "counselling" as a treatment for depression, although supportive therapy is not by itself an effective intervention for depression. More generally, sites should ensure that they consider the consumer perspective, both with respect to content and style.
Finally, there is a need to provide comprehensive, high quality information about depression for adolescents. Currently, no site adequately fulfils this role.
One aim of our study was to identify possible indicators of quality of treatment content. This is an important issue for all health sites, not just those concerned with depression. We found that ratings on the DISCERN instrument and specifying an evidence base were consistently associated with the quality of site content. The advantage of DISCERN is that it can be administered by people with no content knowledge in an area. It seems unlikely that the typical consumer or carer seeking depression information would take the time or be sufficiently motivated to learn how to apply the instrument. Nevertheless, DISCERN might assist people who are constructing lists of or links to recommended resources (eg, government portals such as HealthInsite, HealthySA) and for people who are developing health content for the Web.
The other aim of this study was to determine how accessible Australian depression sites are to people not already aware of their existence. We found that local sites are not easily found with search engines, even using search strategies designed to limit the search to Australian sites. It has been reported that few people search beyond the first 10 links in a search engine list. Apart from DepressioNet and, to a lesser extent, beyondblue and Climbing the Cliffs of Depression, the depression sites we reviewed rarely, if ever, appeared in the first 10 results. Significantly, there is no association between the quality of the content of a site and the order in which it appears on the search engines.
Detailed consideration of how high-quality sites can be accessed more readily by the Australian public is beyond the scope of this article. However, it would be helpful if such sites could be promoted by high-traffic government and other portals. At the time of writing, the government-owned quality portal HealthInsite listed only one of the sites reviewed in this study (myDr) and none of the four "recommended" sites.
Our study has a number of limitations. First, we are the joint authors of one of the sites (BluePages) and this might have influenced our judgements. In addition, one of the measures (evidence+) is based on a review of which we were co-authors and which forms the basis for the treatment section of the BluePages.17 However, the general pattern of the results does not change if BluePages is excluded from the analysis.
A second limitation of this study is that the judges rated the sites on both content and potential quality measures. Future studies using different raters for these measures and a larger sample size are needed.
In addition, this study did not incorporate evaluations by consumers. We plan to conduct further studies using DISCERN and other ratings produced by consumers and non-technical assessors.
We also acknowledge that websites are rarely static and that no account was taken of information on linked external sites.
A final limitation of this study is that it is confined to depression information. Online counselling is becoming increasingly popular. DepressionNet alone claims to attract more than 1 000 000 visits per quarter. Clearly, the next challenge will be to develop methods for assessing the quality of online counselling services and online support groups.
1: Evidence-based and non-evidence-based treatments counted in assessing the websites17
Evidence-based treatments |
|
Antidepressants |
Bibliotherapy |
Cognitive behaviour therapy |
Electroconvulsive therapy |
Exercise |
Interpersonal therapy |
Light therapy (seasonal affective disorder) |
St John's wort |
Non-evidence-based treatments |
|
Alcohol for relaxation |
Caffeine avoidance |
Chocolate |
Colour therapy |
Dance therapy |
Fish oils |
Ginkgo biloba |
Ginseng |
Glutamine |
Homoeopathy |
Hypnotherapy |
Lemon balm |
Meditation |
Music |
Natural progesterone |
Painkillers |
Pets |
Phenylalanine |
Pleasant activities |
Selenium |
Sugar avoidance |
Supportive counselling |
Tranquillisers |
Vervain |
Vitamins (other than folate) |
2: Site characteristics of Australian depression information websites (n = 15)
Characteristic |
Number (%) of sites |
||||||||||
Ownership structure |
Individual Organisation Unknown |
3 (20%) 11 (73%) 1 (7%) |
|||||||||
Scope |
Specific (mental health) Broad (health) |
12 (80%) 3 (20%) |
|||||||||
Editorial board |
No Yes |
10 (67%) 5 (33%) |
|||||||||
Healthcare professional involved |
No Yes |
5 (33%) 10 (67%) |
|||||||||
Promotion of products/services |
No Yes |
10 (67%) 5 (33%) |
|||||||||
Privacy policy |
No Yes |
6 (40%) 9 (60%) |
|||||||||
Disclaimer |
No Yes |
1 (7%) 14 (93%) |
|||||||||
Feedback mechanism |
No Yes |
1 (7%) 14 (93%) |
|||||||||
Collects personal information |
No Yes |
6 (40%) 9 (60%) |
|||||||||
Register to access all information |
No Yes |
10 (67%) 5 (33%) |
|||||||||
3: Intercorrelations* between content measures and potential quality indicators
Issues |
Evidence+ |
Evidence− |
Global |
DISCERN |
Scientific |
||||||
Guideline |
0.85† |
0.75† |
0.26 |
0.96† |
0.92† |
0.71† |
|||||
Issues |
0.85† |
– 0.07 |
0.85† |
0.89† |
0.67† |
||||||
Evidence+ |
0.17 |
0.86† |
0.76† |
0.66† |
|||||||
Evidence– |
0.26 |
0.12 |
0.38 |
||||||||
Global |
0.91† |
0.78† |
|||||||||
DISCERN |
0.85† |
||||||||||
* Results are based on Pearson rho correlations, except for results involving Evidence+, Evidence– and Scientific, which are based on Spearman rho correlations. Evidence+ = number of endorsed evidence-based treatments. Evidence– = number of endorsed non-evidence-based treatments. † P < 0.01. |
|||||||||||
4: Consumer guide to Australian depression websites
Recommended sites overall (alphabetical order) |
|||||||||||
beyondblue <http://www.beyondblue.org.au> |
|||||||||||
BluePages <http://bluepages.anu.edu.au> |
|||||||||||
CRUfAD <http://www.crufad.unsw.edu.au> |
|||||||||||
Infrapsych <http://www.infrapsych.com.au> |
|||||||||||
Recommended subsections or aspects of sites (alphabetical order of features) |
|||||||||||
Accessible and above-average quality |
Dark Side of the Mood |
||||||||||
Antidepressants |
|||||||||||
management of depression |
depression doctor.com, Infrapsych |
||||||||||
side effects |
myDr |
||||||||||
ways of coping with side effects |
depression doctor.com |
||||||||||
Cognitive behaviour based strategies * |
beyondblue, CRUfAD |
||||||||||
Evidence-based information about treatments |
BluePages |
||||||||||
Focused search of other depression sites |
BluePages |
||||||||||
Online consumer support |
DepressioNet |
||||||||||
Online depression screening |
BluePages, CRUfAD, InfraPsych |
||||||||||
* A cognitive behaviour therapy module is under construction on Infrapsych. |
|||||||||||
- Kathleen M Griffiths1
- Helen Christensen2
- Centre for Mental Health Research, The Australian National University, Canberra, ACT.
This study was funded by NHMRC New Program Grant No. 179805. We would like to thank Kimberley Evans, Claire Kelly, Sara Vancea, and Rhonda Sattler for their assistance with the project.
Kathleen Griffiths and Helen Christensen were co-authors of the depression website BluePages, which was included in this review. They are also co-authors of a review of the effectiveness of interventions for depression17 on which one of the quality-of-content measures was based.
- 1. Australian Bureau of Statistics. Household use of information technology, Australia. Canberra: ABS, 2001. (Catalogue no. 8146.0.)
- 2. Taylor H. Explosive growth of "cyberchondriacs" continues. (The Harris Poll #47) 5 August 1999. <http://www.harrisinteractive.com/harris_poll/index.asp?PID=117>. Accessed 26 January 2001.
- 3. Murray CJL, Lopez A, editors. The global burden of disease: a comprehensive assessment of mortality and disability from diseases, injuries and risk factors in 1990 and projected to 2020. Cambridge, MA: Harvard University Press, 1996.
- 4. Griffiths KM, Christensen H. Quality of web based information on treatment of depression: cross sectional survey. BMJ 2000; 321: 1511-1515.
- 5. Gretchen K, Berland MD, Elliott MN, et al. Health information on the Internet: Accessibility, quality, and readibility in English and Spanish. JAMA 2001; 285: 2612-2621.
- 6. Lissman TL, Boehnlein JK. A critical review of internet information about depression. Psychiatr Serv 2001; 52: 1046-1050.
- 7. Cain MM, Sarasohn-Kahn J, Wayne JJ. Health e-people: the online consumer experience. Five-year forecast. Oakland, CA: California HealthCare Foundation, 2000. Available at <http://www.chcf.org/topics/view.cfm?itemID=12540>. Accessed 27 January 2001.
- 8. Jadad AR, Gagliardi A. Rating health information on the internet: navigating to knowledge or to Babel? JAMA 1998; 279: 611-614.
- 9. Charnock D, Shepperd S, Needham G, Gann R. DISCERN: an instrument for judging the quality of written consumer health information on treatment choices. J Epidemiol Community Health 1999; 53: 105-111.
- 10. The Major Search Engines, Metacrawlers, Regional search engines. In: SearchEngineWatch. <http://searchenginewatch.com/links/>. Accessed November 2001.
- 11. Searchme. The definitive list of Australian search engines and directories. In: The Web shed. <http://searchme.com.au>. Accessed November 2001.
- 12. Centre for Evidence Based Mental Health, University of Oxford. A systematic guide for the management of depression in primary care: treatment. 1998. <http://cebmh.warne.ox.ac.uk/cebmh/guidelines/depression/treatment.html>. Accessed 27 January 2002.
- 13. Jorm A, Christensen H, Griffiths K, et al. Help for depression: what works (and what doesn't). Canberra: Centre for Mental Health Research, 2001.
- 14. Hickie IB, Davenport TA, Scott EM, et al. Unmet need for recognition of common mental disorders in Australian general practice. Med J Aust 2001; 175 Suppl Jul 16: S18-S24.
- 15. Silverstein C, Henzinger M, Marais H, Moricz M. Analysis of a very large Alta Vista query log. SRC Technical Note 1998-014. California: Digital Systems Research Centre, 1998.
Abstract
Objectives: To provide information about Australian depression sites and the quality of their content; to identify possible indicators of the quality of site content; and determine the accessibility of Australian depression web sites.
Design: Cross-sectional survey of 15 Australian depression web sites.
Main outcome measures: (i) Quality of treatment content (concordance of site information with evidence-based guidelines, number of evidence-based treatments recommended, discussion of other relevant issues, subjective rating of treatment content); (ii) potential quality indicators (conformity with DISCERN criteria, citation of scientific evidence); (iii) accessibility (search engine rank).
Results: Mean content quality scores were not high and site accessibility was poor. There was a consistent association between the quality-of-content measures and the DISCERN and scientific accountability scores. Search engine rank was not associated with content quality.
Conclusions: The quality of information about depression on Australian websites could be improved. DISCERN may be a useful indicator of website quality, as may scientific accountability. The sites that received the highest quality-of-content ratings were beyondblue, BluePages, CRUfAD and InfraPsych.