The public gains much of its health and medical knowledge from newspapers, radio and television.1-5 Even doctors may find out about new medical developments through the lay press. Medical news reports can have dramatic effects: they can alter consumer behaviour, like the recent wholesale abandonment of hormone replacement therapy after adverse publicity; they can put pressure on politicians and drug subsidy programs; and affect company shares.6
Medical news reports obviously need to be balanced and accurate. One study of television coverage of medical issues in Australia has shown a bias towards bizarre stories, and those that issue moral warnings, discredit well known people, spruik medical “breakthroughs”, or affirm folk remedies.1
Another concern is honesty in the attribution of media reports. For example, media releases about new pharmaceutical products may be prepared by public relations companies and used as the basis of news stories directly and without attribution. A study in the United States found that most articles about pharmaceutical products failed to include complete information about benefits, harms and costs of treatment and did not report financial ties between study groups or experts and pharmaceutical manufacturers.7 Similar findings came from a recent survey of the media in Canada, and there have been calls for more direct and honest reporting of the results of research into the effects of medicinal drugs.8,9
There are many reasons why journalists may find it difficult to write accurate and balanced articles about new medical advances, including lack of time or space, pressure from editors, and difficulty in accessing independent expert opinion.10 Some of these factors may be hard to change, but there is evidence in other professional areas that timely feedback on performance can improve standards.11 We believe that web-based audits of medical news reports have the potential to improve the quality of media articles.
We report here the early experience of operating a website media doctor, which publishes quality assessments and critiques of Australian lay press news articles about medical treatments, including drugs, procedures and diagnostic tests. The primary aim of the critiques is to encourage journalists to report all important information about new treatments, including novelty, availability, benefits, harms, costs, adverse effects and, where possible, the opinions of experts who are free of obvious conflicts of interest. Media doctor does not provide medical advice and does not assess the quality of the evidence on which the reviewed articles are based; instead it concentrates on the articles themselves. In this respect it differs from the “Hitting the headlines” project in the United Kingdom.12
Current news articles about medical treatments are identified by daily website searches. During the study period, we concentrated on five websites — The Age (Melbourne), The Australian, The Sydney Morning Herald, “ABC news online” and “ninemsn” — which form the basis of this report. Subsequently, a further eight print and media outlets have been added to the search list.
Articles were eligible for inclusion if they were relevant to the management and prevention of disease in Australia, and particularly if they were related to claims about new treatments, procedures and diagnostic tests. Interventions must have been the subject of clinical research (or a claim to that effect made).
Our rating instrument (http://www.mediadoctor.org.au/content/ratinginformation.jsp) (Box 1) is an extension of one previously used to assess the quality of medical news reporting in the US.7 We added criteria covering novelty of the treatment, availability, treatment options, presence of elements of “disease mongering”,13 and the reporting of evidence (study methodology). These criteria are consistent with those used in other media surveys and with advice provided by the Australian Press Council.8,15,16
Two reviewers assess each article. Reviewer pairs are flexible and change according to availability. Consensus scores are agreed on by the reviewers, who write a short commentary based on the criteria listed in the rating instrument. Most reviewers (7/10) were involved in piloting the original assessment tool and review process, and new reviewers receive training and supervision of their initial reviews. All reviews go through one of two “gatekeepers” before they are posted on the website. This allows reviews to be checked and edited to ensure consistency and quality.
We attempt to locate any relevant media releases, journal articles or other supporting literature that might assist the reviewers in making their assessments. Reviewers try to determine the extent to which the article relies on a media release from a company or organisation that has a vested interest in the publication of the article. It has proved difficult to locate media releases. Of the 104 news articles described in this paper, only six relevant media releases relating to six articles were located.
For each news article, the 10 criteria are scored as “satisfactory”, “not satisfactory” or “not applicable”. Total scores are posted for articles that have two or fewer “not applicable” ratings, and are expressed as proportions. For the purposes of display, the total scores are translated into a star rating (0–25% = 0 stars; 26%–50% = 1 star; 51%–75% = 2 stars; 76%–100% = 3 stars).
Star ratings and commentaries are published on the website (www.mediadoctor.org.au), which also presents cumulative scores for major media outlets.
Reviews are posted on the media doctor website as soon as possible after the news article is published. The process of locating and reviewing articles usually takes between 1 and 7 days after the article is published. Most reviews are posted within 2 weeks of their publication, and many within 1 week.
The website invites readers to add their observations and to comment on the site as a whole. Links to reliable sources of information (eg, Cochrane Library, Informed Health Online, Bandolier) are provided for those wanting more information on treatments featured in the news articles.
Pilot testing of the rating instrument (nine articles and seven reviewers) produced criterion κ scores between 0.49 and 0.74 (a κ score of 0.6 to 0.7 or greater is generally taken to denote a high level of agreement). Cumulative total scores for the five media outlets were calculated. Inspection of the data showed they were normally distributed, and unweighted cumulative scores were compared between media outlets using analysis of variance. Criterion and total scores for online and print media were also compared. In the case of individual criteria, we calculated the differences in proportions of satisfactory scores with their 95% confidence intervals; P values were two sided and calculated by an exact method. All statistical calculations were made using Stats Direct Statistical software (version 2.3.5, StatsDirect Ltd, Sale, Cheshire, UK).
Between 1 February and 1 September 2004, we identified 199 potential articles, but only 104 of these were reviewed. Reasons for non-inclusion were: articles not satisfying the eligibility criteria; the same article appearing in different media outlets; a glut of articles on the same topic (up to four of the first published articles were then included); and no reviewers available to complete the review within an acceptable time period.
Total scores for proportions of items satisfactorily reported were low overall, with statistically significant differences among the five media outlets (F4,103 = 4.16, P = 0.004). This appears to be almost entirely due to the much lower scores for the 53 online articles, which averaged 40.1% satisfactory compared with 56.1% for the 51 print articles, a difference of 15.9 (95% CI, 8.3–23.6) percentage points (Box 2).
The proportions of individual criterion scores that rated as satisfactory ranged from 23% to 94% for print media, and 14% to 89% for online media (Box 3). Seven of the 10 criteria had an average score of less than 50% in the online media, with the lowest scores being for “quantification of benefits” and coverage of harms and costs of treatment. The item scores were higher for print than online media, with the greatest differences seen for the use of independent information sources, quantification of the benefits and the coverage of “harms of treatment”. These were also the items with the poorest scores, apart from “costs of treatment”, for which both media types scored poorly.
Twenty articles scored no stars (overall scores 25% or less) compared with 11 that scored three stars (scores 76% or above). Examples of high and low scoring articles are given in Box 4. These illustrate the fact that news articles can be informative even when brief: the average length of print articles reviewed was 370 words, and 325 for online articles.
The short period of observation and the modest number of articles rated means it was not possible to draw any conclusions about changes in overall scores over time.
These data show that the Australian media outlets we reviewed often do a poor job in conveying important information on new medical treatments to the public. While the print media’s overall mean score was around 55%, that of the online news services was substantially lower, rating just over 40% satisfactory. Analysis of criterion scores showed that the main areas of weakness were inadequate presentation of evidence on benefits and harms of the interventions, failure to mention the costs of treatments (when relevant) or to obtain independent expert commentary.
It is unclear why journalistic standards should be lower for online news. There is no shortage of space that would justify very brief reports. We suspect the sense of “immediacy” that goes with online news reporting may be compromising standards.
We have not presented an analysis of the by-lines of articles (ie, who they were written by) we reviewed in media doctor. Many articles were short and a number were reproduced from overseas outlets without any consideration of local factors (for instance whether a drug or diagnostic test was available in Australia or subsidised under the Pharmaceutical Benefits Scheme/Medical Benefits Scheme). We do not doubt the articles would have rated more highly if written by dedicated health journalists.
Our main findings are in line with those published elsewhere.7-9 For instance, Moynihan et al found that, of 207 news stories covering new drugs in the US, only 30%–47% adequately covered costs and harms.7 Our estimates for the Australian media are close to these values. Similar observations were made by Cassels et al in regard to the Canadian media.8 The main advantages of our work are that it is ongoing, the results are accessible (there are now more than 400 articles reviewed on the website) and a regularly updated series of reviews is available for journalists and the public to read. Our data provide a continuously updated measure of the performance of the main Australian media outlets.
The main weaknesses of our study are that (1) it covers only five major media outlets (which, because of limited resources, were all we were able to cover during the study period), so the results cannot be considered representative of all Australian media; and (2) we concentrate on the evidentiary aspect of news reporting. This has led to criticisms from journalists.
Initial press coverage of the activities of media doctor attracted critical feedback, which is summarised on the media doctor website (http://www.mediadoctor.org.au/content/feedback.jsp) and in a medical journal news article.17 Supportive feedback included “. . . your website is a great new tool, collecting current treatment information together in the same spot and offering another perspective on the contents”. Criticisms from journalists included statements that “They’re trying to turn the media into a medical journal” and “I do get a bit cynical about non-journalists critiquing journalism”. We accept that the skills involved in writing medical news articles and medical journal articles are different. However, we consulted with journalists in establishing media doctor, and there are three experienced ex-journalists/medical writers in the media doctor review team. We make suggestions about what we think should be included in an article about a new medical treatment. We are not trying to advise journalists about how to write the articles. Our concerns about the quality of health reporting seem to be in line with a warning to journalists from the Australian Press Council.15 The “cultural divide” between medical journalism and medical practice may not be that wide. The “detective skills” and capacity for critical analysis required of journalists and doctors are similar.
We recognise that providing simple feedback to media outlets is unlikely to have a major impact on the quality of medical news reporting. Accordingly, we intend to augment the service in a number of ways. In future we will provide an automatic “alert” to media outlets when one of their articles features on the site. We also intend to send an annual “report card” to editors and producers, which summarises the performance of their company and their competitors, and to extend our coverage to other media outlets.
It is important that the responsibility for poor medical news reporting is not borne solely by journalists and editors. In a number of the articles we reviewed, it was apparent that the high quality was due, in part, to the involvement of researchers in disseminating information to the press. We do not have direct evidence but it is likely that many poor articles could have been improved if investigators had taken the trouble to communicate effectively with journalists. A survey of media releases from medical journals showed that “press releases do not routinely highlight study limitations or the role of industry funding”.18 If medical journals do not highlight these issues it is difficult for journalists to do their job properly. In our view both editorial staff and authors should take responsibility for improving this situation.
1 Criteria for reviewing media articles
Media doctor assesses articles for the extent to which they inform readers according to the following 10 criteria:
Whether the treatment is genuinely new
The availability of the treatment in Australia
Whether alternative treatment options are mentioned
Whether there is evidence of “disease mongering”13
Whether there is objective evidence to support the treatment
How the benefits of the treatment are framed (in relative or absolute terms)14
Whether harms of the treatment are mentioned
Whether costs of the treatment are mentioned
Whether sources of information and any known conflicts of interests of informants are disclosed
Whether the article relies heavily on a media release for content
2 Analysis of mean scores for proportion of items satisfactorily reported, by media outlet and type (104 articles)
|
No. of articles |
Mean score |
SD |
||||||||||||
The Australian |
18 |
56.6% |
18.8 |
||||||||||||
The Sydney Morning Herald |
16 |
56.0% |
17.9 |
||||||||||||
The Age (Melbourne) |
17 |
55.6% |
24.1 |
||||||||||||
ninemsn |
21 |
41.3% |
20.3 |
||||||||||||
ABC online |
32 |
39.4% |
18.9 |
||||||||||||
|
|
|
|
||||||||||||
51 |
56.1% |
20.1 |
|||||||||||||
Online |
53 |
40.1% |
19.3 |
||||||||||||
Comparison of mean total scores for all five outlets: F4,103 = 4.16; P = 0.004. Difference between mean scores for print and online 15.9 (95% CI, 8.3–23.6) percentage points (F1,103 = 17.0; P < 0.0001). |
|||||||||||||||
3 Comparison of individual criterion scores for percentage satisfactory — online v print media
Percentage satisfactory* |
Online |
Print media |
Percentage points difference (95% CI) |
Exact P (two sided) |
|||||||||||
% (no.) |
% (no.) |
||||||||||||||
Disease mongering |
88.7% (53) |
94.1% (51) |
5.4 (− 6.3, 17.7) |
0.33 |
|||||||||||
Novelty of treatment |
75.0% (52) |
91.8% (49) |
16.8 (2.3, 31.5) |
0.02 |
|||||||||||
Reliance on press release |
74.2% (31) |
93.5% (31) |
19.4 (1.0, 38.2) |
0.05 |
|||||||||||
Availability of treatment |
43.4% (53) |
57.1% (49) |
13.7 (− 0.57, 32.2) |
0.17 |
|||||||||||
Treatment options |
34.0% (53) |
37.5% (48) |
3.5 (− 15.0, 22.1) |
0.68 |
|||||||||||
Objective evidence |
24.5% (53) |
43.1% (51) |
18.6 (0.36, 35.9) |
0.04 |
|||||||||||
Independent sources of information |
20.8% (53) |
43.1% (51) |
22.4 (4.5, 39.2) |
0.01 |
|||||||||||
Quantification of benefits of treatment |
18.4% (49) |
40.0% (50) |
21.6 (3.7, 38.5) |
0.02 |
|||||||||||
Costs of treatment |
15.2% (46) |
23.1% (39) |
7.9 (− 9.1%, 25.5) |
0.29 |
|||||||||||
Harms of treatment |
14.0% (50) |
44.0% (50) |
30.0 (12.5, 46.1) |
0.002 |
|||||||||||
* Percentage satisfactory is calculated as the number rated as satisfactory divided by the total for that criterion; the denominators vary as it was not always possible to rate each criterion with the information provided in an article; denominators are given in parenthesis. |
|||||||||||||||
4 Examples of articles covered on the media doctor website
"US researchers develop new breast cancer treatment": ABC Online, 25 Jul 2004. Rating: 1 of 9
This article was so brief it was difficult to tell exactly what treatment it was describing. A web search revealed a similar US article about a technique in which a probe, inserted into a breast tumour using magnetic resonance imaging, was heated, so destroying the tumour. The US article stressed the need for controlled trials of this technology and revealed that the doctor in the article worked for the company that manufactures the device. The ABC article lacked any useful information yet portrayed the technique as “hope on the horizon”.
"Managing cholesterol: a new approach": ninemsn, 28 Jul 2004. Rating: 2 of 10
This article concerned the launch of a new cholesterol-lowering drug, ezetimibe. The article very closely paralleled a press release written by a public relations company on behalf of the drug manufacturer. The article’s language was promotional and sources cited were limited to those provided by the PR company. There was no mention of research studies, adverse effects or costs of treatment. The drug’s trade name always appeared in upper case as though in an advertisement. The Pharmaceutical Benefits Scheme listing restrictions were not provided even though this information was in the press release.
"World's first skin cancer cream launched": ninemsn, 02 Sep 2004. Rating: 9 of 10
This was the best of several stories on an imiquimod-containing cream used to treat basal cell carcinomas. The article described the type of research used to test efficacy, quantified the benefits and provided accurate information on availability in Australia. It also provided links to detailed information about the drug and prevention of basal cell carcinomas. However, there was no information about the cost of treatment.
"Drug prevents AIDS spread": The Sydney Morning Herald, 11 Jul 2004. Rating: 9 of 10
This described a study showing that a single dose of a generic form of an antiretroviral agent, nevirapine, when taken in combination with another established drug, zidovudine, by pregnant women, reduced transmission of the AIDS virus to the fetus. The article described the form of research used and the size of the treatment benefit in an appropriate and accurate way. The importance of the low cost of the generic drug was discussed and independent comments from experts were provided. However, there was no mention of the adverse effects of treatment.
- David E Smith1
- Amanda J Wilson2
- David A Henry3
- on behalf of the media doctor study group*
- 1 South Lakes Medical Group, Dora Creek, NSW.
- 2 University of Newcastle, Newcastle, NSW.
Media doctor is a project of the Newcastle Institute of Public Health (the population health research arm of Hunter Medical Research Institute), which receives funding from NSW Health through its population research capacity building program.
None identified.
- 1. Chapman S, Lupton D. Freaks, moral tales and medical marvels: health and medical stories on Australian television. Media Information Australia 1994; 72: 94-103. Available at: http://tobacco.health.usyd.edu.au/site/supersite/contact/pdfs/MIAfreaks.pdf (accessed Jul 2005).
- 2. Entwistle V. Reporting research in medical journals and newspapers. BMJ 1995; 310: 920-923.
- 3. Phillips DP, Kanter EJ, Bednarczyk B, Tastad PL. Importance of the lay press in the transmission of medical knowledge to the scientific community. N Engl J Med 1991; 325: 1180-1183.
- 4. Johnson T. Shattuck Lecture: medicine and the media. N Engl J Med 1998; 339: 87-92.
- 5. Nelkin D. An uneasy relationship: the tensions between medicine and the media. Lancet 1996; 347: 1600-1603.
- 6. Lawton B, Rose S, McLeod D, Dowell A. Changes in use of hormone replacement therapy after the report from the Women’s Health Initiative: cross sectional survey of users. BMJ 2003; 327: 845-846.
- 7. Moynihan R, Bero L, Ross-Degnan D, et al. Coverage by the news media of the benefits and risks of medications. N Engl J Med 2000; 342: 1645-1650.
- 8. Cassels A, Hughes MA, Cole C, et al. Drugs in the news: an analysis of Canadian newspaper coverage of new prescription drugs. CMAJ 2003; 168: 1133-1137.
- 9. Schwartz LM, Woloshin S. The media matter: a call for straightforward medical reporting. Ann Intern Med 2004; 140: 226-228.
- 10. Larsson A, Oxman AD, Carling C, Herrin J. Medical messages in the media — barriers and solutions to improving medical journalism. Health Expect 2003; 6: 323-331.
- 11. Jamtvedt G, Young JM, Kristoffersen DT, et al. Audit and feedback: effects on professional practice and health care outcomes. Cochrane Database Syst Rev 2003; (3): CD000259.
- 12. National Health Service. National Electronic Library for Health. Hitting the headlines. Available at: http://www.nelh.nhs.uk/ (accessed Oct 2004).
- 13. Moynihan R, Health I, Henry D. Selling sickness: the pharmaceutical industry and disease-mongering. BMJ 2002; 324: 886-891.
- 14. Moxey A, O’Connell D, McGettigan P, Henry D. Describing treatment effects to patients: how they are expressed makes a difference. J Gen Intern Med 2003; 18: 948-959.
- 15. Australian Press Council. General press release No. 245 (April 2001). Reporting guidelines. Available at: http://www.presscouncil.org.au/pcsite/activities/guides/gpr245.html (accessed Oct 2004).
- 16. Schwitzer G. Improving health and medical journalism. Website available at: http://www.tc.umn.edu/~schwitz/ (accessed Oct 2004).
- 17. Sweet M. Website gives media the tough treatment. BMJ 2004; 329: 178.
- 18. Woloshin S, Schwartz LM. Press releases: translating research into news. JAMA 2002; 287: 2856-2858.
Abstract
Objective: To analyse the reviews of medical news articles posted on media doctor, a medical news-story monitoring website.
Design and setting: A descriptive summary of operating the media doctor website between 1 February and 1 September 2004.
Main outcome measures: Consensus scores for 10 assessment criteria for the medical intervention described in the article (novelty, availability in Australia, alternative treatment options given, evidence of “disease mongering”, objective supportive evidence given, quantification of benefits, coverage of harms, coverage of costs, independent sources of information, and excessive reliance on a press release); cumulative article rating scores for major media outlets.
Results: 104 news articles were featured on media doctor in the study period. Both online and print media scored poorly, although the print media were superior: mean total scores 56.1% satisfactory for print and 40.1% for online; percentage points difference 15.9 (95% CI, 8.3–23.6). The greatest differences were seen for the use of independent information sources, quantification of benefits and coverage of potential harms.
Conclusions: Australian lay news reporting of medical advances, particularly by the online news services, is poor. This might improve if journals and researchers became more active in communicating with the press and the public.