Pharmaceutical advertising is claimed to be "the most organised and comprehensive information system for updating physicians about the availability, safety, efficacy, hazards and techniques of using medicines".1 Advertising of pharmaceutical products in Australia is regulated through the Therapeutic Goods Act 1989 (Cwlth).2 In addition, the Australian Pharmaceutical Manufacturers' Association (APMA) produces a Code of Conduct, stipulating that all promotional claims should be current, accurate, balanced and not misleading.2 Adherence to this Code is a condition of APMA membership. Because there is no independent screening of advertisements, however, transgressions cannot be systematically identified. In any case, the harshest penalty is rescission of APMA membership.3
There has been a longstanding interest in Australia in the independent assessment of the quality of pharmaceutical advertisements. More than a decade ago, the Australasian Society of Clinical and Experimental Pharmacologists found that 31% of pharmaceutical advertisements were "misleading" or "unjustifiable".4 Another 22% were in "technical breach" of the Code as current at the time.4 Improvements were tardy, despite changes to the Code of Conduct.5,6 A further study examined acceptability, references and graphics for 127 unique advertisements appearing in four Australian medical publications during 1992.7 While acknowledging their assessments were subjective, the authors were optimistic that "unacceptable" claims appeared to have decreased over time.7
Internationally, aspects of content in pharmaceutical advertising pertinent to evidence-based decision-making have been studied.8,9 For example, 22 of 130 advertisements appearing in 38 issues of North American journals reported quantitative outcomes.8 Moreover, half reported relative risk reduction (RRR) exclusively in presenting data, rather than absolute risk reduction (ARR) or the number needed to treat (NNT) (see Box 1). Physicians are more likely to prescribe drugs when presented only with information about RRR.12-14 This is known as the "framing effect". Incomplete or partial presentation of quantitative data might manipulate prescribing behaviour. Hence, it has been argued that "For those who are likely to be influenced by data presentation, never, ever, accept information on the basis of relative risk alone".15 Neither these aspects of the quantification of outcomes nor the level of evidence underpinning pharmaceutical claims in Australian publications has been previously ascertained.
We selected six medical publications: Australian Doctor, Australian Family Physician, Current Therapeutics, the Medical Journal of Australia, the Medical Observer and Medicine Today. All advertisements promoting pharmaceutical products were manually retrieved from consecutive issues of each of these publications over three months (October to December 2000). We first recorded how many times each advertisement appeared in our sample (multiple appearances had to be identical in size, wording, layout, and graphics). We determined the number of appearances because the impact of multiple appearances of the same advertisement on readers would exceed that of a single appearance. For any quantitative information (whether benefit or harm), we noted how the outcome was expressed (RRR; ARR; NNT).
Next, we documented the claims made about outcomes. Outcomes could be benefits (eg, relief of pain) or harms (eg, frequency of upper-gastrointestinal complications with drug use). Each claim was classified into one of four categories:
A: Unambiguous clinical outcome: When compared with DRUG X, DRUG Y delivers faster symptom relief.
B: Vague clinical outcome: DRUG X is the new, effective 20 μg pill with a low incidence of discontinuation due to skin problems.
C: Emotive or immeasurable outcome: DRUG X – one of a kind or DRUG X – a source of healing power.
D: Non-clinical outcome (eg, drug plasma half-lives or biochemical markers): Using DRUG X resulted in a 30% increase in arterial luminal diameter in post-mortem dissections.
For each claim referring to clinical outcomes (A or B), we noted whether it was supported by any reference to evidence, namely a specific citation, footnote or text (including Product Information, if so referred). For any reference retrievable through Medline, we obtained the full article or abstract to determine the level of supporting evidence16 as follows:
Meta-analysis or systematic review.
Randomised controlled trial.
Other study such as a cohort study.
If there was only one reference for a claim, the level of evidence of that claim was that of its sole reference. Where a claim was supported by more than one reference, we obtained all retrievable references and assigned levels to each; that single highest level of evidence was then afforded the claim. Claims attributed to proprietary files, claims with references to conferences, presentations and claims supported by citations from journals not included in Medline were not pursued, reflecting the realities of busy general practice.
To strengthen methodological rigour, we first developed operational definitions and explicit criteria. Before commencing our audit, we assessed independent inter-rater reliability between T L and F C K in a pilot study of classification of claims in 36 advertisements from issues of four journals published in January 1999. Kappa (κ) for inter-rater reliability was calculated as 0.93 ("very good").17
From the 31 issues of the six publications, we counted 1000 advertisements: Australian Doctor, 498 in 11 issues; Medical Observer, 210 in six issues; Medicine Today, 150 in three issues; Current Therapeutics, 65 in three issues; Australian Family Physician, 56 in three issues; and Medical Journal of Australia, 21 in five issues.
Within this sample, we found 174 distinct advertisements for 116 pharmaceutical products. The number of appearances per distinct advertisement ranged from one to 32 (median, 3; mode, 1).
Of the 174 distinct advertisements, 13 (7.4%; 95% CI, 4.20%–12.71%) reported quantitative statistics to convey information about outcomes, significantly lower than in North America8 (χ2 = 56.1; df = 1; P < 0.001). Of these 13, 10 (77%) reported RRR without any additional information; one reported RRR but included additional information, making it possible to calculate ARR and NNT; and two reported original data without specifying RRR, ARR or NNT but permitting their calculation by the reader.18 No advertisement explicitly reported ARR or NNT.
In 1000 advertisement appearances, the total number of claims was 1504. There were 640 advertisement appearances that made one claim, 227 appearances that made two claims, 122 appearances that made three claims, and 11 appearances that made four claims.
The claims were classified as D (23%), C (20%), B (29%), and A (28%).
For the 418 A claims, 6 (1%) were unreferenced and 146 (35%) cited references not searchable on Medline (Box 2). These proportions for B claims were 13% and 40%, respectively (χ2 = 43.2; df = 1; P < 0.001). Of the 266 A and 205 B claims with references retrievable by Medline, 55% and 38%, respectively, were based on evidence generated in randomised trials (Box 2). The proportion of A claims supported by Level 1 evidence was significantly lower than that for B claims (χ2 = 13.2; df = 1; P < 0.001).
Ours is the first Australian audit of pharmaceutical advertising to focus on quantitative aspects of claims and underpinning evidence. As only 8% of claims quantified specific clinical outcomes, we wonder whether advertising agencies have kept pace with emerging expectations of quantitative information necessary for evidence-based decision-making.15 Within this small proportion of claims, RRR was the most commonly used statistic, suggesting a risk of framing effect. No advertisement explicitly provided absolute risk reduction (ARR) or number needed to treat (NNT).
We also found the quality of claims unsatisfactory, with only 28% of claims being unambiguous. Further, much of the underpinning evidence was either impossible for the typical clinician to retrieve, or, where retrievable, was of low value. While nearly half (46%) of clinical claims were supported by evidence from at least one randomised controlled trial or more (Level 1 or 2), a similar proportion (45%) could not be substantiated at all (unretrievable evidence). Interestingly, the percentage of unambiguous claims (Class A) that were supported by Level 1 evidence (meta-analyses or other reviews) was significantly lower than that of ambiguous claims (Class B).
Because of shortcomings in the communication of quantitative data about clinical outcomes, proponents of evidence-based medicine have advocated the use by journals of structured abstracts that are almost entirely numerical in content.19 Recently, inconsistent editorial practice in presenting quantitative data in clinical research has been highlighted.20 As pharmaceutical advertising is considered a substantial source of information for medical practitioners, efforts to improve the quality of advertisement content are needed.
Limitations of our method invite two improvements if similar audits are to be conducted again. There would be benefit in auditing a larger number of journals, and hence more advertisement appearances and claims. In our study, time and other resource constraints precluded this. Further, we did not examine the impact of claims on actual prescribing practice. Should another audit be contemplated, classification of claims could responsibly be divided among a large number of researchers, as training and clear protocols assure inter-rater reliability.
1: Definitions
Relative risk reduction (RRR) compares the effectiveness of a new drug in reducing the risk of an adverse outcome against the effectiveness of a standard drug (typically prescribed to the control group in a randomised controlled trial that ethically must receive current best treatment). As it is a proportion, it is "relative". Specifically, RRR is calculated by first deducting the rate of an adverse event in the control group receiving the standard drug from the rate of an adverse event in the intervention groups receiving the new drug, and then dividing the difference by the event rate in the control group.
Absolute risk reduction (ARR) conveys effectiveness, not as a proportion that is relative, but as an absolute value. ARR is the difference between the rate of an adverse event demonstrated in the intervention group receiving a new drug when deducted from the rate of an adverse event in the control group receiving the standard drug or current best treatment.
Number needed to treat (NNT) is defined as the number of patients needed to be prescribed a new drug to avoid one single additional adverse outcome when compared against the results obtained when prescribing the standard (control) drug. It is calculated as the reciprocal of the absolute risk reduction.10,11
Example: When Drug F is used for Disease X, mortality is 15 per 1000. Drug F is the best available standard treatment that ethically is used in all control groups of clinical trials evaluating other drugs. A new drug, Drug G, is discovered to have a lower mortality rate of 5 per 1000. By using Drug G instead of Drug F, mortality is reduced by two-thirds (very impressive). Specifically, the risk of mortality is reduced from 15 (control) to 5 (new) per 1000 (ie, by 10 per 1000). When 10 (the difference) is expressed as a proportion of 15 (the rate in the control group), the relative risk reduction is 66% (15 – 5 = 10, then divided by 15 = 66%).
Using the same data, the absolute reduction in risk of mortality can be calculated. Specifically, the rate in the control group (15 per 1000) is deducted from the rate in the new group (5 per 1000), resulting in a rate of 10 per 1000, or 1% (a lot less impressive). In other words, there is a 1% reduction in risk of mortality by using Drug G instead of Drug F. The number needed to treat to avoid a single additional death otherwise predictable at a rate of 15 per 1000 in the control group is the reciprocal of the ARR, or 100 (ie, 1 divided by 0.01) (also less impressive).
2: Level of evidence for Class A and B claims
Class A claims |
Class B claims |
||||||||||
Unreferenced claims |
6 |
(1%) |
58 |
(13%) |
|||||||
Claims with references not searchable on Medline |
146 |
(35%) |
174 |
(40%) |
|||||||
Claims supported by Level 1 evidence (meta-analyses) |
40 |
(10%) |
59 |
(14%) |
|||||||
Claims supported by Level 2 evidence (at least one randomised controlled trial) |
189 |
(45%) |
108 |
(25%) |
|||||||
Claims supported by Level 3 evidence (all other evidence) |
37 |
(9%) |
38 |
(9%) |
|||||||
Class A: Claim of unambiguous clinical outcome. Class B: Claim of vague clinical outcome. |
|||||||||||
Received 4 January 2002, accepted 2 April 2002
- Tim W Loke1
- Fong Chee Koh2
- Jeanette E Ward3
- 1 Faculty of Medicine, University of Sydney, Sydney, NSW.
- 2 Needs Assessment and Health Outcomes Unit, Central Sydney Area Health Service, Sydney, NSW.
We thank Neil Donnelly for statistical advice.
None identified.
- 1. Levy R. The role and value of pharmaceutical marketing. Arch Fam Med 1994; 3: 327-332.
- 2. Roughead EE. The Australian Pharmaceutical Manufacturers Association Code of Conduct: guiding the promotion of prescription medicines. Aust Prescr 1999; 22: 78-80.
- 3. Harvey K. Pharmaceutical promotion. Med J Aust 1990; 152: 57-59.
- 4. Moulds R, Bochner F, Wing L. Drug advertising [letter]. Med J Aust 1986; 145: 178-179.
- 5. Moulds R, Wing L, Shenfield G, Day R. Drug advertising [letter]. Med J Aust 1987; 147: 52.
- 6. Moulds R, Wing L. Drug advertising [letter]. Med J Aust 1989; 150: 410-411.
- 7. Carandang E, Moulds RF. Pharmaceutical advertisements in Australian medical publications — have they improved? Med J Aust 1994; 161: 671-672.
- 8. Lexchin J. How patient outcomes are reported in advertisements: review of Canadian medical journals. Can Fam Physician 1999; 45: 1213-1216.
- 9. Gutknecht DR. Evidence-based advertising? A survey of four major journals. J Am Board Fam Pract 2001; 14: 197-200.
- 10. Young J, Glasziou P, Ward JE. General practitioners' self ratings of skills in evidence based medicine: validation study. BMJ 2002; 324: 950-951.
- 11. Tutorials in evidence-based medicine. <http://www.med.ic.ac.uk/divisions/63/phcgp/ebmtutorials/ebmtutorials.htm>. Accessed 13 May 2002, no longer available.
- 12. Bobbio M, Dimichelis B, Giustello G. Completeness of reporting trial results: effect on physicians' willingness to prescribe. Lancet 1994; 343: 1209-1211.
- 13. Naylor D, Chen E, Strauss B. Measured enthusiasm: does the method of reporting trial results alter perceptions of therapeutic effectiveness? Ann Intern Med 1992; 117: 916-921.
- 14. Forrow L, Taylor W, Armold R. Absolutely relative: how research results are summarized can affect treatment decisions. Am J Med 1992; 92: 121-124.
- 15. Muir Gray JA. Evidence-based health care: How to make health policy and management decisions. 2nd ed. Edinburgh: Churchill Livingstone, 2001.
- 16. Levels of evidence and grades of recommendations. Oxford Centre for Evidence Based Medicine. <http://www.minervation.com/cebm2/docs/levels.html>. Accessed 1 August 2002.
- 17. Altman DG. Practical statistics for medical research. London: Chapman & Hall, 1992.
- 18. Cook RJ, Sackett DL. The number needed to treat: a clinically useful measure of treatment effect. BMJ 1995; 310: 452-454.
- 19. Sackett Dl, Haynes RB. Summarizing the effects of therapy: a new table and some more terms [letter]. ACP J Club 1997; 127: A15-A16.
- 20. McLaren H. Evidence-based medicine in editorials [rapid response]. <http://bmj.com/cgi/eletters/324/7343/929>. Accessed 3 May 2002.
Abstract
Objective: To determine the quality of claims in advertisements published in Australian medical publications, describe how benefits and harms are presented, and examine the level of underpinning evidence.
Design and setting: Audit of a consecutive three-month sample of advertisements appearing in six popular Australian medical publications.
Main outcome measures: Proportion of advertisements with quantitative information; proportion of claims conveying clinical outcomes; where retrievable, level of underpinning evidence.
Results: Of 1504 claims, 855 could be substantiated quantitatively. Of these, 45% were supported by compelling evidence (randomised controlled trials or better). Of 13 claims explicitly reporting quantitative outcomes, none provided the absolute risk reduction or the number needed to treat.
Conclusions: Our audit invites greater diligence by pharmaceutical companies in substantiating their claims and greater vigilance among clinicians when reading them.