Over the past few decades, there has been a substantial fall in death rates from cardiovascular disease and stroke in Australia.1 In 2003, Access Economics reported that health research had “directly, indirectly or serendipitously” contributed to at least half this gain in health,2 and estimated that the economic returns from investment in cardiovascular research in Australia are nearly eight times the annual investment.2 The contribution of the National Heart Foundation of Australia (NHF) to Australian cardiovascular research funding over the past 40 years amounts to more than $170 million, equating to a value of $1.36 billion from greater longevity and wellness. Despite these gains, cardiovascular disease is still the leading cause of death in Australia, accounting for 38% of all deaths.1 Strategic investment in cardiovascular research remains an important priority.
The need for accountability for research expenditure was highlighted in the Wills Review.3 The recent report of the Investment Review of Health and Medical Research4 also addressed this issue, and highlighted recent studies addressing both research and economic outcomes of Australian health and medical research. An editorial in the Medical Journal of Australia highlighted the lack of “overt accountability to society for its policy and research directions” by the National Health and Medical Research Council (NHMRC).5 Recently Kingwell et al, on behalf of the NHMRC’s Evaluation and Outcomes Working Committee, reported on a process that has now been developed for ongoing assessment of gains in knowledge, health and wealth from NHMRC-funded research.6
In this context, the NHF began an ongoing review of its research investment in 2002. This annual review was initiated both to evaluate the outcomes of its historical investment, and, even more importantly, to inform decisions about its future investment in research. We report the initial results of the Heart Foundation Research Evaluation Study.
In 2002, the NHF established the Research Evaluation Working Group (REWG) to oversee the NHF’s annual research evaluation. This group comprised leading cardiovascular researchers, representatives from NHF key program areas (cardiovascular health programs, income development, and marketing), and an external consultant.
In October 2002, the REWG met for the first time to examine funding data, reports on individual research projects, marketing data, and publications and policy documents. This meeting considered “baseline” data for each research program objective for the calendar years of 2001 and 2002.
Subsequently, the REWG met in March 2004 and March 2005 to consider data collected for the previous calendar year.
Visibility of NHF publications in international cardiovascular journals was measured by an Internet search of key phrases, using keywords that included “National Heart Foundation of Australia”, “NHF”, “National Health and Medical Research Council” and “NHMRC”. The number of publications in 2002–2004 acknowledging NHF funding support was compared with the number of publications acknowledging NHMRC support.
In 2004, the Australian National University’s Research Evaluation and Policy Project undertook a bibliometric study for the NHF. Its aim was to identify the journal publication output indexed by the Institute for Scientific Information’s (ISI) Science Citation Index attributable to NHF-funded research between 1996 and 2000, and to assist in determining the scientific impact of the research. The basic premise of bibliometric analysis is that the more often an article is cited, the higher is its scientific impact. The analysis focused on Australian publications in ISI’s Cardiac and Cardiovascular Systems journal set, as this enabled us to directly compare the impact of NHF publications with the results of a study on NHMRC-linked publications in the same journal set.7 The method used for the bibliometric analysis is described in detail elsewhere.7
The data presented in this article rest on a comparison of actual and expected citation rates. The actual citation rate refers to the average number of citations received by NHF or NHMRC publications. The expected citation rate refers to the average number of citations achieved by all publications similar to those being assessed. Expected citation values can be calculated for all ISI journals, with separate benchmarks for each year the journal appeared and for the type of publication (ie, research articles or reviews).
The analysis covers all articles published between 1996 and 2000, and the citations they received in this period. Again, this choice was made to enable direct comparison with NHMRC output.
The total amount of research funding allocated to NHF “people and projects” increased by 27% from 2001 to 2005 (Box 1). The increase in funding was partly achieved by co-funding fellowships and scholarships with other funding bodies or research organisations; the total value of this leveraged support was $1.5 million over the 5-year period (Box 1). The NHF’s Pharmaceutical Roundtable (a partnership of the NHF and nine pharmaceutical companies) was also an important source of new funding, providing a total of $2.2 million from 2001 to 2005 for support of fellowships and scholarships.
The balance of funding shifted from 26% for people and 74% for projects in 2001 to 46% for people and 54% for projects in 2005.
Most funding during 2002–2004 was allocated to biomedical research (Box 2). However, there was a 75% increase in the funding allocated to public health research during these years. This shift occurred primarily as a result of $0.4 million additional funding for fellowships and scholarships in the public health field. There was no significant change to the amount of funding allocated to clinical research.
For articles published in international cardiovascular journals during 2002–2004, there were about twice as many publications acknowledging NHMRC support, compared with those acknowledging NHF support (Box 3). However, when calculated relative to the funding allocated to cardiovascular research, the NHF had a much higher relative visibility in all years.
In the ISI cardiac and cardiovascular systems journals, Australian publications received on average 6.1 citations, above the world average citation level of 5.4 (Box 4). At 6.4, the NHF’s citation rate was higher than both the Australian and world benchmarks. NHF articles were published in the category’s high impact journals, although they received slightly fewer citations than expected for this subset of journals (Box 4). The NHF attracted average representation in the top 1%, and above average representation in the top 5% of the most highly cited Australian articles in these journals. No significant difference was noted between NHF and NHMRC publications. The NHMRC also performed strongly, publishing in journals with the same level of impact as the NHF, although receiving slightly more citations than expected for these journals.
The National Heart Foundation is Australia’s leading non-government funder of cardiovascular research and, together with its funding partners, currently allocates more than $7.5 million annually for this purpose. In 2002, the NHF adopted a people/projects policy, requiring a 50% allocation of research funding to people (through fellowships and scholarships) and 50% to projects (through grants-in-aid). The increased investment in research capacity was enabled by the support of nine corporate entities (under the banner of the Heart Foundation Pharmaceutical Roundtable) and the formation of funding partnerships with a number of other organisations (such as the NHMRC). The flow-on effect of this was an increase in overall NHF-associated funding for cardiovascular research by 27% from 2001 to 2005, and a more balanced funding portfolio across the fields of biomedical, clinical, and public health research.
Our results show that NHF-funded research has made a strong contribution to the generation of new knowledge for cardiovascular diseases and related disorders, as measured by publication impact. The research is published in high impact journals attracting citations at a rate above both Australian and world averages. NHMRC-funded research in cardiac and cardiovascular systems journals exhibits a similar performance, with a marginally higher citation rate. One of the considerations in interpreting these data is that NHF project funding occurs through grants-in-aid, which usually only provide a portion of the funding required to conduct any individual research project. It is important to acknowledge that funding from other sources (including the NHMRC) contributes to many of the publications acknowledging NHF support.
The publication output of NHF-funded research is supported by the visibility of NHF-funded articles in international peer-reviewed cardiovascular journals. Our analysis was limited to a subset of journals with a cardiovascular focus to specifically enable comparison with NHMRC-funded cardiovascular research. The confounding variable is the time lag between project investment and publication output. About double the number of articles acknowledged NHMRC support; however, NHMRC’s investment in cardiovascular research over this period was five to six times greater than that of the NHF, suggesting that the knowledge output from the NHF’s investment is greater in relative terms. We attribute this to the specific objective of the NHF to support high quality research on the causes, treatment, prevention and diagnosis of cardiovascular diseases and related disorders, whereas NHMRC’s goal is to “create new knowledge to improve the health of Australians” without a specific and strategic focus on cardiovascular disease.
In the future, for a more holistic evaluation of the impact of NHF-funded research, it will be important to specifically compare the knowledge and health payback from adopting the people/projects policy. Wooding et al have proposed a multi-level quantitative and qualitative payback model to assess returns from research.8 Several organisations have adopted this model or derivatives of it to assess returns. The NHF should consider adopting a similar framework, including specific career indicators such as career path beyond the tenure of the fellowship or scholarship and success in competitive grant funding.
A key outcome of our study was the effect of leveraging research funding from other organisations. Internationally, several government and non-government research funding bodies (Heart and Stroke Foundation of Canada, Health Research Council of New Zealand, NHMRC) have successfully adopted a partnership approach to funding research through a range of strategic research, international collaborations, project or fellowship initiatives. Building research capacity through funding more fellowships and scholarships by working in partnership with other organisations with a stake in Australia’s cardiovascular health is clearly an essential element of the NHF achieving its research objectives. Another essential element is combining resources with other organisations on questions of strategic importance. The increased support of public health research demonstrated in this study will also be vital to achieve this. Strategic questions may relate to a specific health issue or need (eg, closing the evidence–treatment gap), the prevalence of specific risk factors (eg, the increasing incidence of overweight/obesity in Australia1), or the health of “at risk” populations (eg, the significantly higher rate of cardiovascular disease in Aboriginal and Torres Strait Islander peoples1).
It has become critical for research funding agencies to evaluate the outcomes of health and medical research. Research performance assessment has gained international attention with high profile frameworks being implemented in the United Kingdom and New Zealand. In Australia, a research quality framework has been proposed which will take a holistic approach to evaluating the outcomes of publicly funded research through addressing research quality and a broad variety of research impacts. Several non-government organisations have also commissioned performance measurement and benchmarking studies on the outcomes of research and research funding strategies (Bev Heim-Myers, Senior Manager Research Programs, The Heart and Stroke Foundation of Ontario, personal communication).8 In Australia, the NHMRC has established a performance measurement framework and has published the first results from this.6,7,9,10 These exercises are challenging, but their value in demonstrating return on investment and informing funding strategy to maximise returns is clear. By adopting an evaluation mechanism, the NHF has positioned itself to take a more strategic approach to investment in research. In the future, this can only lead to greater improvements in cardiovascular health in Australia.
1 People and projects funding ($ million) by the National Heart Foundation of Australia
2 Research funding ($ million) by the National Heart Foundation of Australia allocated by discipline
3 Research funded by the National Heart Foundation of Australia (NHF) or the National Health and Medical Research Council (NHMRC) and published in international cardiovascular journals
Articles / $ million expenditure on cardiovascular research |
|||||||||||||||
The international cardiovascular journals were the European Heart Journal (http://www.escardiocontent.org/); the American Heart Association journals (Circulation; Circulation Research; Hypertension; Stroke; and Arteriosclerosis, Thrombosis, and Vascular Biology) (http://www.ahajournals.org/); the Journal of the American College of Cardiology (http://www.cardiosource.com/); American Journal of Cardiology (http://www.ajconline.org/); and the American Heart Journal (http://www.mosby.com/ahj). Articles not including NHF- or NHMRC-funded researchers in the author list were excluded. Any publications arising from the LIPID Study were excluded. Data are based on research expenditure of $6.3 million, $6.5 million and $6.7 million for the NHF in 2002, 2003 and 2004, respectively, and $31.0 million, $36.7 million and $42.4 million for the National Health and Medical Research Council (Mick Hoare, Director, Centre for Research Management and Policy, personal communication) in 2002, 2003 and 2004, respectively. |
|||||||||||||||
4 Citation rates for all “cardiac and cardiovascular systems” publications
Received 17 January 2006, accepted 31 May 2006
- Moira A Clay1
- Claire Donovan2
- Linda Butler2
- Brian F Oldenburg3,4
- 1 Murdoch Children’s Research Institute, Royal Children’s Hospital, Melbourne, VIC.
- 2 Research Evaluation and Policy Project, Research School of Social Sciences, Australian National University, Canberra, ACT.
- 3 School of Public Health, Queensland University of Technology, Brisbane, QLD.
- 4 National Heart Foundation Research Evaluation Working Group, Brisbane, QLD.
Some data in this article are derived from the Australian National Citation Report prepared by the Institute for Scientific Information Inc, Philadelphia, Pennsylvania, USA. Copyright ISI 2000. All rights reserved. The contribution of National Heart Foundation staff and honorary committee and working group members to this study is gratefully acknowledged.
None identified.
- 1. Australian Institute of Health and Welfare. Heart, stroke and vascular diseases — Australian facts 2004. Cardiovascular Disease Series No. 22. Canberra: AIHW and National Heart Foundation, 2004. (AIHW Catalogue No. CVD 27.)
- 2. Access Economics. Exceptional returns. The value of investing in health R&D in Australia. Canberra: Australian Society for Medical Research, 2003. http://www.asmr.org.au/general/Except.pdf (accessed May 2006).
- 3. Wills P (Chairman). The virtuous cycle. Working together for health and medical research. Health and Medical Strategic Review. Canberra: Australian Government Department of Health and Ageing, 1998.
- 4. Grant J (Chairman). Sustaining the virtuous cycle for a healthy, competitive Australia. Investment Review of Health and Medical Research. Canberra: Australian Government Department of Health and Ageing, 2004.
- 5. Van Der Weyden MB. Modernising the National Health and Medical Research Council [editorial]. Med J Aust 2005; 183: 340-342. <MJA full text>
- 6. Kingwell BA, Anderson GP, Duckett SJ, et al. Evaluation of NHMRC funded research completed in 1992, 1997 and 2003: gains in knowledge, health and wealth. Med J Aust 2006; 184: 282-286. <MJA full text>
- 7. Butler L. NHMRC-supported research: the impact of journal publication output 1996–2000. Canberra: National Health and Medical Research Council, 2003. http://www7.health.gov.au/nhmrc/publications/_files/butler03.pdf (accessed Jul 2006).
- 8. Wooding S, Hanney S, Buxton M, et al. The returns from arthritis research. Leiden: RAND Europe, 2004. http://www.rand.org/pubs/monographs/2004/RAND_MG251.pdf (accessed Jul 2006).
- 9. Performance measurement report 2000–2003. A report on the performance of the National Health and Medical Research Council 2000–2003. Canberra: National Health and Medical Research Council, 2003. http://www.nhmrc.gov.au/publications/_files/pmf2003.pdf (accessed Jul 2006).
- 10. National Health and Medical Research Council Evaluations and Outcomes Committee. Ten of the best. NHMRC funded health and medical research successes. Canberra: NHMRC, 2005. http://www.nhmrc.gov.au/publications/_files/thebest.pdf (accessed Feb 2006).
Abstract
Objective: To evaluate the outcomes of the research investment of the National Heart Foundation of Australia (NHF).
Design and setting: The NHF Research Evaluation Working Group was established in 2002 to oversee evaluation of research funding and outcomes data collected over a 5-year period. The evaluation included a bibliometric analysis conducted by the Research Evaluation and Policy Project at the Australian National University.
Outcome measures: Level and leverage of research funding; funding levels across the disciplines of biomedical, clinical, and public health research; and visibility and knowledge impact of NHF-supported research in international cardiovascular journals.
Results: The NHF’s investment in research increased by 27% from 2001 to 2005. This increase resulted from leveraged support for fellowships and scholarships of $1.5 million over this period, and $2.2 million from the pharmaceutical industry. There was an increase in fellowship and scholarship funding from 26% in 2001 to 46% in 2005. There was a 75% increase in the funding allocated to public health research from 2002 to 2004. NHF-funded research publications were found in high impact journals at levels above Australian and world averages, but received fewer citations than expected based on citation rates for all similar articles.
Conclusions: The NHF has been successful in implementing a policy to allocate 50% of its research funding to people and 50% to projects. This strategy has led to an increase in funding support for public health research. NHF-funded research has performed very well in terms of knowledge impact. The NHF is now well placed to strategically fund relevant research in the future.