Evaluation of NHMRC funded research completed in 1992, 1997 and 2003: gains in knowledge, health and wealth

The National Health and Medical Research Council (NHMRC) is Australia’s leading agency responsible for the development and maintenance of public and individual health standards. It funds health and medical research through a highly competitive grant system, and in 2005 awarded grants amounting to $412 million.

The past decade has seen increased pressures for accountability in the public sector, including for research funding.1-4 Australian medical research is highly innovative and translated into medically useful treatments.5-8 However, funders of health and medical research expect quantitative evidence of research impact. Additional funding for the NHMRC will need to be supported by evidence of health and wealth (commercial) gains from the increased funding delivered following the Health and Medical Research Strategic (Wills) Review in 2000.9 In particular, the NHMRC will be required to report against Australia’s National Research Priorities10 and National Health Priorities11 (Box 1). Recognising this challenge, the NHMRC established an Evaluation and Outcomes Working Committee in 2003.

The NHMRC has previously commissioned bibliometric analyses of Australia’s health and medical research publication output.12-14 Published articles indicate knowledge gain, but are not a complete measure of value.15 The task of the Evaluations and Outcomes Working Committee is to evaluate health and wealth gains derived from knowledge gained through NHMRC funding. A key strategy was to enhance the content, compliance rate for completion, and evaluation of end-of-grant reports (Box 2). The new grant reports18 allow better monitoring of compliance, and provide an input to strategic evaluation. They are designed to provide summary statistics of grant outcomes. In addition, database management procedures were strengthened to enable easy retrieval of information. Another strategy was the retrospective follow-up of the outcomes of grants completed in 1992 and 1997 and awards primarily supporting people held between 1992 and 2002 (Box 2). We report the collective outcome of these evaluations.

One hundred and thirty-one reports of grants completed in 2003 (29% of 454 expected reports) were returned and reviewed by a panel of 18 experts.

For the surveys of grants completed in 1992 and 1997, the response rate was 44% of the contactable sample in 1992 (61/139), and 51% (131/259) in 1997.

Of the eligible population of 1897 recipients of people awards, 596 (31%) completed the survey.

The bibliometric analyses reported previously12-14 and the preliminary NHMRC grant outcome evaluations reported here indicate that NHMRC grants make a significant contribution to knowledge creation through scientific publication. Importantly, knowledge from a substantial number of grants over a 7–12-year time frame is translated to outcomes that affect Australia’s health and economic wellbeing. However, it is important to consider the potential for sample bias, given the relatively low response rates in our preliminary assessments. In the future, it will also be important to determine methods to quantify the economic benefit of changes in practice and policy (in terms, for example, of both changed health care costs and improved patient outcomes leading to human productivity). Likewise, it will be important to capture evidence of the economic effect arising from commercialisation and patenting activity.5

Comparison of results from the 1997 and 2003 analyses is of interest. However, there are two major confounding variables. First, the size of grants and the cost of research both increased from 1997 to 2003. Second, the time frame for assessment in the two sets was different: 6 months after grant completion for the 2003 sample, and 7 years after completion for the 1997 sample. However, it is likely that a 7-year retrospective analysis of the 2003 grants would reveal further publications, and that grant expenditure per publication would be largely similar to that for the 1997 analysis. The publication output for population health was higher in the 2003 compared with the 1997 analysis. This may be due to greater recent investment in this area, reflected by the increase in the average grant expenditure per publication in 2003 compared with 1997.

Although Australian medical research has been translated into medically useful treatments,5-7 most development of Australian medical research was not performed in Australia.8,9 Australian ideas and publications were over-represented in the US patent literature on patent applications filed by non-Australian entities.9 If Australia is to contribute significantly to international health advances, a larger fraction of innovation must be developed in Australia. The level of engagement with the commercialisation process in the 1997 analysis is a gratifying outcome given that this is often the first step in developing a product which may contribute to health gain. The local protection of intellectual property allows inventors to interact with Australian biotechnology or small innovative companies and/or larger multinationals in a way that should bring substantial economic returns. It will be important to track the future development and commercialisation of patents reported in the current cohort of grants.

There was an overwhelmingly positive assessment of NHMRC people awards in contributing to the development of knowledge and skills.19 In addition, the high reputation and competitive standard of these awards was considered very important in improving future career prospects. On the negative side, there was insecurity about career pathways and employment prospects which was perceived to inhibit research development. This probably relates to the highly competitive and fixed term nature of people awards, both of which reflect the limited funding available.

Our first attempt to evaluate outcomes beyond publications is also useful for refining the mechanism of data capture. Measures are currently being implemented to improve the rate of end-of-grant report return, which exceeded 80% for 2005. Although the retrospective analyses collected data from grants completed in 1992 and 1997, it was only possible to obtain an adequate sample size from grants completed 7 years before the survey (1997 sample). Such retrospective analysis is important, as not all manuscripts arising from a grant are accepted for publication at the time a final report is due. Similarly, not all higher degree students will have completed their candidature. The health and wealth outcomes of grants are also unlikely to have developed significantly 6 months after completion of funding. However, future potential is certainly identifiable. Linkage between funding and outcomes over longer periods may be better tracked in the reverse direction from measurable outcomes to their funding sources.17 However, this approach may not be appropriate, given the increased variety of funding contributions that underpin research.

We suggest that a 5–7-year reporting time frame allows a sufficient period for health and wealth outcomes to have appeared, while minimising loss to follow-up. The magnitude and effect of research translation is still difficult to quantify, given both problems with outcome data collection and the time lag between grant completion and research translation. The end-of-grant report and the retrospective survey instrument will be further refined on the basis of this initial experience.

Communication of outcomes of research funded by the NHMRC to the Australian community is an important aspect of medical research accountability. In the 2003 analysis, a panel of experts identified 10 grants for showcasing as case-studies in a booklet highlighting research outcomes to the community (“Ten of the Best”,20 Box 5). This type of community reporting will probably be expanded as the NHMRC moves to an outcome-based approach to research reporting. Strategic groupings, organised according to major disease, population and discipline groups, will be responsible for identifying the strengths and gaps and for recommending strategies to ensure that there is appropriate capacity (people, funds or facilities) to best meet future challenges.

As the evaluation process is expanded, it will be possible to relate outcomes back to specific funding schemes so that the most effective and efficient funding mechanisms can be identified.21 Outcome analyses can then be used to inform future funding strategies. As well as identifying Australian research strengths, outcome evaluation will highlight areas of weakness that require specific investment and capacity-building.

In conclusion, outcome analyses and attribution to the initial funding source presents a significant challenge to all research funding agencies. There is no single established method that provides valid assessment of the specific outcomes of a single grant. In the last triennium (2003–2005), the NHMRC has laid the foundation for a number of different evaluation methodologies. However, further significant work is required to refine databases and outcome analyses. The goal will be to establish the highest standards of accountability and community dialogue, and to refine current funding mechanisms to most efficiently deliver health and economic returns for Australia.