Evaluation of the impact of National Breast Cancer Foundation-funded research

The role of charities in supporting health and medical research is substantial. In 2011, Australia’s total federal, state and non-government organisation spending on cancer research projects, programs and infrastructure was $300 million, of which $83 million was awarded by major national cancer research funding charities, including $9 million by the National Breast Cancer Foundation (NBCF).1 With growing competition for the charity dollar, it is becoming increasingly important for medical research charities to demonstrate the outcomes or benefits of the research that they fund and to identify research strategies that are most likely to produce future benefits.2-4

The NBCF was established in 1994, and by early 2012 had invested over $81 million in over 300 Australian-based research projects. The NBCF seeks to invest in breast cancer research that will have the greatest impact, aiming to reduce the burden of a breast cancer diagnosis for all those living with or at high risk of developing breast cancer, and their families. All funds are raised from and within the Australian community. The NBCF’s research investment strategy supports a commitment to funding across the research continuum from basic to applied research, and the translation of research into clinical settings.5,6 In early 2012, the NBCF’s board of directors sought an independent evaluation of its total investment in breast cancer research to identify and communicate the impacts of that research to the Australian community, and to use the findings as an evidence base to inform and guide the NBCF’s future funding strategies.

Traditional approaches to evaluating the benefits of health and medical research focus on impacts on knowledge production and the research system. However, research funders in Australia have increasingly sought to demonstrate the wider benefits of research, such as improved treatments and health gains.7,8 This has resulted in interest in developments such as the Payback Framework, which fuses traditional measures with broader categorisations of the benefits from health and medical research,9 and is internationally acknowledged as the most widely used comprehensive method for assessing returns from research.10-13

The NBCF commissioned the Health Economics Research Group (HERG) at Brunel University to conduct a Payback study to record the range of impacts across its research investment portfolio. It is the first study of its kind of a program of breast cancer research.

The Payback Framework9 was applied to the portfolio of NBCF-funded research from 1 January 1995 to 31 March 2012. Ethics approval was obtained from Brunel University’s Research Ethics Committee. Data were collected and analysed in 2012 from March onwards.

Data were collected to measure the impact of NBCF research in terms of:

• scientific peer-reviewed publications as the central means of sharing knowledge with the research community;

• dissemination of knowledge produced to academic and non-academic audiences;

• interaction with the potential end users and beneficiaries of research, which increases the scientific and broader impacts of research;13,14

• research training and career advancement;

• capacity building and critical mass to undertake effective research;

• translation of research into clinical practice, evident in changes to health service policy and decision making and best practice in diagnosis and treatment;

• development of drugs, prognostic tools or diagnostic technologies;

• actual health gain, which is often hard to show, but may be evidenced in changes in the behaviour or practice of health care staff, consumers or the public.

The response rate for the CI surveys was 63% (153/242). For numbers of not sent, not delivered and non-returned surveys, see Appendix 1 (pdf). The mean grant start date of non-returned surveys was in the range 2004–2005 and returned surveys reported on grants that started in the range 2006–2007. The mean individual value of grants related to returned surveys was $348 096, compared with $170 203 for those related to non-returned surveys. The grants covered by survey responses represented $53 million (66%) of NBCF expenditure.

In the survey data, the balance of funding mechanisms and of applied and basic research was representative of the NBCF’s investment portfolio, as was the focus according to Common Scientific Outline (CSO) classification, except for a slight underrepresentation of “Biology” and overrepresentation of “Cancer control, survivorship and outcomes research” (Box 1).

The mean end year of grants covered by the surveys was in the range 2008–2009, compared with 2007 for all NBCF grants awarded between 1995 and 2012. Of the grants evaluated, 111 (73%) ended from 2008 onwards, and 42 (27%) of all surveyed grants were still in progress when the survey was administered (Box 2).

Case study data provided illustrations of high impact for each Payback category (Box 3). Three case studies are outlined in more detail in Box 4.

This assessment was based on information collected from 153 of 242 completed CI surveys (63%), a comparatively good response rate on which to base valid conclusions.15 The surveys were broadly representative of the whole NBCF portfolio in terms of grant mechanisms, balance of applied and basic research, and areas of focus (determined by CSO coding). Non-returned surveys tended to relate to grants that were older and smaller than those covered in the completed surveys, being on average 2 years older and half the value.

The relatively short time frame that had elapsed since many of the surveyed grants had been awarded or completed, with a demonstrable bias of returned surveys towards grants awarded in recent years, may have limited the assessment of impact, as most grants were yet to realise their promise. This was especially the case for basic research, for which downstream impacts take longer to accrue.14

The study also relied on subjective reporting of impacts in CI survey responses and case study interviews. However, multiple data collection approaches allowed comparisons to be made with objective measures such as citation data on the scientific impact of publications, or the impact of research on policy or practice in the form of citations in clinical practice guidelines. Interestingly, triangulation of data sources showed that in surveys, if anything, CIs tended to underreport the impact of their research, as has been noted in previous studies.4,15

While taking these potential limitations into account, this study, nonetheless, found a wide range of impacts across the Payback categories from citations through to informing policy — consistent with the NBCF’s investment in breast cancer research across the whole spectrum of the disease. It found that a mixed portfolio of research areas and mechanisms favoured a broad range of impacts. Because the NBCF is a community-funded organisation, a focus on generating health benefits to those impacted by breast cancer is particularly important. The evaluation found that applied research was effective in producing changes in policy and in the behaviour and practice of health care professionals, consumers and the public; thus generating wider health and health sector benefits, such as improvements in service delivery. The study found that basic research was more closely linked to impacts relating to knowledge production, and to product development in terms of developing drugs, prognostic tools or diagnostic technologies. It also highlighted that the funding of shared resources such as biobanks and other infrastructure was highly valuable to the research system for both basic and applied research.

The 16 case studies validated the survey data and provided rich illustrations of the full range of impacts.

The study builds on previous published examples of Australian research funding evaluations7,8 by adopting a more systematic, multidimensional and comprehensive approach. This is the first Payback study of a program of breast cancer research. Except for the separately commissioned bibliometric analysis, there are no benchmarking data for comparing the impacts with those of other breast cancer research organisations; however, this study creates a baseline for the future. Research supported by the NBCF attracted funds from other sources, such as state-based Cancer Councils, Cancer Australia and the National Health and Medical Research Council, and direct attribution of impacts to particular funders is complex.

This study is particularly timely as there is increasing pressure on both government and non-government research funders to provide evidence of the wider impacts of research. There is a renewed focus in the university sector on assessing the wider impacts of research, with trials of a proposed new initiative, Excellence in Innovation for Australia (EIA), having recently been completed. Many proponents hope to see EIA sitting alongside the existing Excellence in Research for Australia framework for assessing academic research performance.

2 Distribution of the 153 grants covered by survey responses, by research type and end year of funding