The role of charities in supporting health and medical research is substantial. In 2011, Australia’s total federal, state and non-government organisation spending on cancer research projects, programs and infrastructure was $300 million, of which $83 million was awarded by major national cancer research funding charities, including $9 million by the National Breast Cancer Foundation (NBCF).1 With growing competition for the charity dollar, it is becoming increasingly important for medical research charities to demonstrate the outcomes or benefits of the research that they fund and to identify research strategies that are most likely to produce future benefits.2,3,4
The NBCF was established in 1994, and by early 2012 had invested over $81 million in over 300 Australian-based research projects. The NBCF seeks to invest in breast cancer research that will have the greatest impact, aiming to reduce the burden of a breast cancer diagnosis for all those living with or at high risk of developing breast cancer, and their families. All funds are raised from and within the Australian community. The NBCF’s research investment strategy supports a commitment to funding across the research continuum from basic to applied research, and the translation of research into clinical settings.5,6 In early 2012, the NBCF’s board of directors sought an independent evaluation of its total investment in breast cancer research to identify and communicate the impacts of that research to the Australian community, and to use the findings as an evidence base to inform and guide the NBCF’s future funding strategies.
Traditional approaches to evaluating the benefits of health and medical research focus on impacts on knowledge production and the research system. However, research funders in Australia have increasingly sought to demonstrate the wider benefits of research, such as improved treatments and health gains.7,8 This has resulted in interest in developments such as the Payback Framework, which fuses traditional measures with broader categorisations of the benefits from health and medical research,9 and is internationally acknowledged as the most widely used comprehensive method for assessing returns from research.10,11,12,13
The NBCF commissioned the Health Economics Research Group (HERG) at Brunel University to conduct a Payback study to record the range of impacts across its research investment portfolio. It is the first study of its kind of a program of breast cancer research.
The Payback Framework9 was applied to the portfolio of NBCF-funded research from 1 January 1995 to 31 March 2012. Ethics approval was obtained from Brunel University’s Research Ethics Committee. Data were collected and analysed in 2012 from March onwards.
Data were collected to measure the impact of NBCF research in terms of:
scientific peer-reviewed publications as the central means of sharing knowledge with the research community;
dissemination of knowledge produced to academic and non-academic audiences;
interaction with the potential end users and beneficiaries of research, which increases the scientific and broader impacts of research;13,14
research training and career advancement;
capacity building and critical mass to undertake effective research;
translation of research into clinical practice, evident in changes to health service policy and decision making and best practice in diagnosis and treatment;
development of drugs, prognostic tools or diagnostic technologies;
actual health gain, which is often hard to show, but may be evidenced in changes in the behaviour or practice of health care staff, consumers or the public.
Archival material that was made available comprised a database with descriptive information about all NBCF grants, a database of publications linked to NBCF grants, and copies of grant proposals, annual reports and end-of-grant reports for the case studies.
A questionnaire was structured around the multidimensional categorisation of impacts in the Payback Framework. It also investigated factors influencing the dissemination and utilisation of research. A total of 242 surveys were sent to the chief investigators (CIs) involved in current and previously awarded grants. This excluded grants that had not yet started or had recently commenced and CIs who were unavailable for personal reasons.
Sixteen case studies were conducted to provide validation of data collected from the survey and to supply richer information about the full extent of wider impacts of NBCF research. The HERG compiled a shortlist of potential case studies based on reported high impact in at least one of the Payback categories in the CI survey and high-impact project publications identified in Elsevier’s Scopus database. The case studies were selected using a purposive, stratified approach to ensure all Payback categories were represented across the range of grant types funded.
Each case study included document and archival review, citation analysis, searches for citations in clinical practice guidelines, and semi-structured telephone interviews with CIs. Each case study was described in a narrative account organised according to the stages of the Payback model, and sent to CIs for final validation. All CIs gave informed consent to participate in the study, and approved the narrative accounts.
Bibliometric analysis of the scientific impact of recent NBCF-linked publications was conducted. This used an existing database of publications from Elsevier’s Scopus citation index developed to analyse the impact of research conducted by the Victorian Comprehensive Cancer Centre (VCCC). Approval was obtained from Elsevier and VCCC for its use in this study. The publication window was the 5-year period 2006 to 2010. Two hundred and sixty-two NBCF publications were identified. The citation window was 1 January 2006 to 1 May 2011. Elsevier calculated world and Australian benchmarks for aggregate citation rates, and thresholds to identify highly cited journal articles.
The response rate for the CI surveys was 63% (153/242). For numbers of not sent, not delivered and non-returned surveys, see Appendix 1 (pdf). The mean grant start date of non-returned surveys was in the range 2004–2005 and returned surveys reported on grants that started in the range 2006–2007. The mean individual value of grants related to returned surveys was $348 096, compared with $170 203 for those related to non-returned surveys. The grants covered by survey responses represented $53 million (66%) of NBCF expenditure.
In the survey data, the balance of funding mechanisms and of applied and basic research was representative of the NBCF’s investment portfolio, as was the focus according to Common Scientific Outline (CSO) classification, except for a slight underrepresentation of “Biology” and overrepresentation of “Cancer control, survivorship and outcomes research” (Box 1).
Box 1
All National Breast Cancer Foundation (NBCF) grants and grants reported in survey responses, by mode of funding and research type, 1995–2012, and by Common Scientific Outline (CSO) classification, 2000 onwards*
The mean end year of grants covered by the surveys was in the range 2008–2009, compared with 2007 for all NBCF grants awarded between 1995 and 2012. Of the grants evaluated, 111 (73%) ended from 2008 onwards, and 42 (27%) of all surveyed grants were still in progress when the survey was administered (Box 2).
Box 2
Distribution of the 153 grants covered by survey responses, by research type and end year of funding
Case study data provided illustrations of high impact for each Payback category (Box 3). Three case studies are outlined in more detail in Box 4.
Box 3
Description of case studies and high impact Payback categories
Box 4
Case studies
The bibliometric study found that between 2006 and 2010, the relative citation impact of the 262 NBCF-linked publications was 2.03, double the world benchmark (1.0) and exceeding the Australian average (1.40). The journals in which NBCF-linked publications were cited had an impact of double the world average. The analysis also found almost double the expected number of NBCF publications among the most cited articles (12 in the top 5%).
NBCF grant holders had made considerable efforts to disseminate the knowledge produced to academic and non-academic audiences (Appendix 2 (pdf)). Case studies showed that the findings of NBCF research were broadcast in Australia and internationally through a wide variety of platforms, thereby contributing to the pool of global scientific knowledge, supplying an evidence base for consumer groups, policymakers and clinical practice, and also raising community awareness of breast cancer and related concerns.
Data on interaction with research users can be found in Appendix 3 (pdf).
As a direct consequence of the NBCF grant, 185 higher degrees were gained or expected, including 121 PhDs, and almost half of CI surveys (70 [46%]) reported career progression for at least one team member, such as promotion or gaining tenure.
The surveyed CIs reported that 101 grants (66%) generated tools for future research that would help to build capacity across the research system, such as new or significantly improved animal models, cell lines, physiological or biochemical markers, assays, models of disease, or validated patient questionnaires.
The survey found that 15 applied grants had impacted on policy and 32 expected to do so, compared with one basic research grant impacting on policy and 13 expecting to do so. Data on actual and expected wider impacts can be found in Appendix 4 (pdf).
Ten basic grants and four applied grants had led to product development of drugs, prognostic tools or diagnostic technologies. Three applied research projects had informed a broader spectrum of product development; for example, development of a decision aid, a psychological treatment protocol and a sports bra.
This assessment was based on information collected from 153 of 242 completed CI surveys (63%), a comparatively good response rate on which to base valid conclusions.15 The surveys were broadly representative of the whole NBCF portfolio in terms of grant mechanisms, balance of applied and basic research, and areas of focus (determined by CSO coding). Non-returned surveys tended to relate to grants that were older and smaller than those covered in the completed surveys, being on average 2 years older and half the value.
The relatively short time frame that had elapsed since many of the surveyed grants had been awarded or completed, with a demonstrable bias of returned surveys towards grants awarded in recent years, may have limited the assessment of impact, as most grants were yet to realise their promise. This was especially the case for basic research, for which downstream impacts take longer to accrue.14
The study also relied on subjective reporting of impacts in CI survey responses and case study interviews. However, multiple data collection approaches allowed comparisons to be made with objective measures such as citation data on the scientific impact of publications, or the impact of research on policy or practice in the form of citations in clinical practice guidelines. Interestingly, triangulation of data sources showed that in surveys, if anything, CIs tended to underreport the impact of their research, as has been noted in previous studies.4,15
While taking these potential limitations into account, this study, nonetheless, found a wide range of impacts across the Payback categories from citations through to informing policy — consistent with the NBCF’s investment in breast cancer research across the whole spectrum of the disease. It found that a mixed portfolio of research areas and mechanisms favoured a broad range of impacts. Because the NBCF is a community-funded organisation, a focus on generating health benefits to those impacted by breast cancer is particularly important. The evaluation found that applied research was effective in producing changes in policy and in the behaviour and practice of health care professionals, consumers and the public; thus generating wider health and health sector benefits, such as improvements in service delivery. The study found that basic research was more closely linked to impacts relating to knowledge production, and to product development in terms of developing drugs, prognostic tools or diagnostic technologies. It also highlighted that the funding of shared resources such as biobanks and other infrastructure was highly valuable to the research system for both basic and applied research.
The 16 case studies validated the survey data and provided rich illustrations of the full range of impacts.
The study builds on previous published examples of Australian research funding evaluations7,8 by adopting a more systematic, multidimensional and comprehensive approach. This is the first Payback study of a program of breast cancer research. Except for the separately commissioned bibliometric analysis, there are no benchmarking data for comparing the impacts with those of other breast cancer research organisations; however, this study creates a baseline for the future. Research supported by the NBCF attracted funds from other sources, such as state-based Cancer Councils, Cancer Australia and the National Health and Medical Research Council, and direct attribution of impacts to particular funders is complex.
This study is particularly timely as there is increasing pressure on both government and non-government research funders to provide evidence of the wider impacts of research. There is a renewed focus in the university sector on assessing the wider impacts of research, with trials of a proposed new initiative, Excellence in Innovation for Australia (EIA), having recently been completed. Many proponents hope to see EIA sitting alongside the existing Excellence in Research for Australia framework for assessing academic research performance.
Received 14 June 2013, accepted 24 October 2013
Abstract
Objective: To evaluate the impact of the National Breast Cancer Foundation’s (NBCF’s) research investment.
Design and participants: Surveys based on the Payback Framework were sent to chief investigators involved in research funded by the NBCF during 1995–2012; a bibliometric analysis of NBCF-funded publications in 2006–2010 was conducted; and a purposive, stratified sample of case studies was obtained.
Main outcome measures: Research impact on knowledge production, the research system, informing policy, product development and broader health and economic benefits.
Results: Of 242 surveys sent, 153 (63%) were returned. The average impact of journals in which NBCF publications appeared was double that of world publications. Seventy surveys (46%) reported career progression, and 185 higher degrees were obtained or expected, including 121 PhDs. One hundred and one grants (66%) produced tools that built capacity across the research system, and research teams leveraged an additional $1.40 in funding for every dollar invested. Fifteen applied grants and one basic grant impacted on policy. Ten basic and four applied grants led to the development of drugs, prognostic tools or diagnostic technologies. Twenty applied and two basic grants led to changes in practice and behaviour of health care staff, consumers and the public, with further impacts anticipated. Case studies provided illustrations of high impact.
Conclusions: NBCF’s strategy of investing in a mixed portfolio of research areas and mechanisms encouraged a broad range of impacts across all Payback categories. The impacts from basic research tended to focus on knowledge production and drug development; while applied research generated greater impacts within the other Payback categories. The funding of shared infrastructure stimulated impact across the research system.