Tracking funded health intervention research

Expectations that publicly funded health research should be productive, in terms of both research publication outputs and contributions to better health outcomes, are becoming increasingly explicit.1,2 This has directed attention to methods for tracking research outputs, where scholarly publication metrics — impact factors and citations — are currently the dominant indices.2,3 Publication of research is expected to disseminate new knowledge and facilitate “real-world” policy and practice impacts.

While the recent emphasis on research productivity spans all types of research,1,4-6 intervention research is particularly relevant, as its findings are likely to be more directly applicable to health policy and practice.7-9 Intervention studies tend to be less prevalent in peer-reviewed journals than descriptive and epidemiological studies, and this has been partly attributed to the practical and scientific challenges of conducting intervention research.9-11

Few studies have empirically investigated the implementation and outputs of health intervention research. As part of a project on the impact of a sample of intervention research funded by the National Health and Medical Research Council (NHMRC), here, we examine the research outputs of these grants. Specifically, we report:

• the descriptive profile of NHMRC-funded intervention research in terms of topics, settings, funding terms, and stages of development of the interventions; and
• whether and how statistically significant intervention effects on primary outcome variables influenced research productivity.

Methods

Data were collected between 23 July 2012 and 10 December 2013 on studies funded by the NHMRC between 1 January 2003 and 31 December 2007. Studies were eligible if they fitted our definition of health intervention research, which was: “any form of trial or evaluation of a service, program or strategy aimed at disease, injury or mental illness prevention, health promotion or psychological intervention, conducted with general or special populations, or in clinical or institutional settings”. Clinical trials of potentially prescribable drugs, vaccines and diagnostic tests were excluded.

Eligibility was assessed by two coders who reviewed titles, application abstracts, end-of-project reports to the NHMRC and publications arising from the grant. The 5-year period was selected to allow enough time for completion of the research and publication of the findings, balanced against limiting recall bias about studies completed too long ago.

Descriptive profile

Basic information on sample grants was collated, including the duration of funding and the topic of the intervention. The studies were classified according to “stage of intervention development” based on definitions from a previously published guide, distinguishing controlled interventions (efficacy), those carried out in real-life conditions (effectiveness), those that were replication or adaptation studies in different settings, or dissemination studies.12 Additional information was gathered from online surveys of chief investigators. A full description of the data collection process and response rates is provided elsewhere.13

Bibliometric analysis

To collect consistent information, we reviewed all publications submitted by chief investigators, and conducted literature searches (in Web of Science and Google Scholar databases) for the years following the commencement of each completed grant. Publications were reviewed to check if they were related to the grant in question. Key search terms included chief investigators’ names, grant numbers, project titles, intervention descriptions and relevant health issues. In the case of grants for which publications of study results could not be found, we attempted to contact chief investigators, including previous non-responders.

Assessing published results of intervention research

Two assessors reviewed publications that reported results of interventions to identify whether there were any statistically significant changes to the primary outcomes proposed in the research application summary. Where there was any uncertainty, decisions about what constituted primary outcomes were checked by other authors in a panel process. We classified interventions as: (i) those that showed statistically significant effects on primary outcomes; (ii) those with “mixed” results (eg, significant changes for some but not all primary outcomes), or if unintended or secondary outcomes were emphasised; and (iii) those that found no statistically significant effects.

Ethics approval

This project had approval from the University of Sydney Human Research Ethics Committee (15003). All chief investigators were assured that projects would not be identified because of anticipated sensitivities about publication output, ineffective interventions or lack of real-world impact.

Results

Completion

Sixty-six (80%) of the 83 intervention studies we identified were completed at the time of data collection, and 13 were ongoing. The status of four was unknown, with no responses from chief investigators. Of the 13 that were ongoing, reasons stated for incompleteness included problems recruiting study participants, being part of larger international trials or being longitudinal studies with longer follow-up. The proportion that were incomplete or ongoing was highest for the eight studies that commenced in 2007, the most recently sampled year, and included three grants scheduled for completion in 2011 or 2012.

Description of funded intervention research projects

The mean duration of funding of the 66 completed projects was 3 years (range, 2–5 years). Interventions included treatment and management (30 studies), screening and early intervention (12 studies), and primary prevention (24 studies), implemented in clinical or community settings, with many dealing with aspects of chronic disease. Topics reflected a variety of health disciplines, including medicine, psychiatry, psychology, dietetics, dentistry, physiotherapy and nursing. In terms of stage of intervention development, most focused on intervention efficacy (28 studies) or effectiveness (27 studies); 10 were replications or adaptations of an intervention in a new setting or population group; and one tested dissemination of the intervention.

Intervention effects

We could not locate published results on primary outcomes for 12 of the completed studies. There were equal numbers of studies that produced statistically significant effects (including “mixed” results; 27 studies) and those that did not show significant effects (27 studies). An example of mixed results was a school intervention that prevented (or delayed) age-related increases in students’ alcohol consumption, but did not reduce the prevalence of students’ depressive symptoms, which had been nominated as the primary objective.

Publication outputs

Publications related to each completed grant were categorised according to whether they reported on intervention effects or on “other” descriptive topics, such as measurement, intervention feasibility, epidemiological questions, or commentaries. The mean number of published articles per grant was 3.3 (range, 0–13), with 2.0 reporting results. Many investigators reported that their publication process was ongoing; eight had not yet published any articles, and twelve had not published articles on intervention effects. Among grants with published results, those with and without significant intervention effects had similar numbers of “other” publications (mean, 1.3 per grant), although the latter had smaller numbers of publications reporting intervention results and of total publications (Box).

Discussion

Our study describes the publication outputs for intervention studies funded by the NHMRC from 2003 to 2007, inclusive, and provides a benchmark to inform expectations about the publication yield of such research. We found that publications covered many aspects of intervention development8,12,14 and were not restricted to intervention effects, although studies reporting no statistically significant intervention effects produced slightly fewer results-based publications.15

While the number of publications is not an indicator of relevance to health policy,6 publication volume remains a basic metric of academic productivity.2,16 Analysis of Australian health promotion intervention research has previously identified between one and seven publications per study,17 while another Australian study of primary care research reported a mean of 2.3 publications per grant (range, 0–7 publications).18 However, the contexts and funding sources for these two studies and our study vary, and there is no endorsed benchmark for assessing numbers of publications across different areas of research.

In relation to our estimates, we acknowledge that later assessment may be required to capture complete publication outputs, and that the impact of non-responders on estimates (whether they would be lower or higher) is unknown. Further, the output estimates in our study cannot be extrapolated to non-intervention research.

Our findings on the stage of intervention development are consistent with those of other reviews of intervention research.9,10 As research type is not routinely documented by the NHMRC, the proportion of available funding that is invested in intervention research is currently unknown. However, our methods indicated that intervention research accounted for a small proportion of NHMRC grants in this period, although the interventions studied related to national health priorities and major causes of mortality and morbidity across Australia.

While intervention research typically tests effectiveness, the statistical significance of study results is not an indicator of study value.2 Some studies reporting non-significant results generated findings with important implications for policy and practice — for example, that an intervention should be discontinued or modified. It was beyond the scope of our study to critically appraise the methods of each funded study, and thus assess whether studies had sufficient statistical power to detect the changes they hypothesised.

While it is often claimed that researchers are discouraged by the difficulty of publishing statistically non-significant findings, we found no evidence for this. However, the length of time to intervention study completion and the relatively small number of intervention study publications may constitute disincentives for researchers to embark on these kinds of studies, particularly as there are no established methods to demonstrate other forms of impact, such as measures of policy change and influence on practice. Reviews of research funding have called for an increase in intervention research and for strategies to help remove the barriers faced by intervention researchers (such as dedicated funding for intervention research, longer funding periods, support for pilot studies and separate review panels).19 Meanwhile, policy agencies have suggested similar remedies to redress their concerns about a lack of definitive evidence on effective interventions in many areas.20

This is the first independent study to document the publication outputs of a set of intervention studies funded through a major national funding body. Tracking research publication outputs is important as a mechanism to ensure accountability in expenditure of public funds and, potentially, as a basis for quality improvement of research funding systems. Ongoing investigations of this kind are needed to provide information on whether current research investment patterns match the need for evidence about health care interventions.