NHMRC grant applications: a comparison of “track record” scores allocated by grant assessors with bibliometric analysis of publications

Australia’s National Health and Medical Research Council (NHMRC) provides research funding to individuals and groups through a variety of mechanisms. Until recently, the majority of researchers were funded by project grants, which provide support for individuals and small teams of researchers undertaking biomedical, clinical, public health or health services research. These grants are generally of 3 years’ duration, and typically 20%–25% of applications are successful in obtaining funding.

In assessing project grant applications, the NHMRC uses a system of anonymous peer review, with assessors’ scores providing a guide to committees in priority ranking of all applications, which effectively determines which applications are funded. One part of the assessment is the allocation of a “track record” score based on the research publication output of the project’s investigators during the preceding 6 years (Box 1).

In 2001, the NHMRC initiated a revised program grants scheme. The scheme aims to provide support for research teams to pursue broadly based collaborative activity, and grants are typically of 5 years’ duration. Inter alia, the teams are expected to contribute knowledge at a leading international level and tackle problems for which longer-term stable funding is essential. In 2001, 60% of the program grant assessment was based on the record of research achievement, with 35% of the total score relating to the applicants’ publications (Box 1).

The primary aim of our study was to examine the track record score given to applicants for project grants and to compare this with bibliometric analysis of the publications on which that assessment was based. A secondary aim was to compare the citation impact of publications from program grants with the impact of publications from other NHMRC grant schemes.

In analysing project grants, we anticipated strong correlation between track record scores and bibliometric measures, as other studies have shown strong correlation between peer assessment and bibliometric analysis, even when the assessment took into account factors beyond the body of published research.4,5 We expected that high track record scores would be primarily associated with grants with high publication and citation counts, but our results did not reflect this.

Studies such as those by Oppenheim4 and Aksnes and Taxt6 have shown much stronger correlations between bibliometric indicators and peer review rankings, with coefficients of 0.7 or better. Yet the rankings to which they were relating their measures were generally based on a much wider remit — the “quality” of the units of assessment — rather than the much more specific focus of the track record assessments we were using. As our bibliometric indicators were direct measures of the published criteria for track record scores, we expected the correlations in our study to be even stronger.

The considerable differences in results across panels may in part explain the poor level of correlation. For example, ISI citation index coverage of the publication output in the area of public health is relatively poor, and much of the output is found in other formats.7 Weaker correlations were therefore expected for this discipline — although not the complete absence of association that we found. On the other hand, the lack of correlation in the data for the microbiology panel was unexpected and counterintuitive. As journals in this discipline are comprehensively covered by ISI indexes, bibliometric data should have correlated strongly with the scores based on the selection criteria (Box 1). Differences in ISI coverage between different grant review panels does not provide the complete answer to the poor correlations. This result raises the possibility that assessors deviated from the scoring criteria in providing track record scores.

In contrast to the perplexing outcomes of our analysis of project grants, the results for program grants were in line with our expectations. Previous studies of NHMRC-supported research2,8 have shown that the block-funded institutes, and research fellows located in these institutes, have a citation impact well above that of other NHMRC funding schemes and other Australian research sectors. Thus, given the standing of researchers targeted by the program grants scheme and the substantial weight given to publications in the assessment criteria, we anticipated that successful applicants would have a very strong citation record. Our results confirmed this.

As the track record score is only a single component of a much larger peer review process for project grant applications, the identified lack of correlation between track record scores and bibliometric measures in project grant applications cannot be used to question the validity of the final outcomes of the application process. The assessors do appear to “get it right”: the 2003 bibliometric study of NHMRC-funded research found that research projects funded by the NHMRC performed at a much higher level than those undertaken without NHMRC support, and their performance was above world and Australian benchmarks.2

In a study of grant proposals to the US National Science Foundation, Abrams identified two possible reasons for similar low correlations.9 He suggested that “the ability to produce a highly-rated proposal inherently has little correlation with the ability to carry out and publish high quality research”. He also suggested that the limited time scientists can devote to evaluating proposals can introduce considerable uncertainty into the process.

We have also received anecdotal evidence that, rather than rating “significance”, “approach”, “feasibility” and “track record” independently, assessors may judge an application as a whole, decide whether it should be funded, and give scores to each element that they believe will make it successful. With the pressure on researchers’ time, and the increasing calls on them in peer review scenarios, it is not surprising that shortcuts may be taken, as Abrams suggested.

What our results appear to highlight is a lack of transparency in the process. While it would be unrealistic to expect perfect correlation, it is clear that there should have been a much closer relationship between our project grant data and the scores. In contrast, the approach adopted for program grants does lead to the expected close relationship.

Perhaps now is the time to develop a more automated system of track record assessment. Why ask peers to assess track records from scratch, when there are defensible surrogates for this aspect of the grant application? Surely their scarce time is best reserved for where it is most useful, and where no alternative is possible — assessing the significance, approach and feasibility of applications. They could be relieved of the burden of assessing track record, only delving into it in the relatively few cases in which there are concerns about the automatically generated scores. Concerns about the use of such measures, raised recently in an article by Lehmann et al,10 related not to the measures themselves, but to their potential “harmful misuse”. Bibliometrics has progressed significantly in recent years, and measures are now available that are sensitive to field-specific characteristics and the concerns of researchers who are at an early stage of their careers.

2 Comparison of mean track record scores with total publications for project grant applications in 2001*

* Each data point represents the total publications (ie, total count of individual publications published by all authors on each grant application during the previous 6 years) for each project grant application plotted against the mean track record score given to each application by reviewers. The correlation coefficient for the relationship between the two measures was 0.375.

3 Comparison of mean track record scores with total citations for project grant applications in 2001*

* Each data point represents the total citations to all the publications published by all authors on each grant application during the previous 6 years plotted against the mean track record score assigned to each application by reviewers. The correlation coefficient for the relationship between the two measures was 0.327.

5 Relationship between actual and expected citation rates for publications by successful program grant applicants in 2001 and 2002, compared with results for other NHMRC grant schemes reported in 20032*†

NHMRC = National Health and Medical Research Council. * The “actual” citation rate is the average number of citations received by the publications being analysed. The “expected” citation rate is the average number of citations achieved by publications similar to those being assessed. The diagonal on the graph indicates the point at which the two rates converge; the vertical line indicates the world citations-per-publication rate in the medical and health sciences. A point above the diagonal and to the right of the world average indicates a citation performance above the world average in journals with high expected rates of citation. † On the graph, “All fellows” indicates all publications from all fellowship grant recipients, including institute fellows; “Australia” indicates all Australian publications in the field; “Institute fellows” indicates all publications from NHMRC fellows in the former block-funded medical research institutes; “NHMRC” indicates all publications supported by NHMRC grants; “NPG” indicates new program grants (2001 and 2002 application rounds combined); “NPG 2001” indicates new program grants (2001 application round only); “NPG 2002” indicates new program grants (2002 application round only).