Metrics in the Research Excellence Framework

Thursday, 18 June, 2009

As the Research Excellence Framework (REF) begins to take shape it is worth keeping an eye on developments over at HEFCE. As has been widely reported, there is an intention to make much greater use of metrics. Whilst no explicit “formula” has been finalised, there has been much preparatory work on bibliometrics, that is, the use of article citation to measure quality. An interim report has been published on the use of bibliometrics, running some sample models over 46 Units of Assessment at 22 institutions (for both RAE entered staff and all staff). The models tested were: all publications based upon address, all publications based upon author and “nominated” publications. HEIs will clearly be concerned that the first 2 models will not necessarily include all published outputs of their staff; indeed they probably don’t know all the published outputs!! That only leaves the “selected” papers model (which is what RAE used) which possibly unduly impacts upon those staff that are prolific in terms of quality and quantity. It may also have the impact of researchers trying to manipulate citations through:

-self-citation
-colleague citation
-guest authorship
-deliberate wrong/contentious reporting


The list of misdemeanors could get quite long; some can be mitigated against and others can’t. The report only used journal articles and review papers; conference proceedings were not included (although they may be in the future) and this will no doubt irk scholarly societies that publish fully peer reviewed articles arising from conferences (but which haven’t been classified as journals).

The number of citations for each paper are then normalized with respect to field (using average number of citations worldwide by field), year and document type. The interim report discusses some of the problems with defined subject types and how subject normalisation is applied will be contentious, particularly for interdisciplinary areas.

A further problem is “differential lag.” It takes time for papers to start getting cited and this “lag” will vary depending upon the subject area, journal, article quality etc. And clearly there will be a minimum lag period, probably to be set at ~2 years.

The report has anonymised the results until its full publication in autumn 2009 at which time it will be interesting to see not only how each model performs, but the variations between REF and RAE at each institution and, naturally, what the impact upon funding would have been.

So what are the likely impacts of all this?? Well, if HEFCE goes for a nominated list model (say 6 best papers), then it is in an individual researcher’s best interest to maximise the number of citations using some of the methods above. If this can be achieved later in the REF cycle, then the normalisation by year should significantly boost the score (although the report notes the opposite effect). All of which means it is important to: (i) publish in highly cited/visible journals and (ii) be aware of the subject/journal average and target citations above this. Of course one knock-on effect will be a decrease in the number of submissions to non-citation listed journals.

Add comment

Fill out the form below to add your own comments