Having given an overview of the REF in my last post, in this one I will provide a commentary on it from a digital scholarship perspective.
As readers will probably know, I’m not a fan of such exercises in general – the inevitable experimenter’s effect phenomena comes into play, particularly when there is considerable money involved, with the result that we don’t encourage new, exploratory types of behaviour. But let’s put aside these more general reservations, and look at the REF proposal itself. My particular take on this is to what extent does it reward, recognise, encourage activities which we might broadly term digital scholarship?
From this perspective the REF starts well with the aim to “support and encourage innovative and curiosity-driven research, including new approaches, new fields and interdisciplinary work.” So there is a very explicit aim to ‘support and encourage’ new approaches, new fields, interdisciplinary work. I was rubbing my digital scholarship hands at this point.
Similarly their definition of research is sufficiently broad to encompass digital scholarship: “a process of investigation leading to new insights effectively shared”. That phrase ‘effectively shared’ surely harks at blogs, social networks, data visualisation, wikis, etc.
So feeling encouraged, I read on. Sadly the rest of the document does not live up to these opening intentions, indeed it seems to actively undermine them in places.
Making digital activity explicit
Given that part of the aim of the REF is to place UK research at the fore and to demonstrate both its excellence and relevance to society, I would suggest that encouraging new forms of scholarly activity would be an explicit aim. Particularly as this would tie in with the aim set out in the Digital Britain report about establishing the UK as an economic hub in the digital age. Yet anything resembling digital scholarship is noticeable by its absence in the document.
It is noticeable for instance that the two drivers for change to the RAE were to make it less cumbersome and to have a unified
approach. These seem rather unambitious – I would have liked to see
something like ‘to recognise the changing nature of research and
dissemination in a digital age and to both reward and encourage this’.
For example, it does make a gesture towards moving away from the traditional article as the sole output when it states
“All types of outputs from research that
meets the Frascati principles (involving original investigation leading
to new insights) will be eligible for submission. This includes ‘grey
literature’ and outputs that are not in conventional published form,
such as confidential reports to government or business, software,
designs, performances and artefacts”
But this would have been a prime opportunity to explicitly recognise new forms of output – blogs, video, podcasts, etc.
The wrong metrics
Having looked at the possible use of metrics to inform the panels they conclude that they are not robust enough, but that some citation metrics will be used. These however are limited to 3-4 pre-approved databases. This is not even a forward step for digital scholarship – by limiting it to these databases and then suggesting that staff are selected on this basis, they are effectively limiting outputs to journal articles.
Although they do argue that “This approach to the assessment of outputs
retains scope for the assessment of grey literature and work published
in non-standard forms (for which citation data are unlikely to be
available)” my suspicion is that the presence of metrics in the sciences will effectively become a selection filter.
There is no suggestion here of embracing the broader world of data and metrics which new forms of activity would be well represented by.
Despite it being mentioned in the high level aim of the REF, interdisciplinary research will not be well served. They are reducing the number of units of assessment, which will mean more researchers being forced into inappropriate categories. In addition they state that fluidity between the units should be reduced:
“For the REF we propose to have
substantially fewer UOAs with fewer fluid boundaries between them than
in previous assessment exercises”
The REF must be the only body that feels the way to embrace the digital age is to have less fluid boundaries.
As with the use of citation metrics restricting outputs to journal articles, they anticipate the criticism of interdisciplinary work:
“we aim to ensure that whichever panel
interdisciplinary research is submitted to, there will be effective
mechanisms for ensuring it is reviewed fairly by people with
But as with the outputs, the actual proposal seems to undermine this, and merely stating it as an objective will not ensure that it happens.
I think the inclusion of impact could be an area that digital scholars can excel at – we have data about blog readership, numbers of views and embeds on videos, podcast downloads, etc. In Appendix D they list some of the evidence of impact that will be admissible. These are disappointingly conservative however – eg Staff movement between academia and industry, Research contracts and income from industry, Research income from government organisations, etc.
Openness doesn’t get a mention in the REF, and you would have liked to see this as a key theme. Indeed there is an element of a closed world of researchers that pervades much of the document, despite the talk of impact and relevance. There is no mention for instance of recognising data as an output that should be released. The use of citations is limited to pre-defined ones that are purchased. There is no encouragement to publish openly, or the use of open APIs to explore different metrics.
Overall I found it a highly frustrating, deeply conservative and mildly schizophrenic document.
I wonder if part of this schizophrenia arises from not wanting to be seen to be directing research too much, or a denial that such exercises do exactly that. I think they should acknowledge the experimenter effect (it is ironic that a research exercise ignores it) and embrace it. If we are to have exercises such as the REF, which effectively control the direction of UK research, then would it not be better for them to have some worthy goals and vision that would see research be competitive, highly regarded and relevant?
So an opportunity to have a vision that encourages new forms of scholarship, to embrace the potential of digital technologies, to make openness a central theme in UK research is missed. What we have is a curious mix of strategy document and justification of the difficulties of the process. Having decided to measure something they now come up against all the difficulties that entails, such as imposing categories, regulating workload, etc. The process becomes the artefact and thus instead of promoting interdisciplinary work, fluid boundaries, new forms of output which a strategic proposal would surely do, all of these are curtailed because that makes the process more manageable.
As a document it seems to consistently undermine its own ambition, and whenever it approaches new forms of scholarship it veers away at the last moment and reverts to the comfort of what it knows best.