I've been thinking a lot about recognition of digital output in formal systems, in short, can you get promotion for doing all this online stuff? One of the great things about being online is that as soon as you start to openly think these things through, people start to point you at stuff (mainly Tony). It seems a lot of people have been doing the same (and getting further than me).
For example Eric Schnell says that
promotion and tenure guidelines (all the way) back in 2007 with redefined criteria in the form of alternative recognition measures. Their documents identify nine alternatives to the standard 'article in a peer-review journal' model
He then lists the nine alternatives: Invited/edited publications; Live conferences, Citations; Download/visitor stats; Impact in online discussions; Impact in the real world; Net-native recognition metrics; and Reference letters.
I think the inclusion of download/visitor stats and net-native recognition metrics is significant here. Similarly, Geoffrey Rockwell has a wiki for a project on evaluating digital work in the humanities. The types of digital output they list as relevant are:
Online peer reviewed publication; Scholarly Electronic Editions; Specifications; Research Tools; Hypermedia/Hyperfiction; Instructional Technology/CALL; and Research blogs.
They also provide a list of questions for those evaluating such work (I'm not sure I agree with the list, it seems focused on a specific product, rather than, say overall significance to a community).
Gerry McKiernan blogs about a study which examined 39 different impact factors for science. The authors of the study (Bollen J, Van de Sompel H,
Hagberg A, Chute R, 2009 A Principal Component Analysis of 39
Scientific Impact Measures. PLoS ONE 4(6)) state:
impact in the digital era, a variety of new impact measures has been
proposed on the basis of social network analysis and usage log data.
Here we investigate how these new measures relate to each other, and
how accurately and completely they express scientific impact.
After reviewing a number of impact factors, including social network measures, they conclude that:
can not be adequately measured by any single indicator, although some
measures are more suitable than others. The commonly used citation
Impact Factor is not positioned at the core of this construct, but at
its periphery, and should thus be used with caution.
So what we currently have is too simplistic in an online world.
Which brings me on to the idea of equivalence. I wonder if we could follow Eric Schnell's line of creating equivalents for existing, and well understood, criteria. Let's take the Keynote speech as an example. Being invited, and giving, keynotes is often listed as one of the marks of esteem if you are seeking promotion. The reasons are twofold I believe:
- Reputation – it demonstrates that you have gained significant standing in your field to be asked regularly to give a keynote talk at a conference.
- Impact – if you are giving the keynote then everyone at the conference hears it, and you can therefore claim a significant impact in your subject.
The important element then is not the keynote itself, but what it signifies. If we start with this basis, then we can think of online equivalents. For example, if I give a talk and then put up a slidecast of that presentation, a certain number of views might equate to impact (how many people would hear a live presentation?). If the presentation is retweeted, linked to, embedded, then this might give an indication of reputation.
I don't think we can provide simple translations (500 views + 5 embeds = 1 keynote), which means there needs to be both a case made by the individual, and an interpretation by the panel, but by focusing on the existing criteria and considering what it is they are meant to demonstrate fundamentally, we can then talk about online equivalents.