The book crowdsourcing has given me inspirations for a number of ideas. There are many examples in this book of how the crowds are collaborating, participating and sometimes even competing to make things better. The part that I find most exciting is to do with citizen science and citizen journalism.
- Take the patent review process for example. Peer-to-patent is a great example of how the crowds can work together with experts to speed up the broken patent system.
- The InnoCentive project that uses the diversity of knowledge from crowds to find solutions to tough research problems.
- How Talking points memo asked its readers to help parse through pages of documentation in the DoJ scandals.
- Assignmentzero and current TV are changing the way we think about reporting and journalism.
These examples make me wonder if we might be able to do the same thing with the academic review process. Academic reviews can be time consuming and lets just say --- "have their own problems". Academic research is built on the idea of sharing knowledge, tools and insights. To ensure their quality, results are scrutinized by a peer-review process. Ultimately, if approved, it leads to a publication. In theory, everything should work the way it is supposed to.
Like any system there are some practical problems. Conferences can adopt a double blind or a single blind review process. In either case, reviewer incentive and motivations vary drastically -- some are motivated by a sense of giving back to the community, others are just helping organize an important event in their research field, some might just like to see their name on the conference PC list, some like the privilege of reading interesting papers first, others like the privilege of having a say in how the field progresses, some (grad students) are volunteered by their professors -- to list just a few.
Reviewers are never "paid" and often have to find time outside their normal work commitments to fulfill these responsibilities, which may not be easy at times. Well I could write a whole post on this topic alone but that is for some other time....
Following are few ways in which I think crowdsourcing can help reform this system:
- Completely public: One approach is to make the review process completely public. Identity of the authors and the reviewers would be public and this could cause HUGE problems and for practical reasons this might be impossible to implement.
- Revealed Reviews: Once the papers are accepted, only those papers that made it and the corresponding reviews for these papers can be made public --without revealing the identity of the reviewer directly. Readers can comment on the reviewers' review and rate them. They could even write a review themselves if they like.
- Crowdsourced Reviews: The problem with revealing the reviews is the same as that of making the identity of the reviewer public. Subtle points mentioned in the reviews can betray the anonymity of the reviewer. The alternative, which I think is most practical, would be to conceal the reviews but allow readers to rate the paper and write their own reviews. Other users can read the reviews and rate the reviews themselves. The crowd is usually good at reaching a consensus in such systems. This would mean that for all accepted papers we have a pretty accurate judgment of what the community as a whole thinks of the paper. Now the scores of the reviewers can be matched against these ratings.The neat thing about this model is that not just academics, but also folks from industry and other walks of life can read academic papers and write reviews about them. And as the book Crowdsourcing mentions "diversity often trumps ability". More people reading and scrutinizing academic work is good for progress of research.
A point to note here is that compensating a reviewer's time in terms of money is not going to work.I do not believe that money is the motivation for reviewers (the same reason why it did not work with Google Q&A). Free conference registrations in lue of their time would be one way to go. Most often though registrations are paid for by the company or the grant. Finally, some conferences have started inviting PC members for a special lunch. Ofcourse, there is no such thing as a free lunch! :-)