Crowdsourcing Academic Paper Reviews?

The book crowdsourcing has given me inspirations for a number of ideas. There are many examples in this book of how the crowds are collaborating, participating and sometimes even competing to make things better. The part that I find most exciting is to do with citizen science and citizen journalism.

• Take the patent review process for example. Peer-to-patent is a great example of how the crowds can work together with experts to speed up the broken patent system.
• The InnoCentive project that uses the diversity of knowledge from crowds to find solutions to tough research problems.
• How Talking points memo asked its readers to help parse through pages of documentation in the DoJ scandals.
• Assignmentzero and current TV are changing the way we think about reporting and journalism.

These examples make me wonder if we might be able to do the same thing with the academic review process. Academic reviews can be time consuming and lets just say --- "have their own problems". Academic research is built on the idea of sharing knowledge, tools and insights. To ensure their quality, results are scrutinized by a peer-review process. Ultimately, if approved, it leads to a publication. In theory, everything should work the way it is supposed to. 

Like any system there are some practical problems. Conferences can adopt a double blind or a single blind review process. In either case, reviewer incentive and motivations vary drastically -- some are motivated by a sense of giving back to the community, others are just helping organize an important event in their research field, some might just like to see their name on the conference PC list, some like the privilege of reading interesting papers first, others like the privilege of having a say in how the field progresses, some (grad students) are volunteered by their professors -- to list just a few.
Reviewers are never "paid" and often have to find time outside their normal work commitments to fulfill these responsibilities, which may not be easy at times. Well I could write a whole post on this topic alone but that is for some other time....

Following are few ways in which I think crowdsourcing can help reform this system:

• Completely public: One approach is to make the review process completely public. Identity of the authors and the reviewers would be public and this could cause HUGE problems and for practical reasons this might be impossible to implement.
• Revealed Reviews: Once the papers are accepted, only those papers that made it and the corresponding reviews for these papers can be made public --without revealing the identity of the reviewer directly. Readers can comment on the reviewers' review and rate them. They could even write a review themselves if they like.
• Crowdsourced Reviews: The problem with revealing the reviews is the same as that of making the identity of the reviewer public. Subtle points mentioned in the reviews can betray the anonymity of the reviewer. The alternative, which I think is most practical, would be to conceal the reviews but allow readers to rate the paper and write their own reviews. Other users can read the reviews and rate the reviews themselves. The crowd is usually good at reaching a consensus in such systems. This would mean that for all accepted papers we have a pretty accurate judgment of what the community as a whole thinks of the paper. Now the scores of the reviewers can be matched against these ratings.The neat thing about this model is that not just academics, but also folks from industry and other walks of life can read academic papers and write reviews about them. And as the book Crowdsourcing mentions "diversity often trumps ability". More people reading and scrutinizing academic work is good for progress of research.

A point to note here is that compensating a reviewer's time in terms of money is not going to work.I do not believe that money is the motivation for reviewers (the same reason why it did not work with Google Q&A). Free conference registrations in lue of their time would be one way to go. Most often though registrations are paid for by the company or the grant. Finally, some conferences have started inviting PC members for a special lunch. Ofcourse, there is no such thing as a free lunch! :-)

Comments

[Spiros Papadimitriou]

Totally agree, and I hope to eventually see more along those lines. I don't have any concrete plan, but I would seriously consider to volunteer and help implement a site for what you say if any conference organizers would take a chance on this.

[Akshay Java]

hi Spiros, I think that would be super cool. I have been seriously thinking of doing something on these lines but have been tied up with my PhD dissertation writing. Here is an idea -- I am on the organizing committee for ICWSM. I believe this would be a perfect place to try out such a site. The paper deadlines are in January so that gives us some time to work on this project. The audience at ICWSM totally 'gets it' when it comes to social media and crowdsourcing. I think it would be a fun! Would you like to sync up on Gtalk or email somtime? For starters, I was trying to modify Pligg, an open source digg to adapt it for paper submissions. If this works, who knows the best paper can be picked by the participants at the conference!

[Akshay Java]

As I dig further into this, I find that some venues and journals in other fields are already using an Open Review process. Interestingly, I found the alt.CHI community which, "invites controversial ideas, novel prototypes, failed but valuable user studies, bold experiments, and anything else that can give a fresh perspective on CHI.". They use an open, peer reviewed process. Following is their description:

"Alongside the invitation to submit papers, we also invite reviewers to take part in the alt.chi process. The alt.chi 2007 program will be selected through a non-anonymous process, where all submissions and reviews are completely open. There will be an open forum were anyone can register to submit and review, and all submitters will be required to take part in the selection process. All reviews and submissions will be available to anyone who registers."

Indeed very interesting to see this!

[dissertation help]

I’ve used other services before and I’ve always had problems. Either the work was really bad, or I never received the work! So when I found dissertation-help.co.uk, I just thought I was going to be the same problems, but I was desperate. I decided to call them, because I’d had to email all the other companies, and I thought it would be good to grill an actual member of staff! I was surprised to find the staff really helpful and then even more surprised that the work was great too! The price was higher than the services I’d used before, but I’d rather pay more for quality!
________________________________________

Dissertation Help