Social Media Research Blog

Crowdsourcing Academic Paper Reviews?

The book crowdsourcing has given me inspirations for a number of ideas. There are many examples in this book of how the crowds are collaborating, participating and sometimes even competing to make things better. The part that I find most exciting is to do with citizen science and citizen journalism.

• Take the patent review process for example. Peer-to-patent is a great example of how the crowds can work together with experts to speed up the broken patent system.
• The InnoCentive project that uses the diversity of knowledge from crowds to find solutions to tough research problems.
• How Talking points memo asked its readers to help parse through pages of documentation in the DoJ scandals.
• Assignmentzero and current TV are changing the way we think about reporting and journalism.

These examples make me wonder if we might be able to do the same thing with the academic review process. Academic reviews can be time consuming and lets just say --- "have their own problems". Academic research is built on the idea of sharing knowledge, tools and insights. To ensure their quality, results are scrutinized by a peer-review process. Ultimately, if approved, it leads to a publication. In theory, everything should work the way it is supposed to. 

Like any system there are some practical problems. Conferences can adopt a double blind or a single blind review process. In either case, reviewer incentive and motivations vary drastically -- some are motivated by a sense of giving back to the community, others are just helping organize an important event in their research field, some might just like to see their name on the conference PC list, some like the privilege of reading interesting papers first, others like the privilege of having a say in how the field progresses, some (grad students) are volunteered by their professors -- to list just a few.
Reviewers are never "paid" and often have to find time outside their normal work commitments to fulfill these responsibilities, which may not be easy at times. Well I could write a whole post on this topic alone but that is for some other time....

Following are few ways in which I think crowdsourcing can help reform this system:

• Completely public: One approach is to make the review process completely public. Identity of the authors and the reviewers would be public and this could cause HUGE problems and for practical reasons this might be impossible to implement.
• Revealed Reviews: Once the papers are accepted, only those papers that made it and the corresponding reviews for these papers can be made public --without revealing the identity of the reviewer directly. Readers can comment on the reviewers' review and rate them. They could even write a review themselves if they like.
• Crowdsourced Reviews: The problem with revealing the reviews is the same as that of making the identity of the reviewer public. Subtle points mentioned in the reviews can betray the anonymity of the reviewer. The alternative, which I think is most practical, would be to conceal the reviews but allow readers to rate the paper and write their own reviews. Other users can read the reviews and rate the reviews themselves. The crowd is usually good at reaching a consensus in such systems. This would mean that for all accepted papers we have a pretty accurate judgment of what the community as a whole thinks of the paper. Now the scores of the reviewers can be matched against these ratings.The neat thing about this model is that not just academics, but also folks from industry and other walks of life can read academic papers and write reviews about them. And as the book Crowdsourcing mentions "diversity often trumps ability". More people reading and scrutinizing academic work is good for progress of research.

A point to note here is that compensating a reviewer's time in terms of money is not going to work.I do not believe that money is the motivation for reviewers (the same reason why it did not work with Google Q&A). Free conference registrations in lue of their time would be one way to go. Most often though registrations are paid for by the company or the grant. Finally, some conferences have started inviting PC members for a special lunch. Ofcourse, there is no such thing as a free lunch! :-)

Book Recommendation: Crowdsourcing

I have already raved about this book in my previous posts. Crowdsourcing, by Jeff Howe is a book that came out of a wired magazine article (just like Long Tail) and a blog by the author. I picked up this book almost on an impulse on my return trip from KDD conference in Vegas last month. At first, I thought it would be a lot of the same -- examples seen over and over again..Well, this book was far from that! The book is just fabulous and I really enjoyed reading every page. Jeff Howe does a great job presenting crowdsourcing in three parts: "how we got here", "where we are" and "where we are going". In essence, crowdsourcing is about the idea that "diversity trumps ability". Bring enough people together to take a crack at a really difficult problem and the diverse backgrounds and knowledge that the crowd brings to the table will inevitably outsmart the smartest people who, in some cases, may have been working on the problem for a long time. The author gives several examples from InnoCentive, where research problems are posed for the crowd to predictive markets (like IEM for politics and Marketocracy for stocks) which are much more effective than election polls and fund managers. To understand how some of these work you really should grab a copy of the book. Crowdsourcing is also about how we are now both producers and consumers of information. Prosuming news and media is part of our online activity and be it editing wikipedia, commenting on a blog, uploading or rating a video we are all participating in it (sometimes in surprising ways -- without even noticing that we are!). The book is full of several interesting examples from Threadless.com a crowdsourced tshirt design site to how a punk rock band that marketed itself on myspace. Counterintuitive the way large Newspapers have been thinking, those news sites that have adapted to this new trend have actually benefited and increased online readership. Crowdsourcing makes a big leap in how a billion people connect and someday might even help solve difficult, big problems like cancer and Lou gehrig's disease. This book is really about how crowdsourcing is changing human communication, content production, the way we share ideas thoughts and solve problems and ultimately our culture. Finally, as Jeff puts it... the spin on Robert Putnam's classic work is that we are no longer bowling alone, we are now just bowling online.

Semantic Ecoblogging and Citizen Science

Today during our weekly eBiquity lab group meetings, Joel Sachs updated us on the Spire project. Spire is a research project investigating how semantic web technologies can be used to support science in general and the field of ecoinformatics in particular. During lunch, Joel and I talked about crowdsourcing and how social media and citizen science can help innovation and discovery in scientific applications. Many of the tasks of a field researcher involves tedious and careful logging of the ecology and habitat in an environment. This is exactly where amateur scientists and hobbyists can make a big difference and contribute to science. Most amateurs contribute to such projects, just because they enjoy it and love working on it as a hobby. One example of how crowdsourcing is currently done in Spire is through: Splickr tool: that allows you to query for geotagged photos on Flickr. Spotter tool: "Spotter allows you to easily create RDF/OWL for species observations. It places a button on your Firefox toolbar that calls up a form with fields taken from our observation ontology (Observer, Reporter, Location, Common Name, Scientific Name, etc.)." (description quoted from the site) While reading the book crowdsourcing, I learned of similar efforts by NASA, and Cornell University's Ornithology lab. Many discoveries in science are now coming from amateur researchers -- ant not just from folks with multi-million dollar grants backing them. A few examples from the book are: InnoCentive many open research problems are being solved by a part-time-hobbyist-amateur type researchers. INSPIRE NASA is harnessing wisdom of the crowds to help analyze images The rediscovery of Cozumel Thrasher, a rare bird species by a bunch of amateur birders. And many more examples .... (I really recommend reading the book - it is well written and you will not be able to put it down!) During the meeting and later, we were intrigued by the idea of developing an iPhone app to allow researchers and 'citizen scientists' to take pictures of species in the wild and upload them to spire. One important point here is that often an amateur researcher might not be aware of the scientific name of a species, but given we have enough eyes looking at the data being uploaded, we can divide the task of labeling and categorizing the findings. Citizen scientists can contribute at various levels -- performing field studies, annotating, building ontologies, etc. Since, Spire is a Semantic Web application backed by ontologies, one can now reason over the findings and learn that an invasive species is posing a significant threat to the local habitat. This is the ultimate aim of Spire and to me, this is the killer app for Web 3.0 -- if you really insist on a term like that! For more information on Spire please refer to the following publications.