Social Media Research Blog

SXSW Panel on Collaborative filters

I am at SXSW this weekend and wow what a conference this has been! Met a lot of cool people and been to a bunch of happening parties! I plan to post a few updates about the conference. In the mean time here is a quick post from notes I wrote during the panel discussion on Collaborative Filters.

Anton Kast from Digg.com gave a brief history of collaborative filtering from the research literature covering tapestry (from parc) grouplens (UMN) and a few other projects mostly from 1990s and draws comparisons to current systems used in spam filters, pagerank , tagging systems and applications in facebook ads etc. The talk wasnt very technical and meant to be a quick overview for the audience that might not be familiar with the research background. Kast described recommendation as the output of collaborative filtering that is personalized for a particular user in the system. For example  amazon (people who bought this also bought..), behavioral ad targets, google news etc. I wish Kast had spent more time discussing the last slide which covered practical problems: sparsity problem in recommendation systems, early rater problem/cold start, gray sheep (small group very popular).

Erik Frey from last.fm shared some interesting stats -- 25m unique users/month. The types of relations that last.fm derives: songs-users, users-users. He also shared some insights on what were described as lean forward and lean backward mode in music recommendation systems. The lean forward mode is when a user is feeling a little experimental and would like to discover new stuff. The lean backwards mode is when the emphasis is on continuity and the desire to listen to something familiar (i.e. radio experience). The two main data sources for last.fm are : scrobbles, social info like tags. Frey suggests that tags are quite useful to explain recos

The next speaker was the cto of baynote -- a recommendation system that is provided as service using javascripts that enables clients to customize their website in real time based on user behavior. They also apply it for reordering the search results.

Finally, Jon Sanders from Netflix gave a very good talk covering the history of netflix and its evolution. Today netflix has about 2 billion ratings and gets about 2 M ratings per day. 60% of movies watched are via personalized recos -- amazing!

Following were the steps in evolution of netflix:

step 1 At first netflix was a editorially managed site and the first evolution was that of adding a rating widget.

step 2 score and sort all movies using rating alone (top k list)

step 3 movie similarity by k-nearest

step 4 interest based recos (same actor director  etc)

step 5 ask ( get users to rate genre)

step 6 ask others (top 10 lists from users)

step 7 explain why (because you enjoyed)

step 8 $1M prize

step 9 develop a unique experience and customized website for each customer

future: streaming video specific recommendations, incorporating implicit and explicit metadata, discovery focus, let people drive and not be led.

Community-Driven Moderation in Twitter

Twitter played an important role during the mumbai terrorist attacks. The community that formed around the #mumbai tag, actively discussed how they felt and provided live, uptodate information filtered from different news sources. This was a tipping point for twitter that transforms it from a social tool to a news source. The most important factor here was that the dynamic community that emerged around the #mumbai tag was not constrained to discuss one single topic. Something that News sources struggled with as the developments happened so quickly in this critical event. Most news sources need to stay on a story for at least a few minutes before they can move on to another story.

As #mumbai became popular, trolls showed up on twitter and posted messages with the hashtag only to draw attention to their website or link to other sources of little or no relevance whatsoever to the current crisis. This made me wonder, what if anything could be done in Microblogging environments to restrict users from misusing hashtags. A quick glance over at the current use of #mumbai shows a number of such sources that should probably moderated. I am refraining from naming any particular users but this is something that could be better dealt with by the community as a whole.

One suggestion I have is to create a new hashtag like #ban followed by the user name and the hashtag for which the user should be banned. Once a particular user is #ban tagged by several people from the community -- Twitter can automatically ban the user from further posting on the specific topic. This would be a simple and effective means for a community driven moderation in twitter.

Obama vs. McCain

Frankly, as an outsider who can not vote, my opinion would not matter much but I am hoping Barack Obama wins!

This election calls for Americans to make a very important decision, one that impacts not just their lives but the World economy and political stability. This election is also important and interesting event from a different standpoint. Never before have there been so many people using the Web. Social Media tools, blogs, twitter, facebook and virtually every imaginable resource online is now a platform for political activism. In the coming months and years, researchers will be collecting, scrutinizing and analyzing these datasets (some folks already are!)d.

A more immediate feel for the current political pulse of the population can be obtained by looking at search results and online electronic prediction markets -- both of which seem to indicate a tight race with a slight lead by Obama in the recent few days. GOBAMA! check out the Google insight graphs below:

 

Note: The bump on 27th Sept due to the Presidential debate and 3rd Oct after the VP debate.

Twitter Election 2008

Check out Twitter Election 2008

It is a fantastic site that Twitter has put out! According to Twitter:

We’re filtering hundreds of Twitter updates per minute to create a new source for gathering public opinion about the presidential election and a new way for you to share your thoughts.

I love it! Unfortunately, there are so many updates that an RSS feed would be pretty much pointless. So I wish that there was some better way to provide an API to the election site with a  live stream so that data junkies like us could totally geek out and apps could be developed around the real time election twitter updates!

Book Recommendation: Crowdsourcing

I have already raved about this book in my previous posts. Crowdsourcing, by Jeff Howe is a book that came out of a wired magazine article (just like Long Tail) and a blog by the author. I picked up this book almost on an impulse on my return trip from KDD conference in Vegas last month. At first, I thought it would be a lot of the same -- examples seen over and over again..Well, this book was far from that! The book is just fabulous and I really enjoyed reading every page. Jeff Howe does a great job presenting crowdsourcing in three parts: "how we got here", "where we are" and "where we are going". In essence, crowdsourcing is about the idea that "diversity trumps ability". Bring enough people together to take a crack at a really difficult problem and the diverse backgrounds and knowledge that the crowd brings to the table will inevitably outsmart the smartest people who, in some cases, may have been working on the problem for a long time. The author gives several examples from InnoCentive, where research problems are posed for the crowd to predictive markets (like IEM for politics and Marketocracy for stocks) which are much more effective than election polls and fund managers. To understand how some of these work you really should grab a copy of the book. Crowdsourcing is also about how we are now both producers and consumers of information. Prosuming news and media is part of our online activity and be it editing wikipedia, commenting on a blog, uploading or rating a video we are all participating in it (sometimes in surprising ways -- without even noticing that we are!). The book is full of several interesting examples from Threadless.com a crowdsourced tshirt design site to how a punk rock band that marketed itself on myspace. Counterintuitive the way large Newspapers have been thinking, those news sites that have adapted to this new trend have actually benefited and increased online readership. Crowdsourcing makes a big leap in how a billion people connect and someday might even help solve difficult, big problems like cancer and Lou gehrig's disease. This book is really about how crowdsourcing is changing human communication, content production, the way we share ideas thoughts and solve problems and ultimately our culture. Finally, as Jeff puts it... the spin on Robert Putnam's classic work is that we are no longer bowling alone, we are now just bowling online.

Crowdsourcing PC ads

Bill Gates, Seinfeld ads barely match up to any of the user-generated ads I have seen over at YouTube. IMHO, $300 Million would be better spent on crowdsourcing Microsoft's ad campaign. There are some really cool ads out there, here are a few:

Why would you pay $300 Million? Why bother if the whole Seinfeld ad is pulled off or not! -- when you can let your users champion your cause. 

Share the Love

What makes social media so compelling is the ability to share information with others. ShareThis is a cool widget I use on my blog to enable users to share the posts with others and to easily post it on different sites like digg, StubleUpon, FriendFeed, etc or via email. Recently ShareThis commissioned a study conducted by Forrester Research, that analyzed the sharing patterns across a number of users. Following is an excerpt from their blog post:

High level findings:

• Email is still primary channel for sharing -- 69 percent of adults cite email as the primary source of receiving information
• Less than one-third of online adults said they learned about the new content from generally shared content sources such as YouTube, a wiki, or social networking sites
• 64 percent of adults and 60 percent of youth still use the traditional cut and paste  method to share a URL or information
  • However this is not trackable by publishers, making sharing by email tough to measure
• Though the primary motivation for sharing differs, 81 percent of adults claim that, when cutting and pasting content to share, they share to help others - believing that a person will benefit from the information they share

Surprisingly, email is still the primary channel for sharing and what was more interesting is that people still do it the old fashioned cut-and-paste way. The neat thing about using a tool like ShareThis is that it gives you a "one-click" way to share something with your network via email, IM or on any social media site. I somehow think that email is not quite the right format for sharing links and interesting articles. I like email to be more about actionable items: but then that is just me. I think FriendFeed, Twitter or Facebook is more suited for sharing links and other cool stuff.

There are a number of interesting nuggets on sharing demographics and differences in sharing behavior based on gender. From the post:

• The motivation for sharing amongst men differs than those of women
  • In general, men are more likely to share something to appear cool or to look like an expert
  • Men more likely to share product recommendations and video than women; 77 percent of adult males and 74 of younger males shared news and web links
• Women are more likely to share products or ideas they like via easy or direct sharing methods
  • Women strongly favor send to a friend feature; more than 60 percent of adult woman use the send to a friend feature on websites
  • Women are more likely to share directly, especially via cell phone texting
  • Adult women tend to face more obstacles when trying to share content online, with 40 percent saying they could find no ways to share the information directly from the Web page

I read through parts of the report I had received from ShareThis -- lots of nice stats there. The full report can be requested from [email protected] (Unfortunately, I cannot post it directly here.)

Semantic Ecoblogging and Citizen Science

Today during our weekly eBiquity lab group meetings, Joel Sachs updated us on the Spire project. Spire is a research project investigating how semantic web technologies can be used to support science in general and the field of ecoinformatics in particular. During lunch, Joel and I talked about crowdsourcing and how social media and citizen science can help innovation and discovery in scientific applications. Many of the tasks of a field researcher involves tedious and careful logging of the ecology and habitat in an environment. This is exactly where amateur scientists and hobbyists can make a big difference and contribute to science. Most amateurs contribute to such projects, just because they enjoy it and love working on it as a hobby. One example of how crowdsourcing is currently done in Spire is through: Splickr tool: that allows you to query for geotagged photos on Flickr. Spotter tool: "Spotter allows you to easily create RDF/OWL for species observations. It places a button on your Firefox toolbar that calls up a form with fields taken from our observation ontology (Observer, Reporter, Location, Common Name, Scientific Name, etc.)." (description quoted from the site) While reading the book crowdsourcing, I learned of similar efforts by NASA, and Cornell University's Ornithology lab. Many discoveries in science are now coming from amateur researchers -- ant not just from folks with multi-million dollar grants backing them. A few examples from the book are: InnoCentive many open research problems are being solved by a part-time-hobbyist-amateur type researchers. INSPIRE NASA is harnessing wisdom of the crowds to help analyze images The rediscovery of Cozumel Thrasher, a rare bird species by a bunch of amateur birders. And many more examples .... (I really recommend reading the book - it is well written and you will not be able to put it down!) During the meeting and later, we were intrigued by the idea of developing an iPhone app to allow researchers and 'citizen scientists' to take pictures of species in the wild and upload them to spire. One important point here is that often an amateur researcher might not be aware of the scientific name of a species, but given we have enough eyes looking at the data being uploaded, we can divide the task of labeling and categorizing the findings. Citizen scientists can contribute at various levels -- performing field studies, annotating, building ontologies, etc. Since, Spire is a Semantic Web application backed by ontologies, one can now reason over the findings and learn that an invasive species is posing a significant threat to the local habitat. This is the ultimate aim of Spire and to me, this is the killer app for Web 3.0 -- if you really insist on a term like that! For more information on Spire please refer to the following publications.