Social Media Research Blog

Leaveraging Web and Social Media for Recommendations

Both Amazon and Netflix's business models rely on effective recommendation systems. The recommendations provided by such systems are based on the purchasing habits of millions of customers. As such, these systems are non-trivial and have evolved out of years of research in both academia and industry.

In addition to mining millions of customer transaction records, for many products there is a vast amount of information available online. While I do not have a lot of familiarity with recommendation systems literature, it seems obvious that the Web and Social Media is a great source of information that could be useful when building such systems.  Bloggers' profile pages, wishlists, netflix queues, book lists and the blog posts themselves are potential clues to learn which two items may be related to each other.

As a simple example, consider the movie "Pulp Fiction", by querying Google for all the inlinks to the IMDB homepage of Sin City Pulp Fiction and counting which are the other movies that are "co-cited" here is a list of five movies that are most likely to be related to "Pulp Fiction":

Most of these look quite relevant. Some critics have claimed similarities between Pulp Fiction and Snatch. One surprise though was LOTR, I wouldn't have expected it to be grouped with Pulp Fiction, but I guess I like them both very much -- so it seems reasonable in my case atleast.

Just for fun, here is another example with "Sin City" another one of my favorite movies.

Unless you have a large index of the Blogosphere or the Web, it would be quite inefficient to mine for such correlations (by passing queries to search engines) on a large scale. I do not know how much of the search engine information is leveraged in recommendation systems built by Amazon or Netflix.  It might also be worth looking into differences in the recommendations produced on the basis of "how people co-cite two products" vs. "how people purchase two products".

LiveBlogging Tools: CoveritLive

I had this idea, but turns out that CoveritLive does exactly the same thing and they have implemented it beautifully!

The current blogging tools usually fall short when trying to put up a liveblogging / running commentary post. The blogging software itself is supposed to be a reverse chronological listing of posts. Now when writing a live blog post of an event that the blogger is covering, what is usually needed is chronological listing of the news. (So its like an inverted blog within a post!). Mostly, people manage their own timestamps and formatting for the liveblogging post. This tends to be tedious and a distraction from the real event that one is trying to cover. Instead, CoveritLive is a great tool that handles all this for you!

The cool thing is that it also has features that support multiple authors, uploading media and rich formatting. Its a great idea and a fulfills a surprisingly obvious need. I think I am gonna use this for the next liveblogging event I cover. Do check it out!

Favorite Commandline Hack

One of my favorite commandline hacks is demonstrated by the following example:

history | gawk -F ' ' '{print $2}' | sort | uniq -c | sort -nr | more

What this does is takes a text file (or history of the commands used in our case), chops it to print the right field, sorts and counts the number of times a particular term occurs. For example here are the top commands I have used on this server:

    373    ls
    268    cd
     42    more
     29    ps
     27    rm
     25    du
     24    ./bin/startup.sh
     22    exit
     17    source
     15    emacs
     14    sudo
     13    ssh

This is immensely useful and a quick way to do anything from process a huge file, count the number of times a link occurs, word counting and all sorts of processing that come up frequently in large social media datasets. Its a really easy way to do some mundane tasks without having to write a script or much code for it. So what are your favorite commandline hacks? share the joy! :-)

SocialDevCamp East: Local BarCamp Event in Baltimore

I am excited about the local BarCamp event organized by David Troy of Twittervision and Ann Bernard, Keith Casey from WhyGoSolo. It looks like a fun event and a great "UnConference" for local entrepreneurs, social media enthusiasts and researchers to get together and talk about the future, innovation of Social Media and the Web.

It promises to be an event that would

bring together forward thinkers – developers, social media gurus, bizdev types – to discuss and Chart the Next Course.

The Agenda and signup details are available on the PBWiki site. Some of the proposed session topics (From the Wiki):

• Mobile Application Development with iPhone & Android
• What are the implications of Geo-based web applications?
• How will Agile Development practices affect ideation and funding processes?
• Platforms Present & Future
• What will matter on the web a year or two from now?
• Where will information aggregation take us?
• Social Media is here to stay – so where will it go next?
• How will Google keep affecting development, the web and our lives?
• What are the next things web users will want from the web?
• What will be the impact of future generations to the current web we know today?
• Social Media in the workplace vs. in life
• How the Web is being broken apart into pieces - Justin Thorp

Fantastic! Cant wait to be there!

There are also some sponsorship opportunities (starting @ $250) available for this event. Also please note that there is no cost for participants (which is great for grad students like me!). :-D

For more details please contact Dave Troy or come and chat with us on Facebook.

The Bane of Fame

I was working on this post and read a timely piece by Erik at Techcrunch on the level of noise and our inability to keep up with the conversations online. Further comments by Alexander as well.

Clay Shirky, in his book "Here Comes Everybody" mentions how fame can come at a cost. The example he presents is that of a blogger with 10,000 strong audience, is interacting with each of his readers for 1 min; it amounts to almost a full-time job ~ 40 Hours/Week. Here the interaction can mean that someone has just created a link and you go check it out; someone has written a blog post in response; posted a video/podcast; just commented on your blog, sent a tweet in response to yours, posted a photo from the conference you attended or plain old fashioned email/phone conversations.

I tried to extrapolate this and see how it would work in a typical scenario. Lets say you are an average blogger who spends 30 Mins /post/day  and your audience created 1000 items for you to responding to. Perhaps on an average you spend 5 Mins per response (this might include reading the comment, tweet, post, email; thinking time; writing time etc.) Then one would pretty much spend about ~25 Hours / Week for about 1000 responses/month.

This is the bane of fame! As you get famous, and receive more attention in your social network -- you cant respond to everyone. I can hardly figure out how Scoble can keep up his pace on his blog and Twitter! I think Tech bloggers are slightly better at managing how they use technology to their advantage and can keep up with a much larger audience than most people. Still, Shirky's book is a great place to understand how the attention economy is changing our social interactions.

That also makes me wonder if someone has seen any research on email/usenet usage patterns? Somehow that is one area that shares similarities and I am certain that usability experts would have done some such studies already. How do you think this would compare with Facebook and Social Media in general? There are people for whom Facebook is their default -- everything from email/IM to SN -- So it does not feel like they are trying to respond to requests from many different networks! I wonder what is the cost of context switching from one network to another and perhaps that is the reason why FriendFeed seems like a neat way to manage multiple information streams?

Twitter + Google App Engine + Huddle Chat = Flash Mob

When Dave Winer beckons the crowd follows. I was looking at Huddle Chat the new demo for Google App Engine and wanted to play around a bit. But before I could even think of inviting someone to join my chat, I saw the Tweet from Dave Winer calling us all to his chat room on the "Armed Services Committee".

Within literally a matter of minutes there was a crowd gathered sharing pics, chat messages and just having a fun time testing the app. It was like the largest virtual Campfire that just got together in a matter of minutes. Sweet! This is the amazing thing about Twitter. Now you have a whole bunch of people you can talk to and almost have a flash mob effect.

Also check out Techcrunch's demo appengine. OK I'm off to chatting with my new campfire friends about the Google App Engine and discuss politics! :-)

Notes from Brad Fitzpatrick's Invited Talk at ICWSM

Brad Fitzpatrick was over at ICWSM and gave a keynote on "Decentralization and Social Graphs". Following are some notes from the talk. Please note these are from rush transcription during the talk and from my recollection of what was presented. Please feel free to comment if I have missed something or if you feel something is incorrect.

Brad talks about Social graph and highlights some of the problems with traditional forms of communication. Email for example, lacks sender authentication or verification which lead to spam. Phone traditionally did not have any sender authentication and later these services were bolted on to the original infrastructure. Instant Messenger services don't always offer interoperability. Brad says that the  future of IM is Jabber/XMPP, which offers a decentralized system protocol. Gtalk, Livejournal both use Jabber and this makes it easily interoperable. In general the his talk highlights the importance of decentralization. Other things that are decentralized are email, wikis, ethernet, bittorrent etc.

Blogs are pretty much decentralized and interoperate using ATOM/RSS, but the difficult thing is commenting. Social Network sites are still "walled gardens" and basically need to re-invent everything -- sign up process, profile etc.... just to be able to provide one "cool feature". So as of now many of them dont work with each other. This is bound to change.

Brad says that the "glimmers of hope" are the following:

• XMPP Gtalk uses XMPP and in fact I have played around with it a bit and it is really a simple and neat way to communicate with instant Messengers.

• RSS/ATOM Ofcourse blog readers have relied on RSS/ATOM. From David Sifry's talk I recollect that ATOM is the preferred form of syndication for Technorati, since it avoids having to crawl the entire HTML content to be sure that partially syndicated feeds are handled correctly.

• OpenID This eases signups/logins. Already many of the providers and sites have adopted the OpenID standard.

• OAuth This provides an easy way for authorization... Often we are faced with a situation where some new random site says "hey why dont you gimme your password?". However, can you trust it with your password? OAuth is an alternative way for one website to request action on another.

• XFN/Microformat/FOAF Machine Readable format of the social graph. While there is a good amount of FOAF data online, it is hardly enough to cover the entire social graph. Apparently XFN is more popular and is also used by Twitter.

• OpenSocial This is a standard way for containers (social network sites) to host social apps. Same app works on Orkut, Myspace, Hi5 and .... now also Yahoo. This is a big leap towards interoperability.

• XRDS (Yardis) provides a discovery mechanism on a URL... suppose a site needs to ask another "where is your OpenID server" ... "Where is your calendar" etc.

• Social Graph API This is Brad's very own pet project. It finds and parses XFN, FOAF, me links. Provides an API for the social graph. A nice way to find missing friends. This solves the problem of not knowing all the alternative userids for my friends.

So to summarize the talk : use OpenID to log in to the site..... Site reuses public profile..... it can ask for pvt stuff using OAuth..... container app uses OpenSocial and Social Graph finds friends. And we all live happily ever after in our nice little online worlds :-) and all sites talk to each other and we are all F.R.I.E.N.D.S! :-)

The talk also highlighted some missing pieces:

    - messaging URLs

    - URL - email interop

    - XMPP/JID binding

    - Reputation server

    - promoting or documenting.

Some startups that are currently using open protocols twitter, dopplr, plaxo, pownce.

Brad says that users expect things to interoperate ... and decentralized is more robust. Social networks are broken today but are getting better.

Overall, it was a very interesting talk but I felt that it was kind of short and it would have also been fun to see more details.

Software, tools on my radar

1. Hadoop: Hadoop is backed by Yahoo! and is quite actively used by many researchers and startups alike. I am so psyched about Hadoop that I have been driving my labmates (and ex-colleagues) crazy about it right now. This weekend Vlad and I managed to set up Hadoop on our servers at ebiquity. Right now we have about six servers in a "mini cluster". For many large scale applications it is almost essential to process the data on distributed, parallel manner.

2. Hbase: is the bigtable implementation for Hadoop. I havent read much about it but from what I understand it is a column database.

3. Heretrix: is a crawler from Alexa. Apparently some of the social media startups like PersAI have been using it. At one point I was excited about Nutch essentially because it is really easy to use and run. Additionally, being a Lucene project Nutch is also capable of being run on hadoop. However, the sad part is that it comes with its own little hadoop installation and integrating it with the latest version of Hadoop seems to be more pain than I dont wish to get into right now.

4. Hama: I think this is a great project that is trying to port typical Linear Algebra operations on to Hadoop. This would essentially speed up development time for typical machine learning algorithms. It is like the LAPACK for MapReduce. Alas, the project is still under development and it would be wonderful if we could all contribute to it so that we can share the power of such tools.

5. Amazon EC2: Honestly, I did waste about 10 bucks playing with EC2 and getting Nutch up and running on the compute cloud. However, after talking to a few people I realized that it is perhaps not that cost effective if we were to run crawlers on EC2. It would be damn expensive to have a bunch of hadoop nodes running 24*7 fetching pages and storing it in S3. I know that Powerset and others use this for some of their processing. (Interestingly, I believe powerset has also contributed to Hbase). However, I think there is a tradeoff wherein for smallish jobs it is makes more economical sense to run the processes on some of the racks locally - which is the model I have chosen for the time being anyway.

6. SVN : Well for a CVS guy, it is a slight shift to move to SVN. But I think it looks like it is much easier to use and I think I would be totally moving my repositories to SVN.

7. LaTeX over PPT : I am so glad that I have discovered the joy of using LaTeX over PPT. It is so simple and easy to use the beamer package and the presentations come out looking slick and wonderful.

There are a whole bunch of things I am just waiting to get my hands on but im seriously crunched for time right now. :-(

Any other cool stuff I should watch out for? So, which cool new software/app/tool are you playing with these days? :-)