Social Media Research Blog

Community-Driven Moderation in Twitter

Twitter played an important role during the mumbai terrorist attacks. The community that formed around the #mumbai tag, actively discussed how they felt and provided live, uptodate information filtered from different news sources. This was a tipping point for twitter that transforms it from a social tool to a news source. The most important factor here was that the dynamic community that emerged around the #mumbai tag was not constrained to discuss one single topic. Something that News sources struggled with as the developments happened so quickly in this critical event. Most news sources need to stay on a story for at least a few minutes before they can move on to another story.

As #mumbai became popular, trolls showed up on twitter and posted messages with the hashtag only to draw attention to their website or link to other sources of little or no relevance whatsoever to the current crisis. This made me wonder, what if anything could be done in Microblogging environments to restrict users from misusing hashtags. A quick glance over at the current use of #mumbai shows a number of such sources that should probably moderated. I am refraining from naming any particular users but this is something that could be better dealt with by the community as a whole.

One suggestion I have is to create a new hashtag like #ban followed by the user name and the hashtag for which the user should be banned. Once a particular user is #ban tagged by several people from the community -- Twitter can automatically ban the user from further posting on the specific topic. This would be a simple and effective means for a community driven moderation in twitter.

Live Labs Social Streams

Very cool! Live Labs releases the Political Streams project. It is a lense into the people and places that are receiving a lot of attention on blogs and news.

You can read the post snippet and see neat graphs of how the attention changes over time:

Also browse through the related names and places as you explore different topics and blog posts/news stories. Furthermore there are profiles of people and places in news that are generated using Freebase.

 

This is really cool! If you looked at search trends, many of the search terms are in fact about places and people in News. Users go to a search engine to find out both why they are in news? and often who the person is? Social streams (right now, for political streams) let you explore this in a real time, interactive manner. 

Nice!

Mean Time to Update Feeds

tIn the State of the Blogosphere report posted by Technorati recently, they describe how the top bloggers update their blogs more frequently than the rest of the population. Given the data I have been gathering on the Bloglines subscriptions, I was curious to see what how most subscribed blogs behaved? For the top 100K feeds ranked by the number of subscribers, I fetched their RSS/ATOM files (some failed, some were filtered due to few posts etc). Using this handy PERL module, I parsed all the pubDates (sorry, RSS only for now) and converted them to their corresponding Unix timestamps. Now for each of these feeds, the Mean Time to Update (ie the avg of the number of days between consequent posts) can be obtained. Interestingly, some fraction of the posts were updated after they had been published (hence were out-of-sequence in the RSS feed). For this analysis, I do not consider these updates -- it would be interesting to dig further into it.

Following graph F(x) is defined as the proportion of X values less than or equal to x. Thus you can observe that for the top 10K feeds 40% of the feeds updated in a day or less. For the rest of the feeds it was 30%. I believe as we go lower in the ranking, this would keep decreasing even further. Interestingly, overall, for the highly subscribed feeds, almost 60% update at least once a week. 

For a feed reader keeping an index fresh and up-to-date is then a function of how many users subscribe to a feed, how often the feed is updated, when was the last time the feed was requested (by any user in the system) and perhaps even when the last ping was registered for a particular feed in the ping stream. I believe in Google reader's case, clicking "show details" displays the last time when the feed was refreshed -- which I have never found to be over 3 hours for any feed that I subscribe to.

ACK: The above plot was made in MATLAB using this script for cdfplot.

When Did You Last Update Your Blog?

Think of all the blogs you have started till date. How many are still active? Yep! We have all been there: started a blog and never kept up with it, got busy with other interests or simply do not find enough time to keep up with blogging. 

I wanted to see what is the overall distribution of the date when a post was made. From a random sample of 50,000 feeds that were extracted from the Bloglines feed subscriptions, following is the distribution of the date when the post was made.

From the random sample of feeds downloaded (~42K successfully; approx between 10-15 posts per feed) , 66% of the posts were made in 2008. As for the rest, some even date back to 2001. This indicates that from a random sample, a sizable number of feeds have posts that have been updated within the last year.

This also comes back to my previous point about efficient feed crawling strategies. How do you dedicate resources for keeping a feed index fresh and ensure a good coverage/quality? It would be interesting to see how the following two metrics change as we move from most subscribed to least subscribed feeds:

1. Mean Time To Update Between Posts: Do highly subscribed feeds generally update more frequently?
2. lastBuildDate: When was the feed last (re)published?

Twitter Election 2008

Check out Twitter Election 2008

It is a fantastic site that Twitter has put out! According to Twitter:

We’re filtering hundreds of Twitter updates per minute to create a new source for gathering public opinion about the presidential election and a new way for you to share your thoughts.

I love it! Unfortunately, there are so many updates that an RSS feed would be pretty much pointless. So I wish that there was some better way to provide an API to the election site with a  live stream so that data junkies like us could totally geek out and apps could be developed around the real time election twitter updates!

Share the Love

What makes social media so compelling is the ability to share information with others. ShareThis is a cool widget I use on my blog to enable users to share the posts with others and to easily post it on different sites like digg, StubleUpon, FriendFeed, etc or via email. Recently ShareThis commissioned a study conducted by Forrester Research, that analyzed the sharing patterns across a number of users. Following is an excerpt from their blog post:

High level findings:

• Email is still primary channel for sharing -- 69 percent of adults cite email as the primary source of receiving information
• Less than one-third of online adults said they learned about the new content from generally shared content sources such as YouTube, a wiki, or social networking sites
• 64 percent of adults and 60 percent of youth still use the traditional cut and paste  method to share a URL or information
  • However this is not trackable by publishers, making sharing by email tough to measure
• Though the primary motivation for sharing differs, 81 percent of adults claim that, when cutting and pasting content to share, they share to help others - believing that a person will benefit from the information they share

Surprisingly, email is still the primary channel for sharing and what was more interesting is that people still do it the old fashioned cut-and-paste way. The neat thing about using a tool like ShareThis is that it gives you a "one-click" way to share something with your network via email, IM or on any social media site. I somehow think that email is not quite the right format for sharing links and interesting articles. I like email to be more about actionable items: but then that is just me. I think FriendFeed, Twitter or Facebook is more suited for sharing links and other cool stuff.

There are a number of interesting nuggets on sharing demographics and differences in sharing behavior based on gender. From the post:

• The motivation for sharing amongst men differs than those of women
  • In general, men are more likely to share something to appear cool or to look like an expert
  • Men more likely to share product recommendations and video than women; 77 percent of adult males and 74 of younger males shared news and web links
• Women are more likely to share products or ideas they like via easy or direct sharing methods
  • Women strongly favor send to a friend feature; more than 60 percent of adult woman use the send to a friend feature on websites
  • Women are more likely to share directly, especially via cell phone texting
  • Adult women tend to face more obstacles when trying to share content online, with 40 percent saying they could find no ways to share the information directly from the Web page

I read through parts of the report I had received from ShareThis -- lots of nice stats there. The full report can be requested from bizdev@sharethis.com (Unfortunately, I cannot post it directly here.)

Prosuming News

The future is now my people! And it is happening on Twitter+CNN: We are prosuming news.
The term Prosumer is described on Wikipedia as:

Prosumer is a portmanteau formed by contracting either the word professional or producer with the word consumer.
...
In the 1980 book, The Third Wave, futurologist Alvin Toffler coined the term "prosumer" when he predicted that the role of producers and consumers would begin to blur and merge (even though he described it in his book Future Shock from 1970). Toffler envisioned a highly saturated marketplace as mass production of standardized products began to satisfy basic consumer demands. To continue growing profit, businesses would initiate a process of mass customization, that is the mass production of highly customized products.

Incidentally, I first learned about the term prosumer from the book Crowdsourcing by Wired reporter Jeff Howe.

Over at CNN what is happening right now is (Via Venturebeat)

One anchor for the cable news channel CNN has become enamored with the micro-messaging service Twitter over the past several weeks. He used tweets (Twitter messages) to complement his coverage of the recent Hurricane Gustav and now the network has apparently decided to take it one step further, basing a whole show around the real-time citizen journalism/opinions that take place on Twitter.

CNN reporter, Rick Sanchez's Twitter updates are being followed by over 9000 users as of this date (which leads to interesting infrastructure issues over at Twitter -- see venturebeat post). I did watch some of the user questions from Twitter on CNN's coverage of the presidential elections and had planned to write about it earlier. Rick asks his Twitter audience about their opinion on things like

Which refers to Congressman Lynn Westmoreland's words about Barack Obama. News media is now all about citizen journalism and participatory audience. Emails, Web forms and premium rate SMS still work but Twitter is the new way to go to get the audience involved!

So, are we really that far off from watching our nightly news cast from some amateur reporter on YouTube who has stitched together news stories submitted by his/her audience? I am amazed at Toffler's visionary outlook on this.

The future is happening right  and we are prosuming it today!

UPDATE 09/12/08

Rick also has a facebook profile. (via Steve Rubel on FB)

Invited Talk KDD 2008: Internet Advertising and Optimal Auction Design

Following are a few running notes from the KDD invited talk "Internet Advertising and Optimal Auction Design" by Dr. Michael Schwarz from Yahoo! Research:

Dr. Schwarz  explains that the original advertising models were quite simple but now new tweaks and optimizations are being added to, for example, estimate click probabilities, match ads for long tail of queries, etc. So there is a "Fudge" factor in online advertising algorithms, and the question now is how to combine these correctly?

In sponsored search options space, Overture was really the pioneer of pay-per-click. In such a scheme, the bidder would pay whatever price that was fixed during the auction of an advertising position. However, soon people realized that generalized first auction is unstable. This is because if advertisers can keep changing their bids and the minimum bid keeps fluctuating. The situation is illustrated in the following paper and as shown in the image below:

This system was also prone to manipulation by automated robots quite easily.

Next, in 2002 Google came up with the Generalized Second-Price option (GSP): here the advertiser pays for the price fixed by the next highest bidder (plus a minor amount like a cent), upon winning an auction. Later Yahoo! followed. In this mechanism the assumption is that for the ad position i, clicks are almost independent of the advertisement. A clock shows the current price that keeps increasing, a bid price is fixed only at the time of dropping out and the final payments are computed according to GSP.  Dr. Schwarz presents how this is related to "Generalized English Auctions".  The intuition is that the advertisers values the price just at the point where the advertiser drops out of the game. If we think about it, at each point the advertiser is making a decision to go closer to the point where he values the position enough but not to a point where he/she would have regrets if the next highest bidder would drop out of the auction, causing a win. The GSP approach has been shown to have a unique equilibrium. For more details you can read the paper: "Internet advertising, generalized second price options".

Finally, the talk discusses an optimal matching problem. If you know the probability distribution from which the values for the bidders are drawn, in theory, the highest probable match (ie optimal auction) is deterministic. According to further studies in these advertising models, Dr. Schwarz suggests that despite multiple units the value for a slot, the auction is not dependent on the number of slots! This is great from a search engine's perspective. Although I guess this hold to a certain number of slots -- or else search results would have looked rather different! Further, the decline in CTR from one slot to another also does not have any influence on the auction. This is perhaps where clever data mining techniques can be applied to help estimate the probability distribution of bidder's values for each slot and  ultimately, lead to setting optimal reserve price. I guess this is perhaps the point where search engines start guarding their trade secrets closely.

Finally, during discussions with many people at the conference, on theme that kept coming up is that of social advertising and monetization of social networks -- which has been a typically difficult problem. Personally, the way I see this (more of my 2 cents here) is three fold:

• One, are you advertising for the purpose of targeting an individual (based on the profile information)?
• Two, are you targeting an individual based on the actions of her friends and neighbors in the social graph?
• Three, are you targeting a group or a community in the network?

KDD is an exciting venue precisely because it is at the confluence of data mining, social media, large scale text and graph analysis and monetization-- all the themes that have made it a great learning experience for me.

Where is a Company Located?

Geographic information is an important factor in social media analysis. In this post I used Google's super cool new service Google Insights to identify in which state a company is located. It indeed matches quite well with either where the company is headquartered or where it has its production facilities.

Let us look at the search popularity for a few companies:

General Motors

 

Intel

 

Dell

Home Depot

 

Berkshire Hathway

 

Moreover, this also applies pretty well to other named entities:

Joe Biden, Senator Delaware

Bill Richardson, Senator New Mexico

I hadn't expected that for company names this would work so well. I was thinking that most of the search volume for corporations would be biased towards NY due to the stock markets. 

This is pretty awesome and I hope that there is a Google Insights API to be released soon. :-)

Del.icio.us + Google Custom Search = Delizzy

In my April 9, 2008 blog post on "Social Tagging and Vertical Search" I had talked about exactly this idea:

For quite some time now I wanted to build a mashup using the social tagging information from del.icio.us and combine it with Google's Custom Search engine. This is a neat way to build high quality vertical search engines. (For example we could use all the URLs under the tag "astronomy" to build an astronomy vertical).

Well, it was just waiting to happen. Delizzy (via Techcrunch), is a wonderfully simple idea but it is a super cool site built using del.icio.us and Google Custom Search. It lets you do what del.ico.us has surprisingly, still not yet provided -- A full text search over your bookmarked items. I believe this could be taken a step further and built into custom search engines or verticals for different topics. Imagine searching over only highly bookmarked sites for a particular topic like "AJAX" or "Ruby on Rails" or "Artificial Intelligence".

In any case Delizzy is pretty cool. Go check it out!