Social Media Research Blog

One Million Feed Strong

A couple of months back, I had mentioned about a recrawl of bloglines (publicly listed) subscriptions that I was running. Here is an excerpt from my previous post:

Bloglines is any indicator there are a lot more people using feed readers. A study conducted in 2005 found that there were about 32,415 public subscribers and their feeds accounted for 1,059,140 public feed subscriptions. In 2006, we conducted a similar study and found that there were 82,428 users and 1,833,913 listed feeds they subscribed to. This dataset was also used to build the site "Feeds That Matter". In a long overdue attempt to revise the list, I ran another crawl of the Bloglines public directories. This time around, I found that there were about 209,125 users who subscribed to 1,875,027 feeds.

From this new crawl that I have, the roughly 200K public users have about 4,289,988 distinct items in their collective OPML files. These correspond to about roughly over a Million distinct feeds that have at least one subscriber. The following graph shows the distribution of these feed subscriptions, showing the hallmark power-law distribution.

According to Technorati there are 133 Million blogs that they are tracking with about 1.5 Million that have updated in the last 7 days. This is HUGE!

In comparison, it looks like there are about 1 Million feeds that really matter (in the sense that people actually subscribe to them). A few points:

• Here is a list of top feeds from this collection.
• Roughly 11% of all feeds subscribed are served via feedburner. I am a bit surprised by this as I expected this fraction to be much higher.
• The number of feeds that have at least one subscriber has remained relatively stable since last year as far as bloglines is any indicator.

Looking at this data, I wondered what is the best way to index millions of feeds? Crawling is a very heavy duty job requiring huge bandwidth and disk writes. It isn't that easy to implement and I have in the past written my own crawlers for a few projects, indexing tens of thousands of feeds. But to scale it up much further you need a good deal of resources -- which doesn't come cheap either. Ping servers were supposed to be the critical component in the blogosphere infrastructure, unfortunately they are inundated with spam right now. That is why tools like Tailrank's Spinn3r are a great alternative for small startups who want to index blogs. They take the heavy lifting off your shoulders and allow you to build and focus on the main product.

There are a number of research challenges that come up with crawling and ensuring that the index is fresh. Recently, Greg points to a paper at WWW that is pretty remarkable. They crawled 6 Billion pages on a commodity server. (This paper also won the best paper award). I believe that crawling feeds and blog data is quite different from crawling the Web. Ensuring freshness of the index means indexing new posts within minutes of publishing. Given the inundation of ping servers with spam, one option is to poll the feeds and the other approach is to ignore all pings other than those from the feeds of interest.

From Greg's post:

On the question of what content is useful, both in terms of crawling and recrawling, I always find myself wondering how much we should care about pages that never get visited by anyone. Shouldn't our focus in broadening a crawl beyond 6B pages and in recrawling pages in our index depend on how much users seem to be interested in the new content?

This made me think -- how about feeds? If there are only 1Million feeds that are subscribed, perhaps most of the resources should be dedicated to keeping this core set as fresh as possible. In context of this question there is some recent work by Richard Sia on this topic: Efficient Monitoring Algorithm for Fast News Alert, with Junghoo Cho and Hyun-Kyu Cho, in IEEE Transaction of Knowledge and Data Engineering (PDF). They model the posting patterns of the RSS feeds and optimize multiple fetches to minimize the delay in indexing.

But one question that comes to my mind is that of the tradeoffs between index freshness and quality. One could index very few feeds and ensure freshness at the cost of quality. On the other hand one could index a large number of feeds and increase the delays in indexing. What is the optimum balance between these two? How do you incorporate the number of users who subscribe to the feed in the (re-)crawl prioritization algorithm? Can recent link activity prioritize recrawl of feeds?

List Crazy Diggers

[UPDATE 08/22/2008, 12:10 AM]

I wonder why diggers are just list crazy?

At the time of writing this post (yesterday), here are the 'lists' from the top 100 diggs on the digg front page -- almost 16 of them! What this means is that on every page there is at least one 'list' type of an item. (And I have noticed this trend for some time now)

1. 5 Reasons to Write Down Everything You Eat For a Week

2. 6 Worst Movie Posters Of 2008

3. 15 Fugly Faces of George W Bush: Commander in Grief [PICS]

4. 10 Movie Trailers That Were Better Than The Actual Movie

5. 10 Coolest Chuck Norris T Shirts Ever!

6. 7 Principles Of Clean And Optimized CSS Code

7. 15 Images You Won't Believe Aren't Photoshopped

8. 10 Great Webcomics You Should NOT Share With Your Kids

9. The Top 10 Rookies That Will Impact the NFL This Season

10. The 5 Most Baffling Sex Scenes in the History of Fanfiction

11. The Top Five Weirdest Olympic Events!

12. 9 Linux Myth Debunked

13. Stephen Colbert's Top 7 Drug Moments

14. The Top 7 Girl Gamer Stereotypes

15. Top 10 Songs Over 10 Minutes Long

16. The 10 Skimpiest Superheroine Costumes

Lists can be entertaining and fun. But I wonder why we see lists enjoying such high popularity in digg? From a cursory glance, the comment activity on 'lists' is comparable to any other item. So my guess was simply that the 'Top Diggers' have a greater inclination towards lists. 

[UPDATE 08/22/2008, 12:10 AM]

To understand what was going on here, I turned to the expert, Dr. Kristina Lerman, who has extensively studied the Digg Community. Following is an excerpt from her response to my question about the popularity of lists on Digg:

"......did you notice how the amount of communicated information is shrinking? First we had Twitter, that advertised 128 chars or less in a message. Rather than being an inconvenience, other services followed the trend: "videos in 5s or less", or other bite-sized chunks of information. Lists are part of that trend, but not only do they communicate information in a convenient bite-sized chunk, but they also rank them - another passion of the current age. It seems that people are really averse to expressing thoughts in long paragraphs." [Text highlighted by me]

Digg is essentially a news site, but the audience probably loves the quick, bite-size and neatly ranked information that such posts provide. Even on traditional news reports and TV shows the "top 10"   whatever-odd-ball-potpourri land up being a part of the reporting.

Here are links to some of Dr. Kristina Lerman's papers on digg and a video of her talk (@Google) on Social Information Processing.

Kristina Lerman (2007), User Participation in Social Media: Digg Study , in Proceedings of the International Workshop on Social Media Analysis (SMA07), Conference on Web Intelligence and Intelligent Agent Technology [pdf] [bibtex]

Kristina Lerman (2007), Social Networks and Social Information Filtering on Digg, in Proceedings of Int. Conf. on Weblogs and Social Media, Boulder, CO,USA (poster). Extended version available on arXiv (cs.HC/0612046)

Thanks Kristina for your sharing your thoughts on this new trend!

Wikimatix: Wikify and Disqus Hot Search Keywords

Introducing WIkimatix.com: A mashup for wikifying Google Trend's top search keywords. Moreover you can "disqus" and comment on recent hot topics being searched for right now!

Yesterday, while taking a break from writing my dissertation I was looking at the top search terms over at Google Trends. Few things immediately came to my mind:

1. I am a Wikipedia and News junkie -- no surprise there! Sometimes, I do not know a whole lot about the top search trerms and have to look up the wikipedia entry to refresh my mind about the topic or person. BTW, have you noticed that so many of the top terms are named entities? I also spoke about this in my earlier post on Intrestingness and Feshness in Search.
2. Google trends does not have an ability to discuss. I wanted something where I could comment on the terms in a digg like fashion? Perhaps talk to everyone about what I think of this whole "Big foot thing in the news". (Tip of the hat to the awesome widgets powered by Disqus)
3.  Finally, I would like to know exactly why something is in the News or has made it to a top 100 list. I just want a one or two line snippet/summary and a pointer to the most authoritative blog post/news piece about this keyword. While Wikimatix currently does not provide this -- it is something that can be easily extended.

Well, this was a quick hack ( ~150-200 lines of code; ~2 hours) and is meant to be a prototype/proof of concept idea. I think there can be a lot one can do with the idea of capturing the buzz using query logs and having discussions around the search terms and the latest news.

Note: Wikimatix updates itself every two hours. The disqus pages are associated with the individual search term and thus can be accessed even after the term is no longer buzzy.

Some possible improvements that might be good:

• Group related search terms like "big foot", "big foot discovery" etc
• Name disambiguation can be a problem. Sometimes the right person's Wikipedia entry is hard to determine without the context of the story.
• Simple heuristics might work rather well to identify the top categories that are in the news. For instance, today it is all about "Big foot", "Olympics" and a few other major events.
• For search terms, there might be a good video, photograph associated with it. It would be cool to display that as well.

Services That Attract Most Comments on FriendFeed

Indexing about 400,000 posts and over 650,000 comments from about 70,000 users on FriendFeed gives some interesting data. There are about 45 different services that FriendFeed connects to at this point of time. First, I wanted to see what were the absolute volume of posts were from each of these services:

Next, I pulled up the number of comments that every service draws during the time period of my index (which was built by fetching the most recent posts and comments from the API).

Note that this is not normalized for multiple updates by the same user. A few interesting points:

• Large volumes of updates are from URL sharing and bookmarking tools like del.icio.us, Google Reader and StumbleUpon
• Most Comments are attracted by posts that were made directly via FriendFeed.
• del.icio.us does attract an equally large volume of comments.
• While Twitter is not far behind and receives a significant share.

Finally, for the services that attract the most comments, I have calculated the average number of comments per post:

The above chart says something important. On an average, original content like what one would find on blogs, still attracts the most comments. Followed by conversation focussed services like FriendFeed and Twitter. Finally, images posted on Flickr get a lot of attention too. Bookmark sharing sites are now lower in the ladder. See some trend here?

The attention and appeal one receives in social networks is a function of the original content you produce and the value, insights you bring to the table when you share new stuff.

Six Degrees of Bret Taylor

I have been collecting a crawl starting from one of FriendFeed's founders Bret Taylor's subscriptions. In some of the previous posts, I have shared an initial analysis of this crawl. After the crawl came to a completion, I have found about 70,535 users who are either directly or indirectly connected to Bret. This leads me to believe that the true user population of FriendFeed might be around 75-80,000 range (also similar to the numbers found by crawling the public users). I think FriendFeed is still mostly an early adopter tool and hasn't gone as main stream yet.

Well, if you are curious what your bret number (six degrees of Bret Taylor) is, check out this zoho sheet and search for your name by clicking the Auto Filter.

Feed Readership Growth Statistics

Arite! This is the post I lost earlier thanks to Typepad. There is something about a forced re-write. It never sounds as good as the original post. But here it goes any way:

I wanted to talk about how as social media enthusiasts we always assume that people understand what we are going all nutts about. Take for instance a simple question I have been asking my friends -- " Do you know what RSS is?". To my surprise, many of them did not use RSS or any feed reader. Upon showing them what RSS can do one friend commented that  she simply likes to "browse the Web" and RSS somehow boxes her.

Still, if Bloglines is any indicator there are a lot more people using feed readers. A study conducted in 2005 found that there were about 32,415 public subscribers and their feeds accounted for 1,059,140 public feed subscriptions. In 2006, we conducted a similar study and found that there were 82,428 users and 1,833,913 listed feeds they subscribed to. This dataset was also used to build the site "Feeds That Matter". In a long overdue attempt to revise the list, I ran another crawl of the Bloglines public directories. This time around, I found that there were about 209,125 users who subscribed to 1,875,027 feeds. Let us look at this data:
 
We find a significant increase in the number of people using Bloglines. There is a rise in popularity of feed readers, if Bloglines is any significant indicator. Next we look at the number of feeds:

Interestingly, even though the number of users significantly rose, the number of feeds are about constant.

Seen another way, here is a scatter plot of the above data (each dot corresponds to 2005, 2006 and 2008 respectively):

In my view, even though there may be 70 Million plus blogs, the ones that most people are willing to devote significant commitment in terms of adding them to their feed readers is actually around a Million or Two. I also think that the popularity of Twitter and FriendFeed around 2006-2007 time frame has changed how we find new information. Social filtering might work better for long tail content. This is only a hypothesis. It could just be an artifact of Bloglines. Moreover, Google Reader has a greater market share and perhaps FeedBurner has a more accurate measure of growth in feed readership.

What is the Dunbar's Number for Social Networks?

Many folks are really excited about FriendFeed. Personally, I have found that there are a lot more comments when something gets posted on FriendFeed. Recently Yuval Atzmon's User21 blog released a list of most followed users on FriendFeed. Since I too had a crawl of FriendFeed running in much the same way as Yuval, I decided to look at the complementary question: "How many users do people follow on FriendFeed"? While the crawl is not yet complete (and complete statistics would have to wait), the numbers are really striking! Some users follow more than a 1000 "friends":

sthayden	3190
scobleizer	3087
juliomedina	2760
thomashawk	2557
jasoncalacanis	2447
theillife	2045
mrsth	1961
pookakoo	1814
czarphanguye	1736
brynyoungblut	1716
eposter	1562
susangrisantiguitarist	1550

I find this really amazing. Unlike Twitter, FriendFeed posts are accompanied with longer conversations so it can be more involved. I can barely keep up with all the information flying past me everywhere right now! I guess, 1500+ "friends" would be way too much for me!

Sociologists often talk about the Dunbar's Number which

is the supposed cognitive limit to the number of individuals with whom any one person can maintain stable social relationships.

In human contact network the Dunbar's number is said to be around 150. It might as well be the case that social tools and especially, microblogging is pushing this limit further. Studies on Twitter, Livejournal and other social networking sites seem to support this observation. I wonder then: what would be the Dunbar's number on social networks? 300? 500??? Any guesses? Perhaps some comparison across all the published papers that have studied different social networks might have some clues.

[BTW, I am akshayjava on FriendFeed]

Google Calendar Feature Requests

I have started a Google Calendar to keep track of events and conferences in Social Media. Some of you may have already subscribed to it. However there does not seem to be any way for me to tell exactly how many people are using it!

Feature Request #1: Show Number of Subscribers for a Calendar This would certainly be quite a useful feature to have and as it turns out I am not the first person to request for this. 

Feature Request #2: Allow Tagging; Sharing Calendars a User has Added Calendar as a shared resource for planning and organizing events in a community is an important tool. However, the calendar is still not as social as it can be! You can easily find new calendars to add. But what about tags? How about sharing? I would love to be able to create a tag cloud of events and a public list of calendars I have added.

Feature Request #3: Social Event Notification If a user has made his or her events public, then why not show that users friends an update on the event she plans to attend? I think that Dopplr does this at some level but given that Google Calendar is a good place to consolidate all the events and schedules and GMail is our universal contact list -- why not combine it to make it more social?

Anyways, these are just a few quick thoughts I had about Google Calendar. It is a great tool and has made my life much easier. To be fair, I have never used outlook so perhaps my view may be a bit skewed.

Lucky Number 2.0

I wanted to share a quick book recommendation. I picked up Sarah Lacy's recent book "Once You're Lucky. Twice You're Good." for a quick read during my flight. It is a fantastic account of Web 2.0 and profiles successful social media startups, Silicon Valley entrepreneurs, Venture capitalists and angel investors. The book takes you down a nostalgic road starting with the dotcom bubble all the way to the current buzz around social media and Web 2.0 (and why this is not a bubble!). It is an inspiring story and one that every entrepreneur (/wannabe/fanboy) would love to read. ;-) But I recommend it mostly due to the nuggets of wisdom in this book. If you are an entrepreneur, there is much to learn from the experiences and insights of those who have spent years building successful companies. Sarah does a great job at taking us through their voyage and writes in a unique and refreshing style of her own. Indeed, an enjoyable book to read!

The Cold Start Problem in Social Media

The Cold Start Problem in Social Media

Here is a classic cold start problem:

• "Social Tools" can only be social if there are enough people on it.
• And any social site is only as attractive as the number of friends you have on it.
• The social site is only useful if there are enough people contributing to it. (be it annotating images, links or adding reviews)

The question is how do you get enough users to adopt a tool and build sufficient traction around it such that it attracts more users? I dont have all the answers and perhaps entrepreneurs and folks in startups are more knowledgeable about this than I am. But this is a question I have been pondering about for some time. Most of these points might be fairly obvious, but here are a few thoughts I'd like to share:

• Above all, build something cool!
• Realize that in many systems, 1% users acting as contributors is all it takes! (see Clay Shirky's book "Here Comes Everybody" for more on this). Ensure that the reward mechanism is automatically built into the site. These 1% of all users are not the ones that are motivated by money. They use your tools because they enjoy it or it solves some real problem they have been facing. I have seen some sites trying to "pay" users to add data to "seed" their site. For example, check out some of the high paying HITS on Mechanical Turk. In my opinion, this is like throwing money out of the window. Completely bogus way to jump start your site!
• Provide APIs: One of the key factors that contributed to the success of Twitter was that they had a neat API that developers immediately adopted and had fun building cool toys. These 3rd party tools in turn make it easy for users to contribute and engage with your site, thus breaking the cold start problem. For example, even though not developed by Twitter, the plethora of third party twitter client make it easy to easily update your Tweets. 
• Try to "seed" your site with datasets curated by web crawls, APIs, external databases or using the tools yourself. For example, if you are building a site that uses geotagging, you might consider using sites like geonames.org or if your site is around movies -- use IMDB or Amazon data to seed it.
• Make sure stuff is findable and socially visible. Make it easy for your users to find the data they really care about and they will be willing to annotate. Moreover, make sure that it is easy for users to share what they annotate with their friends. The beauty of Facebook is the news feeds. People like to know what their friends are upto. On Twitter, I want to know what my friends are saying and be able to have conversations with them - without that twitter is just a chat room.
• Dont ever SPAM! Every week I have a bunch of emails coming from some random sites that a friend of mine once joined. Make sure that in your invitation email you include an option to not receive any requests from your site in the future. I am amazed when I dont see this option at all. If a user does not wish to join a network, please dont keep sending them emails requesting them to join every time one of their friends sign up! Also, I am a sucker for alpha/beta testing for any new social media/social network site. Sometimes, I try even the ones that eventually land up spamming everyone on your email/IM. As a RULE -- never spam your potential users! That is the best way to piss them off even before they join it.
• Listen and iterate rapidly. Your alpha/beta users are the most important. Listen to what they have to say. Also, if you cannot convince your friends and family to use the tool -- why would anyone else bother?

The cold start problem has been studied in computer science, particularly for recommendation systems**. A good place to start is the paper:

Methods and metrics for cold-start recommendations  Schein, r.I.; Popescul, A.; Ungar, L.H.; Pennock, D.M. [Link]

I am quite interested in knowing how startups have approached this problem in real situations and particularly, if there is any analytical data available to show what worked and what did not? I guess this might be information that few would be willing to share so openly.

** on a related note: The blog "Duke Listens" is an excellent source for more on recommendation systems. Also check out the recent post on cold start problem.