Social Media Research Blog

Geolocation and Twitter: Realtime meets Local

Recently, twitter announced that it was going to add support for geo annotations for tweets. The original announcement is on twitter blog.

One fascinating thing about Twitter is how the community is involved in shaping the way the product itself evolves. There are several examples of how this happens:

• For conversations, the community evolved method of @ replies.
• Retweets have become a great way to pass information along and attribute it to the person who provided this information.
• Hashtags are becoming even more popular, with almost a handful of hashtags landing up on the twitter homepage everyday.
• Location is'nt new either. Many users already provide the location information in their profiles. In addition several formats have evolved where L: is the most popular method for indicating the current location of a user. This was partly popularized by the Twittervision application.

By standardizing the support for geolocation in twitter, the service provides developers with several opportunities to build applications that are geographically relevant. One possibility for twitter is to capitalize on the wealth of user generated geodata:

1. Think realtime meets local: It would be great for instance, to know that there is a live concert that everyone in my town is talking about right now!
2. Local Advertising: This could be one direction that Twitter might be heading towards.
3. Hyperlocal News and Activism: Social media is defining new ways to share hyperlocal news and build community involvement. Twitter is being used by several NGOs, blood banks and Local Police as a means to bring awareness and build campaigns. Applications built using geolocation information can accelerate involvement and social good at a local level.

BTW, here is an interesting stat about twitter users: From a large collection of twitter profile pages I find that roughly 60% of them have some URL associated with the profile. AND of those that have a profile URL, 92% already declare some location for themselves (irrespective of whether the location can be resolved correctly or not). While many worry about the fact that our location privacy is being lost forever; like any technology, it will ultimately be up to the community to come to a consensus on what is acceptable. For most part, I think that users are happy to share their location at some level of general granularity, in exchange of some (perhaps perceived) social value (meeting friends, finding local hangouts or for simply being a bit exhibitionist about what you are currently upto!).

SXSW Panel on Collaborative filters

I am at SXSW this weekend and wow what a conference this has been! Met a lot of cool people and been to a bunch of happening parties! I plan to post a few updates about the conference. In the mean time here is a quick post from notes I wrote during the panel discussion on Collaborative Filters.

Anton Kast from Digg.com gave a brief history of collaborative filtering from the research literature covering tapestry (from parc) grouplens (UMN) and a few other projects mostly from 1990s and draws comparisons to current systems used in spam filters, pagerank , tagging systems and applications in facebook ads etc. The talk wasnt very technical and meant to be a quick overview for the audience that might not be familiar with the research background. Kast described recommendation as the output of collaborative filtering that is personalized for a particular user in the system. For example  amazon (people who bought this also bought..), behavioral ad targets, google news etc. I wish Kast had spent more time discussing the last slide which covered practical problems: sparsity problem in recommendation systems, early rater problem/cold start, gray sheep (small group very popular).

Erik Frey from last.fm shared some interesting stats -- 25m unique users/month. The types of relations that last.fm derives: songs-users, users-users. He also shared some insights on what were described as lean forward and lean backward mode in music recommendation systems. The lean forward mode is when a user is feeling a little experimental and would like to discover new stuff. The lean backwards mode is when the emphasis is on continuity and the desire to listen to something familiar (i.e. radio experience). The two main data sources for last.fm are : scrobbles, social info like tags. Frey suggests that tags are quite useful to explain recos

The next speaker was the cto of baynote -- a recommendation system that is provided as service using javascripts that enables clients to customize their website in real time based on user behavior. They also apply it for reordering the search results.

Finally, Jon Sanders from Netflix gave a very good talk covering the history of netflix and its evolution. Today netflix has about 2 billion ratings and gets about 2 M ratings per day. 60% of movies watched are via personalized recos -- amazing!

Following were the steps in evolution of netflix:

step 1 At first netflix was a editorially managed site and the first evolution was that of adding a rating widget.

step 2 score and sort all movies using rating alone (top k list)

step 3 movie similarity by k-nearest

step 4 interest based recos (same actor director  etc)

step 5 ask ( get users to rate genre)

step 6 ask others (top 10 lists from users)

step 7 explain why (because you enjoyed)

step 8 $1M prize

step 9 develop a unique experience and customized website for each customer

future: streaming video specific recommendations, incorporating implicit and explicit metadata, discovery focus, let people drive and not be led.

ICWSM 09 Dataset Deadline Reminder

A quick note: So far we have had about 185 Dataset downloads!!

The ICWSM-09 Data Challenge Workshop will be held May 20, 2009 in San Jose, California in conjunction with the Third AAAI International Conference on Weblogs and Social Media. Submissions for the workshop are due March 1, 2009. Please see http://www.icwsm.org/2009/data/ for complete submission details.

The ICWSM-09 Spinn3r blog dataset is a collection of 44 million blog posts made between August 1st and October 1st, 2008, and collected by Spinn3r.com.  This dataset is freely available to researchers under a liberal data usage agreement.

Authors are invited to submit papers to a data challenge workshop to be held on the last day of ICWSM-09. This workshop will feature research papers as well as a wide-ranging discussion of data issues facing the social media research community. Good research topics might include:

* link analysis;
* social network extraction;

* clustering and topic identification;
* tracing the evolution of news;
* blog search and filtering;
* psychological, sociological, ethnographic, or personality-based studies;
* analysis of influence among bloggers;
* blog summarization and discourse analysis.

You should feel free to explore any aspect of the data that you feel would be of interest to the ICWSM community.  An award will be presented at ICWSM for the best paper using the dataset.

Papers may be submitted online at http://www.easychair.org/conferences/?conf=icwsm09dcw.

Submissions may be up to 8 pages in length, must be in PDF format, and must follow the ICWSM formatting guidelines.

Best regards,

Ian Soboroff (NIST)

Akshay Java (Live Labs, Microsoft)

ICWSM-09 Data Chairs

Geolocating blogs

A quick note:

Clay Finkand colleagues from UMBC have a new paper on "Geolocating Blogs From Their Textual Content". From the abstract:

"Mashups showing the geographic location of the authors of social media content are popular. They generally depend on the authors reporting their own location. For blogs, auto-mated geolocation strategies using IP address and domain name are not adequate for determining an author’s location. Instead, we detail textual geolocation techniques suitable for tagging social media data, facilitating development of geo-graphic mashups and spatial reasoning tools."

As noted in this paper, trying to geolocate blogs from IP addresses or domain names does not work well due to many blogs mapping to various hosting services. To locate the named entities that refer to a potential location, the authors use GeoNames (This is a really great, free resource for those working with geographical analysis). The location of the blogger is disambiguated by clustering occurances of different geographical named entities in a region. From the paper:

For example, if “Maryland” has already been disambiguated as the U.S. state of Maryland, then the text “Laurel, Maryland” leads us to disambiguate “Laurel” as Laurel, Maryland.

Location is an important element in social media analysis and any results (for example, sentiment analysis or trends) that could be pivoted on this dimension are especially useful in providing an insight into what people in different regions think about a particular topic.

In context of this work, there are a few more related papers (in addition to references in this paper) one might want to check out. Although these papers dont address the same issue, they are interesting from the perspective of location analysis:

• Recent study of "Iran's online public" by Harvard researchers.
• "Spatial variation in search queries" by Backstrom et al.
• A study of blog commentsof persian bloggers by Noor Ali Hasan
• In our work with twitter analysis, we did some basic mapping of location names mentioned in user profiles to lat/long coordinates using Yahoo API.

Community-Driven Moderation in Twitter

Twitter played an important role during the mumbai terrorist attacks. The community that formed around the #mumbai tag, actively discussed how they felt and provided live, uptodate information filtered from different news sources. This was a tipping point for twitter that transforms it from a social tool to a news source. The most important factor here was that the dynamic community that emerged around the #mumbai tag was not constrained to discuss one single topic. Something that News sources struggled with as the developments happened so quickly in this critical event. Most news sources need to stay on a story for at least a few minutes before they can move on to another story.

As #mumbai became popular, trolls showed up on twitter and posted messages with the hashtag only to draw attention to their website or link to other sources of little or no relevance whatsoever to the current crisis. This made me wonder, what if anything could be done in Microblogging environments to restrict users from misusing hashtags. A quick glance over at the current use of #mumbai shows a number of such sources that should probably moderated. I am refraining from naming any particular users but this is something that could be better dealt with by the community as a whole.

One suggestion I have is to create a new hashtag like #ban followed by the user name and the hashtag for which the user should be banned. Once a particular user is #ban tagged by several people from the community -- Twitter can automatically ban the user from further posting on the specific topic. This would be a simple and effective means for a community driven moderation in twitter.

Book Review: Tribes by Seth Godin

I was quite impressed by Seth Godin's small book called "The Dip" so I decided to get a copy of Tribes. The book describes a tribe as: any group of people, large or small, who are connected to one another, a leader and an idea. It almost sounds like the definition of a tribe is identical to that of a community in the social network literature. The powerful idea in this book is that anyone and yes, YOU! can lead a tribe. Godin describes leaders as heretics who challenge the status quo in organizations. The internet and social media tools today make it far easier for all of us to be leaders and inspire others to do the same in their own workplace, organization, school or non profit. Blogs, social networking sites, self publishing tools and twitter are only a few examples of new tools that are at your disposal. This book isnt a "how-to" manual or a dummies guide to leadership but it presents plenty of examples of how individuals have brought about change. In all the examples one thing that is striking is how true leadership is a lot about communication. Leadership requires us to listen and to facilitate communication in a tribe. Overall it was an average read and I must say the book fell short of my expectations. One problem with the style of writing is that it was more like a series of blog posts on this topic rather than a book. The sections are just too hotchpotch and lack depth. Perhaps it would have served much better as a blog since there were several paragraphs and references where individuals and examples seemed to have been mentioned in passing without much detail. I think the author assumed a lot from the reader in terms of familiarity with both his previous works and the references made. Not having internet connectivity while reading this book was equally frustrating since I could not research further on these examples. If you are already a fan of Godin's work you may enjoy it.

ICWSM 2009 Data Challenge

A quick note:

We have worked with Spinn3r to create a dataset for this year's ICWSM conference. More information is available on the site. We look forward to active participation by the social media research community and hope that you would take advantage of this fabulous resource. Also stay tuned for another awesome dataset to be released shortly.

Continuing the ICWSM tradition, ICWSM 2009 is making a dataset available to researchers in the blog and social media fields. We invite you to download the dataset, explore it, learn something interesting about it, and submit a paper about it to ICWSM 2009.

Good research topics might include...

• link analysis
• social network extraction
• tracing the evolution of news
• blog search and filtering
• psychological, socialogical, ethnographic, or personality-based studies
• analysis of influence among bloggers
• blog summarization and discource analysis

But you should feel free to explore any aspect of the data that you feel would be of interest to the ICWSM community.

Instructions for submitting papers to ICWSM may be found here. When submitting your paper, indicate that it makes use of the dataset. Dataset papers will be reviewed for the main conference, and additionally for presentation at the data challenge workshop to take place on May 20th, 2009 (the last day of the conference). While we anticipate that several dataset papers may appear in the main conference, the data challenge workshop will provide an opportunity for in-depth discussion of the dataset in a more focused forum.

Hypertext 2009

via Pranam Kolari

HYPERTEXT 2009, will be held at Torino, Italy between June 29th and July 1st next year. Perhaps, the evolution of HYPERTEXT conference reflects the increasing scope and influence of the Web over the past decade.

The Web, the Semantic Web, the Web 2.0, and Social Networks are all manifestations of the success of the link. The Hypertext Conference provides the forum for all research concerning links: their semantics, their presentation, the applications they have been put to, the knowledge that can be derived from their analysis, and their effect on society.

Papers due Feb 2, 2009. Event has been added to the Social Media Events Calendar.

Live Labs Social Streams

Very cool! Live Labs releases the Political Streams project. It is a lense into the people and places that are receiving a lot of attention on blogs and news.

You can read the post snippet and see neat graphs of how the attention changes over time:

Also browse through the related names and places as you explore different topics and blog posts/news stories. Furthermore there are profiles of people and places in news that are generated using Freebase.

 

This is really cool! If you looked at search trends, many of the search terms are in fact about places and people in News. Users go to a search engine to find out both why they are in news? and often who the person is? Social streams (right now, for political streams) let you explore this in a real time, interactive manner. 

Nice!

Mean Time to Update Feeds

tIn the State of the Blogosphere report posted by Technorati recently, they describe how the top bloggers update their blogs more frequently than the rest of the population. Given the data I have been gathering on the Bloglines subscriptions, I was curious to see what how most subscribed blogs behaved? For the top 100K feeds ranked by the number of subscribers, I fetched their RSS/ATOM files (some failed, some were filtered due to few posts etc). Using this handy PERL module, I parsed all the pubDates (sorry, RSS only for now) and converted them to their corresponding Unix timestamps. Now for each of these feeds, the Mean Time to Update (ie the avg of the number of days between consequent posts) can be obtained. Interestingly, some fraction of the posts were updated after they had been published (hence were out-of-sequence in the RSS feed). For this analysis, I do not consider these updates -- it would be interesting to dig further into it.

Following graph F(x) is defined as the proportion of X values less than or equal to x. Thus you can observe that for the top 10K feeds 40% of the feeds updated in a day or less. For the rest of the feeds it was 30%. I believe as we go lower in the ranking, this would keep decreasing even further. Interestingly, overall, for the highly subscribed feeds, almost 60% update at least once a week. 

For a feed reader keeping an index fresh and up-to-date is then a function of how many users subscribe to a feed, how often the feed is updated, when was the last time the feed was requested (by any user in the system) and perhaps even when the last ping was registered for a particular feed in the ping stream. I believe in Google reader's case, clicking "show details" displays the last time when the feed was refreshed -- which I have never found to be over 3 hours for any feed that I subscribe to.

ACK: The above plot was made in MATLAB using this script for cdfplot.