Social Media Research Blog

Wikimatix: Wikify and Disqus Hot Search Keywords

Introducing WIkimatix.com: A mashup for wikifying Google Trend's top search keywords. Moreover you can "disqus" and comment on recent hot topics being searched for right now!

Yesterday, while taking a break from writing my dissertation I was looking at the top search terms over at Google Trends. Few things immediately came to my mind:

1. I am a Wikipedia and News junkie -- no surprise there! Sometimes, I do not know a whole lot about the top search trerms and have to look up the wikipedia entry to refresh my mind about the topic or person. BTW, have you noticed that so many of the top terms are named entities? I also spoke about this in my earlier post on Intrestingness and Feshness in Search.
2. Google trends does not have an ability to discuss. I wanted something where I could comment on the terms in a digg like fashion? Perhaps talk to everyone about what I think of this whole "Big foot thing in the news". (Tip of the hat to the awesome widgets powered by Disqus)
3.  Finally, I would like to know exactly why something is in the News or has made it to a top 100 list. I just want a one or two line snippet/summary and a pointer to the most authoritative blog post/news piece about this keyword. While Wikimatix currently does not provide this -- it is something that can be easily extended.

Well, this was a quick hack ( ~150-200 lines of code; ~2 hours) and is meant to be a prototype/proof of concept idea. I think there can be a lot one can do with the idea of capturing the buzz using query logs and having discussions around the search terms and the latest news.

Note: Wikimatix updates itself every two hours. The disqus pages are associated with the individual search term and thus can be accessed even after the term is no longer buzzy.

Some possible improvements that might be good:

• Group related search terms like "big foot", "big foot discovery" etc
• Name disambiguation can be a problem. Sometimes the right person's Wikipedia entry is hard to determine without the context of the story.
• Simple heuristics might work rather well to identify the top categories that are in the news. For instance, today it is all about "Big foot", "Olympics" and a few other major events.
• For search terms, there might be a good video, photograph associated with it. It would be cool to display that as well.

Evri: Search Less, Understand More

I just received the beta invite to Evri.com (Yaaayy!). It is a really cool site that aims to help people find information. Right now they just have a browse interface. You can see what are the top concepts and named entities (primarily from News sources) and navigate through semantically related terms. The main idea behind their approach is that you can construct the graph of all the concepts and entities by analyzing the text. Here is an example of the top names in the news. Clicking the terms (from the graph) "Barack Obama" and "Ralph Nader" for example, would pull up all the stories related to recent controversies.

One can browse through the graph or the popular terms. I checked out what they found on Obama. Here is a snapshot on the left. I think that a really neat trick that Evri is using is the idea that working on sentence level semantics can provide sufficient meaning to help organize information. Constructing a complete parse tree that is both syntactically and semantically accurate is a difficult problem. There are many vagaries of natural language text that make this challenging. Evri, at least for now, bypasses some of these problems by organizing information around simple questions like "what is Obama doing?" which can have easy to identify clues directly accessible from the text (critisizing, leading, denying, facing....). Similarly for other entities like organizations one can ask "What is happening with Yahoo?" (bidding, reject, acquire, etc.). 

This is a fascinating approach to organizing information and I think that Evri has a great potential. Lets think about it for a minute. One of my favorite passtime is to go to Wikipedia, pull up a random article and then browse through related articles. It is this serendipity and the feeling of chance discovery of something interesting that is so compelling about Evri.

Evri also reminded me about the way I had hoped to implement SemNews, a semantic search engine, that analyzed RSS snippets of News articles and processed it through OntoSem, an ontological semantics based Natural Langugage Processing system. Once the semantics/meaning representations were extracted, I would store the meanings in an OWL store so that RDQL queries could be performed to find relevant news items. I believe that the way we can accomplish Dr. Tim Berners-Lee's vision of Semantic Web is by advancing both information extraction (web scraping, entity annotation etc) and NLP techniques that would automatically annotate text and make it available in machine readable format.

Although, the founder claims that they are not a search engine, they surely join the group of NLP-based startups like Powerset and Hakia. Another powerful tool is Freebase which uses primarily Wikipedia as its source of information. Finally, it is also worth mentioning that Kosmix is yet another startup that aims to "Organize the Web so that you can explore, learn and discover".

The next obvious question that comes up is regarding the monetization and business model of these startups. Ofcourse, the story goes... the information is more focussed so ads would be more relevant... and no surprise that is indeed so TRUE. Just check out some of the advertisements on Evri. On the left, is a screenshot of an advertisement on Barack Obama's info page. But I think there is an opportunity beyond simply relevant advertising!

Many companies have huge websites with lots of information -- some organized and most not quite as much. If you wanted to ensure that your customers are able to get to the exact information they need -- Evri like approach can be ideal to help them browse through the various facets to get to what they really need. The applications to Enterprises and Enterprise search can be another monetization platform for Evri. 

Finally, IMHO, some hurdles that Evri faces could be dealing with noisy text, especially with Social Media. Many approaches that rely on linguistic or gramatical correctness of sentences simply fail miserably when dealing with social media content. The second problem might be esuring coverage of information. Right now, it seems to me like the News soruces Evri relies on are primarily US centric. As they aim to capture more audience outside US as well, they would have to concentrate on foreign languages, disambiguating named entities and location names. These are all interesting research problems and fun stuff to work on!

SemanticHacker Challenge and Wikipedia Widgets

When writing a blog post, we create links to Wikipedia articles because it is an easy and quick way to provide information to readers, without having to go into too many details. It is not always possible to find, link or even know about all the articles that might be related to the post's topic.

Zareen Syed, one of my colleagues, recently presented a paper at ICWSM on "Wikipedia as an Ontology for Describing Documents". This paper talks about how Wikipedia articles can be used to associate the concepts with a given document. Over at SemanticHacker a similar approach is taken to find the "Simplified Semantic Signatures" for a document.  TextWise, is offering upto a Million Dollars in funding to any idea that can

Turn on the Power of Semantics

I liked their demo, API and tools they provide. Inspired by the effectiveness of Wikipedia in describing documents and my desire to play with widgets, I decided to mash these two goals together. If you look on the right column of this blog, you would find a small widget that displays the related Wikipedia entries for a given page/post and is powered by the SemanticHacker API. This was a quick hack and the system can be improved by

• focusing on the post content alone (right now I just pass the current page URL).
• Adding more features, like identifying people, places etc and highlighting them differently

While I dont think this by itself might qualify for a $1 Million challenge, I think there might be something interesting here. I have a couple of ideas around this and scant time to work on it. Finally, in true Web 2.0 spirit -- not sure what, if any, is the business model (I just built this for kicks; semantic hacker is looking for a bizplan from participants).

I dont think that the widget itself is ready for prime time -- it was an evening hack. But if you would like to play with it leave me a note in the comments and I shall provide a link to you so you can install it on your blog.

Wikipedia and World Languages

Wikipedia is often referred to as the "Sum of all human knowledge". While Wikipedia is a very good resource for English articles, I recently wondered if how well-represented other languages were. On Wikipedia I pulled up a list of most spoken languages in the World. Next, I checked the number of articles and users for each of the top 20 spoken languages in the world.

The following graph shows the languages ordered by the ratio of number of articles to the number of speakers.

It is interesting to find that the European languages (French, German, English Italian, Portuguese, Spanish) have a large number of articles. Japanese is also well represented in terms of the number of articles.

Next graph shows Languages ordered by the Number of Users on Wikipedia to the Number of speakers.

A few points that are striking from these graphs:

• English and European languages speakers are quite prolific on Wikipedia. This is not surprising since Wikipedia also receives most of its traffic from English subdomain (See note below).
• Chinese, despite being such a widely spoken language is clearly way behind in terms of Wikipedia articles as well as users. This is probably because of the Chinese government's policy of blocking Wikipedia.
• Surprisingly, Arabic also has very few articles and users. I am not sure why there is very little participation from Arabic community in Wikipedia.
• Indian languages (Like Hindi, Marathi, Telugu, Bengali and Punjabi) are among the top 20 world languages, in terms of its speakers. However, since many Indians often communicate online in English, the number of articles contributed in these native languages is not very high.

Also From the Wikipedia page:

"According to Alexa, the English subdomain (en.wikipedia.org) receives approximately 55% of Wikipedia's cumulative traffic, with the remaining split among the other languages (Spanish: 17%, Japanese 4%, German: 4%, Polish: 3%, French: 3%, Portuguese: 2%).^[1] As of December 2007, the five largest language editions are (in order of article count) English, German, French, Polish and Japanese Wikipedias.^[55]"