Basque trends, content and user rankings in Twitter: Umap

Three years ago, when Twitter was used just by a few hundred Basque users, we developed Umap, a tool that could gather all content produced in Basque in that network. We also launched other language versions, but after a year working, the project collapsed. There were already more than 5,000 Basque Twitter users and we could not follow more thru Twitter's streaming API.

A rethinking was needed, and that's what we did over the following months. Now, recently, we have recreated the site, and this time we hope it's built over more firm ground.

The basis of the new Umap is not to gather all content Basque in Twitter, but at least, the most interesting one. We don't follow all Basque users in Twitter, but just the most interesting ones, and so we developed a ranking of Basque Twitter users. Among other factors, we measure the percentage of Basque usage, and those that fall below 20% of Basque are discarded from the rank (they can return, as we also have improved our user detection system). For each user, we depict their evolution in the ranks, and we also keep stats of language usage.

Anyway, as in the previous version, we mark language on all incoming tweets from all those users realtime, and we only consider Basque tweets for the front page or for trend detection. Twitter's API has improved in terms of the information they provide, and you can track tweets in a given language to provide it in the metadata, but their system doesn't work with Basque or many other languages.

Moreover, we've improved trend detection, although unlike Twitter's TT system, we only track hashtags. We store them in an archive and using Twitter's new improved search options, we link to that moment of the past when a given term was trendy for Basques. So, on april 7th, the top trends in Basque were #tropela and #ParisRoubaix, both related to cycling, and we can link from there towards Twitter.

So far, the new Umap only works in Basque. We had temporary Welsh and Catalan versions on the old system, but they were as unstable as the Basque one and we were forced to discontinue that effort. I feel personally very sorry for disappointing our welsh friends, but we have also realised that to maintain other language version we need some kind of customer-provider relationship in those other regions. As for Basque, we manage by ourselves, but it's difficult for us to be serious about Welsh without a clear and established alliance in Wales. But nothing's discarded of course. Umap can be interesting in other languages, as well.

And, moreover, the technology and knowledge that we built while we were creating this Basque tool has proven useful for us in other languages. Now, we run a backend technology we call Robsoc (Robotic Social Curation), and we use for distinct purposes in different frontends: Umap is one of those, automatic news gathering is another one (examples in Basque, Spanish and English), and we are also embarked in the Niagarank newssite project, an automatic news aggregator for Spain.