Challenges for PeerIndex, Lessons for Klout

Continuing the discussion in my earlier post, PeerIndex: Rating Authority and Relevancy, let’s look in detail at some of the challenges for PeerIndex and what lessons Klout can learn.

Challenges for PeerIndex

1. Get More Accurate Scores of the Big Guys

I saw several folks yesterday seeing their PeerIndex score as Zero. Apart from that, look at just these examples:

Scoble PeerIndex

JayRosen PeerIndex

Something is clearly not right. These are two of the most influential people in tech and media. These are folks who can make or break a startup, and you better get their scores right if you’re to gain any credibility and leverage their influence.

2. Adapt to Different Content Curation Approaches

Different people use Twitter in different ways. For example, Louis Gray’s sharing is primarily through Google Reader, which is tweeted by @lgstream. Robert Scoble’s content curation is through his Twitter Favorites.

These are influential early-adopters, who consume and filter from a massive information stream, and have hence tweaked their Twitter usage habits to suit their needs. The use of their primary Twitter account is for conversation, while curated content gets a separate, dedicated feed.

PeerIndex probably needs to find a way to incorporate multiple Twitter accounts and Twitter favorites into its ranking.

3. Offline Influence Tracking

This is a tough nut to crack and I’m only reiterating it here for the sake of completeness.

Lessons for Klout

1. Diversify Beyond Twitter

If you’re not leveraging Facebook, you’re yet to capitalize on the social web. The Twitterverse is a significant, but small part of the social web.

2. Remember Your Promises

In January 2010, Klout announced that they will be releasing lists of the top influencers for a new country every week. By August 2010, how many country lists have been published? Three – Brazil, UK, and Germany.

3. Leverage Twitter Lists

For a startup aiming to build definitive influence ranking on Twitter, you would think you can readily follow top influencers by region, topic, etc. from their Twitter account. Here are the only lists Klout has created on Twitter:

Klout Lists

This is a failure to capitalize on and leverage a core Twitter feature.

4. Don’t Sacrifice Functionality For UI

I have said it before and I’ll say it again: If You’re Removing Features, Please Tell Your Users!

In May, Klout launched a revamped site with a new classification system and UI. What was not announced was that you no longer had the ability to view the top influencers in a topic, or see the Klout scores of users in a Twitter List.

It is critical for users to be able to use Klout not just to check scores of people they know, but to aid discoverability, easily create Twitter lists using Klout and so on.

Both these startups are very innovative and doing some great work. These are my thoughts on some of the challenges they face and lessons they can learn.

This entry was posted in Social Web and tagged , , . Bookmark the permalink.
  • http://azeemazhar.com/ azeemazhar

    Thanks Mahendra.
    We did have a small cacheing error which intermittently presented some users scores as zero after they refreshed their profile–we did track and remedy the error within a couple of hours.

    On discerning between bigger names, it's definitely worth getting more feedback from users.

    I think there are a couple of things to note: the authority scores are rank ordered. i.e. 56 is high than 54, but we aren't saying how much higher. (Not because we are being coquettish, but because we don't have a unit of authority — what would that be? a milli-scoble?) So it's not easy for us to claim that how much lower Jay Rosen's 50 is than Henry Blodget's 80 on the the topic of media– we do have some ideas, but we're as open to suggestions at this stage.

    In terms of the top 25 lists we provide, we should do a better job of explaining the task. Out of 100m+ twitter users, out of 1.8m we track, these are people who we have confidence in saying 'these guys are authoritative'

    On that rank ordering, we put Jay Rosen in the top 10 out of 100m people. And the people above him?
    Henry Blodget – publisher extraordinaire of BusinessInsider
    KevGlobal – digital strategist at the Guardian (I admit this looks a bit high to me, sorry Kev!)
    Tim Bradshaw – FT's Media correspondent
    Rafat Ali – founder, PaidContent
    Erick Schonfeld – US blogger
    JD Lasica – US digital media commentator
    David Ffolkenfilk – NPR media editor
    GE Anderson – China specialist
    Dan Gillmor – media professor

    As I mentioned on the video with Scoble, being identified in the top 20 ranks is a pretty strong indicator of how your authority is perceived on the social Web. The problem companies are trying to solve is not a macro-scale one not a precise rank ordering one.
    We do think the rank ordering is interesting, and certainly an interesting tool for exploring how effective we are being. But I'm not sure I would go as far as state that 9th vs 10th is a huge difference.

    We can definitely do a better job at showing the distribution of authority in every topic–which in turn would show that we're putting Jay in the top 25 (actually top 10, as it happens) out of 1.8m people on the topic of media; and in the top 1% (i.e. top 18,000) across all topics. (For comparison, I only make it in the top 90,000 overall, i.e. top 5% ).

    A few challenges:
    Topic definition: we've defined the 'Media' topic as a global media topic (not just a US media topic). Regionalisation of industry topics is definitely a challenge–being based in India you probably take a more global perspective than most of the domestic US market–but in our testing people were frequently surprised to see the odd authority creep in from 'other' markets like China, India or the UK.
    We're open to forking topics more regionally (i.e. having a regional US media topic and a regional Chinese media topic–certainly something to look at). An interesting case we discussed with someone (call them Bob) was in an industrial sector where an unfamiliar name popped in to the top 20 lists, after some investigation the result was that the person was Chinese, writing about Chinese companies in that sector, and Bob's inbound assumption was that 'all authorities would be American, talking about American things' (not his exact words, but a sentiment to make Fukuyama proud.)

    The second issue which you rightly point out — was different behaviours, such as sharing via link feeds or favourites. Scoble i think is moving away from favourites, in favour of RTs (or so I read).

    Louis is a particularly interesting edge case which we have *specifically* discussed. And we decided at this point it is too marginal to do anything with.
    One of the largest problems any link-based analysis has is the cheapness of echoing links via twitterfeed (thanks Mario!)–and if we go back to our principles–we are trying to emphasis connections that impose a cost to the originator.
    We might review this in due course–if it becomes common place. Or if there is a clear user rationale to address it. I have already emailed Louis to get his feedback — and hopefully we'll be able to understand better what the requirements are.

    Thanks for the continued attentions! It is extremely helpful.

  • http://twitter.com/JoeFernandez Joe Fernandez

    Hey Mahendra,

    Appreciate you starting these great conversations about the influence/authority space.

    A couple quick notes about where we are on Klout:

    – We have been pulling data from Facebook and other services and bolstering our analysis for months. We are about to launch a much deeper integration of these services and I think you'll be impressed with the breadth and quality of the analysis we are doing.

    – We did put the country specific Twitter lists on hold. It ended up not being as popular with users as we had hoped so we decided to shelf it for the time being. I would like to get these out though and we may revisit.

    – I would argue that lists in general have not lived up to the hopes that people like you and I (and probably most Twitter employees) had for them. In our analysis of list data we found that most lists across Twitter are followed by only 1 or even 0 people. Even in the screenshot you show the biggest list only has 41 followers. We decided to put this on hold and focus on integrating more services.

    – Yes, we did remove the topic search on Klout. These wasn't something we decided to do just to make our UI look pretty. We actually have these pages built matching our new UI and they look amazing. The problem was that we weren't happy with the quality of our topic results. Instead of focusing on a subset of topics we opened it up to every topic we had data about and this caused the results to be hit and miss. Topic influence has always been at the core of what we are doing here at Klout. We decided to take a step back and make sure we were not doing ourselves a disservice by showing data that was skewed because there weren't enough people talking about any specific random topic we had data on. We should have done better at communicating to our users that this feature would be coming back.

    I agree that discoverability is critical. We want our users to be able to leverage their influence and find value in the influence of others. I think you'll really enjoy the next evolution of Klout!

    Thanks again for these great posts.

  • http://www.skepticgeek.com Mahendra

    Azeem,

    Thanks for the longest comment ever on my blog! :)

    After thinking it through, I realize that you're already pretty much aware of the challenges you face, and at least at this juncture, I don't have any further suggestions.

    Appreciate your taking the time to lay out your thoughts in detail. Will look forward to how PeerIndex evolves to be more useful.

  • http://www.skepticgeek.com Mahendra

    Joe,

    Thanks for the detailed response. I'm glad if my posts provide any value from a user perspective to your team.

    I think the adoption of Twitter Lists has been generally mediocre, but there are several prominent lists that have thousands of followers. I'd argue that those should have been your lists. Maybe I'm missing the deep insight you have from your data.

    I'm happy you are already leveraging Facebook, revamping the Topic approach, and also focusing on Discoverability. Thanks for acknowledging the need for greater transparency.

    Look forward to your next version – keep up the good work!

  • http://www.webtrainingwheels.com Lucy Beer

    May I ask, for what real purpose are such services developed? What are some real life examples of how these could be used practically, beyond boosting egos? If we already know that Scoble, for example, is influential, what’s the need for these numbers? It seems to contribute to the problem you pointed out nicely on Twitter: “”Finding engagement”, “having conversation” r euphemisms for certain “influencers” to show off their following.” Doesn’t having “Klout” contribute to the same issue?

  • http://www.skepticgeek.com Mahendra

    Lucy, agree to some extent, especially with Klout. To be certain, the focus of these services is for marketing and PR folks to identify key targets.

    As I tried to make the point in the above post, PeerIndex’s topic-based approach is clearly different and more useful. It is immensely helpful in people discovery when you want to look up authoritative voices among myriad topics.

    While influencers in tech are most commonly known, those from other topics are not.

  • http://www.webtrainingwheels.com Lucy Beer

    Thanks for your response. Agree, PeerIndex does seem to be more useful. I got a much better understanding after watching the interview you included in your other PeerIndex post. I think my issues are mostly semantic – words like “authority” are a bit of a minefield. In any case, really appreciate your insightful posts – enjoying your blog!

  • http://www.skepticgeek.com Mahendra

    Agree – I prefer to focus on keywords like ‘relevance’ rather than ‘authority’ and ‘influence’ – makes more sense.

    Thanks so much for the kind feedback!

  • Pingback: Klout is a Good Start But We Need More Ingredients for an Influencer Recipe | Lifestream Blog()

  • Pingback: Quora()

  • Pingback: Klout: Is there a service that lists users by Klout score in a given region? - Quora()