This is our second post from the Social Media Monitoring – 2010 review series.
In it we’ll be giving you an insight into how we have set up the comaprison of tools (which proved rather a challenge) and the volume of online conversations that each social media monitoring tool was able to uncover.
Setting up the search string
We decided to use Starbucks as a test brand for our social media monitoring because it’s a global brand that is frequently discussed online. Also, the word ‘Starbucks’ doesn’t have any other meaning or use other than being a brand/company name.
As well as tracking the word ‘Starbucks’, we also tracked the phrase ‘Flat White’, a new addition to the Starbucks coffee range which launched in December 2009 . We also tracked their new ready brew coffee, ‘Via’, which was released in the autumn of last year in the US and in March 2010 in the UK. We wanted to see what impact this new development was having on online conversations about the brand . Finally, because Starbucks is associated with its ‘Reward Card’ and the phrase ‘Fair Trade’ we tracked these subjects too. To keep things fair we created a similar search string for each tool.
It is important to note that some tools are capable of more sophisticated search strings than others. So we were testing to the lowest-common-denominator in this sense.
Comparison challenges
Although the tools are very different, we wanted to try and evaluate them all as fairly as possible. Thus the tools were used ‘out-of-the-box’, as they come, for the fairest comparison. Again, there are limitations with this approach. Some of the more sophisiticated options offered in some tools are only relevant to more experienced users. And some providers (e.g. Neilsen) are set up to provide a much greater level of analyst support than, for example more technology focussed firms like Radian6.
Our sense for the market is that most firms are still learning the art of social media monitoring and that tools are often managed day-to-day by people with only limited training in how to use them in anger. This drove our approach to the research.
Coverage
As the tools all have different coverage, whether it’s for different media or markets, we set up the same filters for each tool to create a comparable ‘universe’ of conversations for Starbucks. Our test was carried out using only the English language and for the same time period on each tool.
Sentiment analysis
One of the areas we wanted to test was the sentiment analysis accuracy of each tool. In order to compare the automated sentiment (ie, sentiment that is coded automatically positive or negative by the tool) with our own analysis we had to extract the conversations and manually code them. Some tools don’t allow you to extract certain conversations, others do. Where we weren’t able to extract sentiment for some reason, we’ve marked the tool:
Number of conversations
The seven tools gave very different results when looking at overall conversations - the smallest number of conversations was found by Biz360 and the largest by Radian6 – over 11x the difference! But remember, more conversations is not necessarily better – there is often duplication.
*You can usually make arrangements with your account manager if you need more data.
Conversation types
When you compare the conversations by media type, again each of the tools shows quite a different result:
*Scoutlabs doesn’t allow you to extract Twitter conversations with sentiment. The tool does allow you to browse the latest twitter conversations though.
At this top-level, it’s clear the tools are each doing something quite different…
Next…
More detail on these tests, and the results, can be found in our final report which will be available to download on Friday 16th April. We’re also holding a free social media monitoring breakfast seminar on 15th April in London, where we’ll be presenting the findings of our report, as well as giving practical tips and advice about social media monitoring and the best way to analyse results. You can register for the event by clicking on the button below:
Read the other posts from our social media monitoring review 2010.
