Annie Pettit of Conversition, in a presentation at ESOMAR 3D 2011, set out to demonstrate that you can use social media to prove more people like bank fees than cookies. All you have to do is get the analysis wrong by making some common mistakes.
When designing social media projects, make sure to take care with each of the following:
- Data collection processes
- Data cleaning processes
- Sentiment processes
- Sampling processes
To prove that people like bank fees more than cookies, Annie collected data from a random sampling of verbatims mentioning cookies. She went ahead and included references to “browser cookies” and “cookie settings” in her verbatims.
For data cleaning, you can leave in the spam and the porn references. You can include or exclude irrelevant data: programming errors, coupon listings, viral games and secondary product usage.
Automated systems will keep all of this information. So the sentiment analysis gives positives to “free” and other spam words. The lazy researcher might omit some of the records and thereby bias the results. The lazy researcher might accept the sentiment analysis as is rather than tuning it. Failing to correctly code individual words in the dictionary will change the overall sentiment results. Treating product descriptors as words with sentiment will also adjust the ratings.
Different types of sources will produce different results making sampling sources very important. Including or excluding Facebook or Twitter updates will make dramatic differences.
Did Annie prove that bank fees are more popular than cookies? Bad data collection, bad sentiment scoring and poor sampling came close to proving that bank fees are more popular than cookies but apparently cookies are more popular than bank fees even with poor social media research methods!
Author Notes:
Jeffrey Henning
