Annie Pettit of Conversition, in a presentation at ESOMAR 3D 2011, set out to demonstrate that you can use social media to prove more people like bank fees than cookies. All you have to do is get the analysis wrong by making some common mistakes.

When designing social media projects, make sure to take care with each of the following:

  1. Data collection processes
  2. Data cleaning processes
  3. Sentiment processes
  4. Sampling processes

To prove that people like bank fees more than cookies, Annie collected data from a random sampling of verbatims mentioning cookies. She went ahead and included references to “browser cookies” and “cookie settings” in her verbatims.

For data cleaning, you can leave in the spam and the porn references. You can include or exclude irrelevant data: programming errors, coupon listings, viral games and secondary product usage.

Automated systems will keep all of this information. So the sentiment analysis gives positives to “free” and other spam words. The lazy researcher might omit some of the records and thereby bias the results. The lazy researcher might accept the sentiment analysis as is rather than tuning it. Failing to correctly code individual words in the dictionary will change the overall sentiment results. Treating product descriptors as words with sentiment will also adjust the ratings.

Different types of sources will produce different results making sampling sources very important. Including or excluding Facebook or Twitter updates will make dramatic differences.

Did Annie prove that bank fees are more popular than cookies? Bad data collection, bad sentiment scoring and poor sampling came close to proving that bank fees are more popular than cookies but apparently cookies are more popular than bank fees even with poor social media research methods!

Author Notes:

Jeffrey Henning

Gravatar Image
Jeffrey Henning, IPC is a professionally certified researcher and has personally conducted over 1,400 survey research projects. Jeffrey is a member of the Insights Association and the American Association of Public Opinion Researchers. In 2012, he was the inaugural winner of the MRA’s Impact award, which “recognizes an industry professional, team or organization that has demonstrated tremendous vision, leadership, and innovation, within the past year, that has led to advances in the marketing research profession.” In 2022, the Insights Association named him an IPC Laureate. Before founding Researchscape in 2012, Jeffrey co-founded Perseus Development Corporation in 1993, which introduced the first web-survey software, and Vovici in 2006, which pioneered the enterprise-feedback management category. A 35-year veteran of the research industry, he began his career as an industry analyst for an Inc. 500 research firm.