If you’re an avid reader of this blog, you know I take a dim view of sentiment analysis, primarily because it’s a fairly shallow measure of your brand’s success. By shallow, I mean sentiment analysis only gives information about positive (or negative) mentions of your brand.
Problems with sentiment analysis
How negative?
While interesting, and potentially creating some value, sentiment analysis doesn’t tell you much about WHY folks feel that way about your brand. Results also don’t say much about HOW positively or negatively people feel about your brand — so real anger gets the same valence as mildly annoyed. Shallow.
Uncovering problems
Sentiment analysis is also a result measure. Not only doesn’t sentiment analysis tell you why people feel good or bad about your brand (although you can do a deep dive into actual conversations to discover this manually), but you’re finding out about the problem AFTER the emotion occurred, when it’s much harder to fix. It would be nice to know there was some problem so you could fix it BEFORE customers started sharing their negative emotions about your brand.
Collecting data
Existing sentiment analysis tools don’t collect EVERY mention of your brand. Most do a fairly good job of social media sites like Facebook and Twitter, but gathering information from blogs and news sites is much more limited. So, you don’t have complete information going into the analysis. That said, I don’t know of ANYTHING capable of collecting every online utterance.
As a manager, you must recognize that the absolute numbers of positive or negative utterances isn’t a true reflection of how folks feel about your brand. So, choosing to ignore a “few” negative comments is dangerous. Those few comments might represent 100′s or 1000′s of similar comments your sentiment analysis tool didn’t find.
Accuracy of sentiment analysis
While I’ve always questioned the accuracy of sentiment analysis results, a comment left on one of my earlier blog posts points to even less accuracy than I’d ever assumed.
Recent experiments suggest sentiment analysis data is LESS accurate than a coin toss (accuracy 50%). That’s really scary if your brand makes strategic decisions based on sentiment analysis.
If you’re a skeptic, like I am, here’s the proof:
Walid Saba is a professional in AI (artificial intelligence) and NLP (natural language processing), the roots of all sentiment analysis software. He conducted an experiment using several tools and found they incorrectly classified content, even content from the tools’ own blogs. This is really chilling and you can read the entire paper here.
Freshminds conducted a similar experiment looking specifically at the ability of top tools like Radian6, Brandwatch, and Sysomos. While the tools accurately predicted between 60 and 80% of utterances, when neutral utterances were removed (80% of the utterances) the accuracy dropped alarmingly.
It’s important to note the Freshminds study dates from 2010, however Dr. Saba’s data is from just last year. Neither “trained” the tools. Training involves a laborious and time-consuming process of correcting predictions using manual coding of utterances so the tool learns to be better at predicting whether an utterance is positive, negative, or neutral. Because of this, many firms use sentiment analysis tools without training, making these experimental results applicable in many contexts.
Speech patterns
Even with extensive training, most tools experience difficulties analyzing utterances to detect sarcasm and other speech patterns that only emerge when utterances are analyzed holistically (in context rather than as isolated word patterns).
Why sentiment analysis continues to grow?
Despite the problems highlighted in this post, the use of sentiment analysis tools continues to grow.
Why, you might ask? For most firms, the answer lies in notions that only an automated tool can possibly analyze the vast amounts of data representing mentions of brands. We commonly hear this expressed as “BIG DATA”. And, certainly, an automated system is the only one capable of analyzing 100′s or 1000′s of mentions of your brand every day. Thus, even a tool that’s slightly accurate looks good when compared with ignoring all these mentions.
What we should be asking ourselves, however, is WHY we need to analyze ALL this data? Hence, requiring automated analysis. When the trade-off is huge inaccuracies and expending huge amounts of time to continually train your tool, is 100% analysis necessary to understand how consumers view your brand?
My answer is a resounding NO. And I offer this option. Wouldn’t it be better to analyze THOROUGHLY (manually) a random sample of utterances? In a recent presentation at the Text Analytics Summit in San Fransisco, I made exactly this case using data from several online sources. My analysis demonstrates the value of manual (computer-assisted analysis using Hyperresearch software, which is an analysis tool rather than a sentiment analysis tool) not only gives a clearer picture of sentiment, but provides a deeper understanding of how consumers view your brand. Companies can use this understanding to identify problems early, before they generate dissatisfaction, uncover opportunities, build advertising that resonates with consumers, and understand deeply the lives lived by their customers. These understanding help the brand strategically must more than simple sentiment analysis.
Need Help
Whether you need a complete social media marketing strategy or some consulting to optimize your existing social media marketing, we can fill your digital marketing funnel or developing a market information system that matches your needs. We can help you do your own social media marketing better or do it for you with our community managers, strategists, and account executives. You can request a FREE introductory meeting or sign up for my email newsletter to learn more about social media marketing.
Be sure to subscribe to our email newsletter, where you’ll get even more insights to make your marketing SIZZLE The Sad State of Sentiment Analysis Hausman Marketing Letter