Sentiment Analysis Software vs. Human Analysts for Media Measurement

Automated sentiment analysis software that grades news articles and social media posts as positive, negative or neutral is becoming more popular in media measurement for public relations and marketing. Investors and stock analysts also use the automated sentiment analysis programs to analyze perceptions about the overall stock market or a particular stock. Political campaigns rely on sentiment analysis to evaluate public reaction to candidates’ positions on issues.

Sentiment Analysis Benefits

In rating public perception of companies and brands, sentiment analysis offers many benefits. Analyzing trends in consumer sentiment helps brands gauge the impact of their PR and marketing campaigns. Declining sentiment can signal the need to revamp the product or its PR and marketing strategy.

Automated sentiment analysis is less expensive than paying human analysts to examine every news and social media mention. Large companies that have many thousands of media mentions per month are especially fond of the cost savings. But here’s the problem. Although the software-based sentiment analysis has improved substantially in recent years, analysis of individual articles and posts is often inaccurate – or totally wrong.

Shortcomings of Automated Sentiment Analysis

One problem is that automated programs cannot place comments in context and often fail to grasp nuances of the human language, notably sarcasm.  Social media can be especially challenging for automated sentiment analysis software. Slang can evolve quickly on social media, and syntax on social media is different from conventional conversational style or standard journalism style.

The very best of the automated sentiment software applications provide reasonably accurate analysis – but many produce woeful results out of the box. Lack of precision seriously undermines the validity of aggregated media measurement data and findings.

Customers should be careful in evaluating and purchasing sentiment analysis software. An effective evaluation method is to compare results produced by each application against the results produced by human analysts. This assessment can often be done cost-effectively by supplying the potential vendors with clips previously evaluated by human analysts.

Initial assessment of automated software results may be disappointing. That’s because most software applications for sentiment analysis do not achieve acceptable levels of accuracy out of the box. They require intensive setup, training and tuning for each given customer in order to achieve satisfactory levels of accuracy. They also require ongoing training as the media content about the customer and its brands evolves. In most cases, investing in tuning and training the software improves accuracy substantially.

The Human Solution

Most experts in media measurement favor combining automated sentiment analysis programs with evaluation of the media mentions by human analysts.

“It takes significant investments of time, people, and effort, but if you want truly accurate sentiment analysis in your gathered public opinion data, it’s the only way to go for now,” writes Christopher S. Penn, Shift Communications vice president, marketing technology, in his blog.

Experts attending the recent Sentiment Analysis Symposium favored using a combination of software and humans for sentiment analysis, according to AdWeek. “Machines do analytics, humans do analysis,” remarked Anjali Lai, an analyst at Forrester Research, as she advocated for a hybrid approach.

Humans can better understand context, slang and sarcasm. Although not always perfect, human analysts are trained to recognize language nuances and idiosyncrasies that trip up automated programs. Humans are also less literal than machines and can better tease out meaning.

CyberAlert LLC, the global media monitoring and measurement service for PR and marketing, has long favored the combination of software and human analysts to assess sentiment of news articles and social media posts. “Sophisticated sentiment analysis systems with appropriate training can be accurate for sentiment, but sentiment is only be a part of a thorough media analysis,” said Doug Chapin, president and CEO of CyberAlert. “In addition to sentiment, media analysis requires evaluation of subject, positioning, messaging, issue and other factors of importance to the client. No software today does all that. That’s why at CyberAlert we use a hybrid approach of software for quantitative analysis and sentiment combined with well-trained human analysts for qualitative assessment.”

Instead of relying solely on sentiment analysis software to control costs, large companies with thousands of media mentions can control costs of human analysis by using proper sampling of posts or focusing the human analysis on the most influential authors or sources, Chapin suggested.

Katie Paine, the noted PR measurement expert, emphasizes the use of trained analysts and well-developed analysis protocols to assure accurate results. “A well-established methodology and well-trained analysts are essential to assure accurate analysis of both news and social media,” Paine stated. The Institute for Public Relations (IPR) Measurement Commission is now conducting a study on evaluation standards for social media measurement. CyberAlert is supplying its trained analysts for the study.

Experts at the Sentiment Analysis Symposium also advised:

• Conducting both sentiment analysis and consumer surveys to make up for the shortfalls of both research strategies. Surveys only report respondent’s feelings at a certain point in time.
• Keeping an open mind about results, as opposed to seeking data that supports a preconceived notion. Waiting to review all the data can lead to unexpected and valuable insights, such as new uses for products.
• Seeking to use social media data for predictions rather than just retrospective analysis.
• Attempting to involve top-level executives to obtain increased resources necessary to take advantage of the full benefits of social media analytics.

Other tips for using the programs include:

• Have human analysts double-check results of automated sentiment analysis programs. While that’s more time consuming and expensive, it delivers the most accurate measurement results.
• Employing reviewers to check only positive and negative results can improve efficiency.
• Human reviews of a random sample of a large number of mentions can be just as accurate as checking all mentions.

For smaller organizations with a manageable number of social media mentions, human sentiment analysis alone is likely to deliver accurate yet cost-effective measurement results.

Bottom Line: While automated sentiment analysis of social media mentions offers the benefits of faster turnaround and potentially lower costs, the accuracy of the automated analysis often falls short of levels needed to produce valid results. One solution is to combine automated sentiment analysis with selective reviews by trained human analysts or to rely solely on human analysis for smaller numbers of media mentions.

This article was originally published on the CyberAlert blog.