How to Resolve Shortcomings of Automated Sentiment Analysis Software

Sentiment analysis software that grades news articles and social media posts as positive, negative or neutral has become more popular in public relations, marketing, and competitive intelligence. Companies use social media sentiment analysis to gauge changes in sentiment toward their brands and products. By rating public perception of companies and brands, sentiment analysis offers tangible benefits. Analyzing trends in consumer sentiment helps brands gauge the ROI of their PR and marketing campaigns. Declining sentiment can signal the need to revamp the product or its PR and marketing strategy.

Investors and stock analysts also use the automated sentiment analysis programs to analyze perceptions about the overall stock market or a particular stock. Political campaigns rely on sentiment analysis to evaluate public reaction to candidates’ positions.

Benefits of Automated Sentiment Analysis

Automated sentiment analysis is less expensive than paying human analysts to examine every news and social media mention. Large companies, those with more than 5,000 or more mentions a month, savor the time and cost savings the automated software provides. The engines are consistent and not influenced by biases like human reviewers may be. In addition, because the social media analytics programs can shift through massive amounts of content quickly, they produce reports much faster than traditional marketing research such as focus groups and surveys.

Shortcomings of Social Media Sentiment Analysis

But here’s the problem. Although the software-based sentiment analysis has improved substantially in recent years, analysis of individual articles and posts is often inaccurate – or totally wrong, skewing aggregated results quite substantially. That’s why automated sentiment analysis poses one of the most difficult challenges in PR measurement.

One problem is that automated programs cannot place comments in context and often fail to grasp nuances of the human language, notably sarcasm. Text-based sentiment analysis doesn’t evaluate memes or graphics that appear with the article.

Social media sentiment analysis can be especially challenging for the software. Slang can evolve quickly on social media, and syntax on social media differs from conventional conversational style or standard journalism style.

The very best of the automated sentiment software applications provide reasonably accurate analysis – but many produce woeful results out of the box. The software needs to be “tuned” to gain even a passable level of accuracy. Lack of precision seriously undermines the validity of aggregated media measurement data and findings.

Fake robot, or “bot,” accounts are another issue. Even though Twitter, Facebook and Instagram have purged large numbers of fake accounts, the bots remain pervasive on the platforms. An automated analysis will likely evaluate mentions produced by fake accounts, skewing their results.

In addition, research has found that online sentiment may differ from off-line or “real world” sentiment. Social media users tend be younger and more female than the general population. Their comments tend to be more extreme, and they may have different motivations. Social media commenters often use brands to promote themselves.

Compare Software to Humans

Organizations should be careful in evaluating and purchasing sentiment analysis software. To evaluate the programs, potential customers can compare results produced by each application against the results produced by human analysts. This assessment can often be done cost-effectively by supplying potential media measurement vendors with clips previously evaluated by human analysts.

Initial assessment of automated software results may be disappointing. That’s because most software applications for sentiment analysis do not achieve acceptable levels of accuracy out of the box. They require intensive setup, training and tuning for each given customer or set of search queries in order to achieve satisfactory levels of accuracy. They also require ongoing training as the media content about the customer and its brands evolves. In most cases, investing in tuning and training the software improves accuracy substantially. But that software training increases the cost, sometimes substantially.

A single version of sentiment analysis software may not be equally effective for different companies. Sentiment analysis can differ from industry to industry and company to company. For instance, sentiment analysis software typically identifies “fraud” as a negative term, but a company offering anti-fraud protection software may view term “fraud” as positive or neutral in most every case.

Human Solution for Automated Sentiment Analysis

Most experts in media measurement favor combining automated sentiment analysis programs with evaluation of the media mentions by human analysts.

“It takes significant investments of time, people, and effort, but if you want truly accurate sentiment analysis in your gathered public opinion data, it’s the only way to go for now,” wrote Christopher S. Penn, when he served as a VP at Shift Communications.  He is now associated with Trust Insights and publishes blog posts at ChristopherSPenn.com.

Automated sentiment analysis typically views keywords as proxies for sentiment, Penn explains. Words such as like, love, enjoy indicate positive sentiment. Hate, angry and upset signify negative sentiment. So the software would report as positive the comment: “Oh, yeah, Fast Food Restaurant. I just LOVE the 30 minute wait for my food.”

Humans can better understand context, slang and sarcasm. Although not always perfect, human analysts are trained to recognize language nuances and idiosyncrasies that trip up automated programs. Humans are also less literal than machines and can better tease out meaning. They can also tease out meaning from the illustrations, photography or memes in an article.

A Hybrid Approach

Glean.info, formerly CyberAlert LLC, the global media monitoring and measurement service for PR and marketing, has long favored the combination of software and human analysts to assess sentiment of news articles and social media posts.

Sophisticated sentiment analysis systems with appropriate training can be accurate for sentiment, but sentiment is only a part of a thorough media analysis. In addition to sentiment, media analysis requires evaluation of subject, positioning, messaging, spokespersons, issues and other factors the client considers important.

No software today does all that. That’s why a hybrid approach of software for quantitative analysis and sentiment combined with well-trained human analysts for qualitative assessment offers the best solution.

Large companies with thousands of media mentions often control costs by relying only on automated sentiment analysis.  But they can also control costs if human analysts use a random sample of posts or examine posts from the most influential authors or sources.

The Best Sentiment Analysis Solution

Experts attending the recent Sentiment Analysis Symposium agreed that combining software and humans often offers the best sentiment analysis solution, AdWeek reported. “Machines do analytics, humans do analysis,” remarked Anjali Lai, an analyst at Forrester Research, as she advocated for a hybrid approach.

Experts at the Sentiment Analysis Symposium also advised:

• Conducting both sentiment analysis and consumer surveys to make up for the shortfalls of both research strategies. Surveys only report respondent’s feelings at a certain point in time.
• Keeping an open mind about results, as opposed to seeking data that supports a preconceived notion. Waiting to review all the data can lead to unexpected and valuable insights, such as new uses for products.
• Seeking to use social media data for predictions rather than just retrospective analysis.
• Attempting to involve top-level executives to obtain increased resources necessary to take advantage of the full benefits of social media analytics.

Other tips for using the programs include:

• Have human analysts double-check results of automated sentiment analysis programs. While that’s more time consuming and expensive, it delivers the most accurate measurement results.
• Employing reviewers to check only positive and negative results can improve efficiency.
• Human reviews of a valid or random sample of a large number of mentions can be just as accurate as checking all mentions.

For smaller organizations with a manageable number of social media mentions, human sentiment analysis alone is likely to deliver accurate yet cost-effective measurement results.

Bottom Line: While automated sentiment analysis of social media mentions offers the benefits of faster turnaround and potentially lower costs, the accuracy of the automated analysis often falls short of levels needed to produce valid results, insights and decisions. One solution is to combine automated sentiment analysis with selective reviews by trained human analysts or to rely solely on human analysis for smaller numbers of media mentions.

This article was first published on July 30, 2015, and updated on Nov. 19, 2018.