How Accurate is Social Media Sentiment Analysis in Measuring Consumer Opinion?

Many companies use social media measurement to gauge consumer sentiment toward their brands. Rating social media comments on a positive to negative scale indicates how consumers feel about company products, the company’s response to current news, or the seriousness of a recent PR mishap.

New research says brands should not rely on social media measurement alone to rate consumer sentiment. Research by Engagement Labs reveals that social media sentiment often differs from off-line, or “real world,” sentiment. Although some brands perform well both online and offline, most perform well in only one channel or the other.

How Online and Offline Audiences Differ

Social media users tend to be younger (no surprise) and more female that overall audiences, Engagement Labs’ Chief Commercial Officer Brad Fay explains to Marketing Land. They also have different motivations. Social media commenters often use brands to promote themselves.

Specific product categories tend to be discussed more online, Fay says. Offline discussions tend to focus on consumer packaged goods and food; online discussions more on media, sports, automotive and technology topics.

Social media comments tend to be more extreme. “People often have stronger opinions on social media. Offline people with modulated opinions tend to be less active on social media,” Fay says.

Researchers used surveys and social media monitoring to examine correlations between social media and offline discussions surrounding 500 brands over the course of a year. They analyzed four key metrics — volume, sentiment, sharing and influence — to measure correlations.

Examples of Brand Sentiment Analysis

When Dick’s Sporting Goods stopped selling assault-style rifles following the Parkland school shootings, social media conversation about its decision were largely negative, but offline sentiment was more positive.

When a Starbucks employee called police to arrest two black men in a Philadelphia store, initial social media comments were mostly critical and negative. Offline the conversation saw little impact.

“In practice, marketers should assume the online and offline conversation channels work independently of each other, and they should formulate strategies and tactics optimized to the strengths of each,” researchers conclude. “This is not to say that marketers who are successful in social media cannot apply lessons learned to offline conversation, and vice versa. One should not be treated as a substitute for the other, however.”

Advantages of Social Media Measurement

Marketers and researchers are likely to continue measuring social media sentiment. Although social media sentiment may not always replicate overall sentiment, social media measurement provides clear benefits:

Speed. Social media analytics can deliver results in hours or days, as opposed to weeks or months for conventional research methods. That’s critical during a PR crisis and can provide an early warning of an emerging crisis.

Affordability. Social media measurement can produce reports at a fraction of the cost of conventional research methods.

Accuracy. It can be more accurate in some instances. Traditional market research – surveys, focus groups and in-depth interviews —reflects feelings of a small group of people at one point in time.

With traditional opinion research, people may not tell researchers what they really believe. They may say what they think they should believe or what they think researchers want to hear. They may not even know what they think at the time. Researchers may ask the wrong questions, speak to the wrong groups or interpret what they say incorrectly. Because information gathered through social media monitoring is unprompted, it’s more likely to reveal customers’ true feelings.

Analyzing social media for brand sentiment will likely become more accurate over time as more people embrace social media and automated sentiment analysis becomes more advanced. While automated analysis has improved, it can still produce errors. Measurement services that combine automated programs with trained human analysts provide more accurate results.

Bottom Line: Marketers should not assume that sentiment toward their brands on social media is identical to offline or overall sentiment. Consumers on social media have different demographics and different motivations from those offline. Experts recommend marketers take advantage of the benefits of social media analytics but augment it with conventional, offline marketing research.