How does sentiment work? And how accurate is it, anyway?
She is a regular blogger for Lithium and in the Lithosphere you'll see her as MargaretF. You can follow Margaret on Twitter at @margaretfrancis
We've fielded a lot of great user questions since launch, and the number one area we've fielded them in is sentiment. This may be too much information for some of you, but if you really want the details, read on!
The sentiment feature in the Lithium Social Media Monitoring/Scout Labs application is the ability for the machine to judge whether or not the author of a story is expressing a positive or negative attitude towards a specific word or phrase. For those companies with only a few posts per day that they can judge for themselves, this feature is a nice to have. But for brand and product marketers looking at a significant volume of posts, this feature is essential to understanding changes in consumer opinion.
So how do we do it? How accurate is it? And how should you use it?
How we do it
Our sentiment is "entity specific". What some products do when they produce "machine generated sentiment" is that they count happy words vs. sad words in a news article. The "tone" of the article is shown by the happy word count. Consider "I love baseball. My happiest memories in life are from sitting in the bleachers at Fenway. It's the greatest game on earth. But guys like Bonds and A-Rod are bringing it down." Despite the high "happy" word count, this does not express a positive opinion about Barry Bonds or Alex Rodriguez.
In the Lithium Social Media Monitoring application, we don't count happy words. We evaluate sentiment for each particular word or phrase you search for. We can tell that the sentiment for baseball is positive but negative for Bonds and Rodriguez. This is done via part of speech tagging: parsing the underlying semantic structure of a sentence and determining which emotion words apply to the key word. Emotion words come from dictionaries of standard English words and have been augmented with phrases and slang to better map to the world of social media. So Scout Labs' sentiment is entity-specific, which is very important.
Lithium SMM sentiment can be changed by users. We use confidence intervals to decide whether something is positive or negative, but if we get it wrong (more on that below), you can change the score, immediately updating that item for yourself and the rest of your team. Charts and graphs update immediately as well. And the really cool part is that every time a user changes a sentiment value, that item becomes a labeled piece of data that we can use to abstract out additional rules and add words and phrases for our dictionary. So our ability to detect sentiment just gets better over time.
Lithium SMM can "backfill" sentiment data for the previous 3 months in less than a day. We have 6 months of live data in our app for our users . We can go backward and score all the posts from the last 3 months in less than 24 hours. So you will have complete sentiment trend for everything going forward and going backward within less than a day from creating a search (All other graphs -- buzz, share of voice, etc.) are real time and have no lag time at all).
Does it work?
Yes. We have done extensive human vs. machine testing and it's accurate in the 70-80% range, meaning our algorithm agrees with humans' scores 70-80% of the time. This is only slightly less than humans agree with each other. Some other insights and findings from our testing:
- College educated people with business experience agree on the sentiment ratings for a blog post about 85% of the time. Using less qualified people, such as you might find in a random Mechanical Turk experiment, produces lower rates of agreement. We were surprised that we couldn't get that rate higher. Some of the discrepancy stems from the human tendency to equate negative opinions and negative information: "I hate Coke" is a negative opinion; "Merrill Lynch just downgraded Coca-Cola" is negative information.
- The Lithium SMM sentiment feature agrees with college educated people about 75% of the time. We try to pad that a little by being conservative about what we call positive or negative -- we call things neutral if they're borderline.
- The Lithium SMM sentiment feature sucks at detecting irony and sarcasm. Posts that are heavy on the irony often end up classed as "neutral" because the machine can't even guess. Consider "Another winner from the almighty Microsoft." That's a tough one.
- Machines don't understand business context. Perhaps you work for Apple and every mention of an unlocked iPhone is negative because people shouldn't unlock their iPhones. An algorithm that uses grammar and vocabulary based rules cannot classify this post as negative about iPhone: "I love my iphone. My boyfriend unlocked it for me last night."
So the Sentiment feature produces a pretty good guess, about what you'd get using if you got a half dozen ratings from Mechanical Turk and chose the rating the most humans agreed on. (See this useful paper from the Dolores Labs blog about how to use Mechanical Turk to get reliable human judgments). And our best guess plus your teams' efforts to quickly change the things we miss or get wrong means really high accuracy levels for you and your team with a minimum amount of work and expense.
How you should use the sentiment feature:
- To find the top positive and negative posts. Click on "Sentiment" and filter for positive or negative posts. You'll get immediate insight into some forceful opinions about what is wrong -- or what is right -- about the product or brand you are searching for.
- As a starting point for your own sentiment analysis. Any user can change the sentiment rating for any post. If you work for Apple and you want all those unlocked iPhone posts marked "negative," you can do that. Just click on the sentiment icon and make the change. These changes will carry through to all graph data, so you can create accurate data sets to view in the application or export data for. We use your rating changes as machine learning inputs, but your specific ratings are proprietary and confidential to your workspace.
- To get insight into consumer opinion via alerts. When you set up a daily, weekly or monthly alert for a search, you'll get buzz, top news, new words, recent tweets, and the top positive and negative posts pushed to your inbox via a text email. It's a great way to stay informed and know when to invest more attention.
- To compare sentiment between brands or products. Do consumers like Symantec or Norton? The Lebron 6 or the KD1? Embarq or Comcast? Sentiment Trend graphs can help you see trends, spikes, and make comparisons.
We have heard over and over again from our users that an affordable, reliable way to assess sentiment, with user override built in, is critical to getting insight into social media, so we continue to work on this feature. We hope you'll let us know how you want it to evolve in the future.
We've already got a slew of new feature requests to work on, including more metrics, visualizations, and customizations. Get your ideas into the mix at the Social Media Monitoring product page at Lithium.com
