It’s pretty amazing how many research tools (social media monitoring in particular) attempt to charm users with meaningless graphical overlays.
When I say meaningless, I’m referring to:
Share of Voice Metrics Based on Porous Datasets
Share of voice is an empty gesture if the underlying dataset is missing a substantial volume of the relevant authors and sources that cover a given topic. If a search for “virtualization” turns up a result set that’s missing scores of relevant blogs and publications, whatever pretty share of voice chart that ensues is useless.
Sentiment Analysis
This is the one that I find particularly objectionable. There are some extreme challenges in natural language processing that have yet to be conquered to make the margins of error even remotely acceptable for sentiment analysis in terms of raw text / tech news analysis. Even processing a large set of unstructured data and determining what the theme is can be extremely difficult, with how many words in the English language have different meanings in different contexts. The idea that you can slap some semantic foo on top of a huge volume of clips and determine which of them are positive or negative in tone is outrageous, and every single representation of sentiment analysis that I’ve seen applied to a substantive dataset of tech articles (and determining whether a specific vendor or product was mentioned in a positive or favorable light) has fallen short when drilled down.
Who buys that crap? I’m guessing the same type of companies that leave real research work to interns … who consider a keen insight of the landscape too “low level” and “not strategic.” IMHO (and in the opinion of the folks that sign up for ITDatabase), the killer app for interpreting news is still the human brain, which turns out to be incredibly efficient when it’s fed truly comprehensive datasets on whatever category of tech news it needs to disseminate.
Can either Share of Voice or Sentiment Analysis be pulled off effectively? I’m sure. I just have yet to see a solution that provides either share of voice or sentiment analysis that both #1- has a comprehensive / accurate dataset and #2- if there is semantic foo / taxonomies under the hood, the results aren’t extremely skewed by false positives.
Posted by Barbara French
January 13, 2010 at 12:39 pm
Agree on the role of human intelligence. Problem is, human intelligence is very expensive and difficult to scale. So the people analyzing sentiment and share of voice under a blanket retainer service tend to be sourced at a low cost, usually off shore and not native speakers in all of the languages they’re evaluating for sentiment and share. Those are significant hurdles for any human-powered service.
I can understand your position that your services are more accurate than contemporary AI and analytics tools without human auditing.
However, take care in implying that human-powered services, by nature, trump contemporary AI tech and analytics tools.
Posted by admin
January 13, 2010 at 1:01 pm
Thanks for the perspective, Barbara.
My point was really more buyer beware of services that provide a false sense of truth, because there are many of them being sold out there today. Not to say that there aren’t some incredibly accurate analytics that push the boundaries of AI and have some very compelling / righteous stats. But any system is only as good as the underlying data, and the metadata that powers the charts and graphs – and most of the tools I see in this category take shortcuts on the data and it undermines these overlays.
Human powered context ALWAYS trumps AI when those humans have real context in the industry. Big Blue struggled to beat Kasparov on a chess board (which has a finite number of boxes and possibilities). Natural language is infinitely more varied than a chess board. A computer can only do what it is literally told to do by a human being – even the most sophisticated algorithms rely on a human (with the necessary context) to steer it.
The idea that machines can be pointed at vast sets of articles and content and provide the kind of deep insights and context that a human being with experience in the given area can – it’s misleading at best, and in many cases with a lot of the monitoring tools today, it’s truly snake oil.
And when it comes to scale – nothing is less scalable than people paying a lot of money for services and having to double and triple check the underlying data to make sure that the graphs are reliable / accurate. I talk with folks all the time who lament about this. They shell out big bucks for solutions with fancy auto-generated reports on share of voice and sentiment analysis, only to spend time cleaning them up.
Posted by Barbara French
January 13, 2010 at 1:09 pm
You bet. It all rides on the particular humans in the equation — both the client and the service provider sides of the equation.
Posted by BillBo
June 8, 2010 at 6:26 am
Most people and agencies purchasing monitoring solutions simply want to automate the charting process. They REALLY don’t care whether the results are accurate or not. They just need to turn in the report card.
And if you doubt it, name a company that USES their charts for anything else. You can’t use Share of Voice or Sentiment to plan or change anything.
Posted by Travis Van
June 8, 2010 at 9:38 am
I agree, BillBo. But what a sad fact.
I had a demo with a large tech vendor once where one of their senior marketing people told me that they knew the data that powered their charts was severely off – but that no one (that they shared them with internally) paid attention to them anyway.
What if sales people had the same lazy attitude about their quotas, or people in payroll being +/- a few hundred bucks here or there on people’s checks?
I actually do not believe that sentiment is a very useful metric – I agree with you on that one. It’s so fuzzy, and so after the fact that it’s disseminated, it is truly just a “point of interest” that is rarely parlayed into any sort of meaningful follow up result. If a company gets stung by the press, they are aware of the pain well before some useless sentiment analysis charts are whipped together.
I disagree about the uselessness of share of voice however. If you have a product category like netbooks and you are one of the top five vendors in that category, it is VERY useful / actionable to be able to see where your product stands month-by-month relative to your competitors. You don’t think that high profile placements in places your customers are trolling for input on purchases (like Gizmodo, Engadget, PC Magazine, etc.) move the needle on sales? If you are aware of share of voice, you are also aware of where your competitors are getting action and you are not – and you can absolutely spend the necessary energies to get traction in same.