Radical New Approach to Content Analysis
27th April 2006 by Angie Jeffrey
Here’s something really new: measuring the ABSENCE of core messages to gauge the effect of public relations! Let me share some highlights from a new white paper, “A New Model For Media Content Analysis,” by David Michaelson, David Michaelson & Company, LLC, and Toni Griffin of MetLife, that is available on the IPR Commission for PR Measurement and Evaluation website. While it isn’t necessarily a practical approach for every-day use, it is very compelling intellectually, and should be seriously considered - especially for complex subject matter.
David’s and Toni’s hypothesis is that measurement isn’t funded because of a “lack of interest, lack of knowledge, lack of budget or a generalized fear of measurement; rather, it is a lack of usefulness of these basic research measures.”
To prove their hypothesis, they first analyze the eight commonly-used methods for media measurement today, which include clip counting, circulation and readership analysis, AVE, simple content analysis, message analysis, tonality analysis, prominence anlaysis, quality analysis (single-score indices) and competitive analysis, showing inherent weaknesses in every approach. Most interesting to me is that even scoring messages for tonality fails to offer a “proper situation diagnosis or prescription for a solution that is tied directly to communications objectives.”
Why? They cite two fatal flaws: 1) the absence of an analytic structure that determines the accuracy of coverage overall and the messages contained; and 2) the failure to link analysis to communications objectives and PR messages.
“Message accuracy” is:
” … based on an analysis of four basic elements: the inclusion of basic facts, the inclusion of misstatements about these basic facts, the inclusion of incomplete, deceptive or misleading information that biases the interpretation about basic facts and the ommission of basic facts.”
So, to analyze a story for these factors, you have to study the context of the article, and see if anything critical is missing that may have rendered the story a non-starter.
As for not linking messages to communications objectives, they challenge us to figure out where, exactly, in the Communications Lifecycle our target audience (and the media) may be. You all know this Cycle includes awareness, knowledge, interest and intent. If we shoot messages out there without knowing this, we will miss the target entirely … and thus, our objectives.
Finally, David and Toni share a couple mind-blowing case studies for MetLife that proved their hypothesis … and that lead to a revamping of PR strategy and tactics at MetLife that included in-depth education of the media (which didn’t understand half of MetLife’s subject matter) and a much better targeting of messages. In one of the cases, they found:
” … between 60 and 85% of the articles on the key issues of concern to MetLife included an error in reporting, a misrepresentation of key information or an omission of basic information that should have appeared in the contexts of the articles in question. By concentrating media relations efforts on those reporters and publications where these errors and ommission in reporting occurred, the eventual result was a significant decline in the proportion of articles with either errors or omissions, as well as an overall increase in the number of articles by 45% on the issue at hand.”
Of course, the problem with all of this is you need really smart people to analyze your coverage this way … and therefore, it is more expensive. However, David and Toni challenge this by saying that the return on such an investment is a significantly improved quality of media relations, and greater chance of success against objectives.
Yeah, baby!
April 27th, 2006 at 5:41 pm
[...] IABC Measurement Commons: Radical New Approach to Content Analysis A look at a new white paper from the Institute for PR that’s proposing a “new model for media content analysis” based on the hypothesis that measurements isn’t funded due to the flaw in the way measurement is approached. [...]
April 28th, 2006 at 2:36 am
This is ancient stuff. These measures were embeded in the Media Maesurement Ltd methodolgy when I was its MD in the 1990’s.
The real issues are that evaluation companies hide thier methodologies behind mumbo jumbo, PR practitioners are not taught and do not find time to work at evaluation methodology and academics and peer reviewers are not knowledgable in the area.
The content analysis capabilities of the specialists are as comepetent as any. The application of semantic analysis and inference engines far exceeds the capability of practitioners who would rely on some of the same technologies to keep a plane in the air but not offer to the client becaus its ‘automated’.
Radical! phooey!
April 29th, 2006 at 9:19 am
David - well, this is a surprise! I deal with controversy in measurement all the time, but didn’t expect any on this topic. Good, let’s dig in!
I was not speaking to sophisticated analysis firms that certainly have trained analysts to do this type of work, and doubt the authors of the paper were, either. But I’ve been in the measurement world a long time, and almost all the teaching I’ve heard from the mainstream PR organizations focuses on scoring items that are IN the article as opposed to those that SHOULD HAVE BEEN. I defer to your knowledge that this is not a truly new approach, but it is new to the mainstream.
I will invite Dr. David Michaelson to join in on our discussion. Stay tuned!
April 29th, 2006 at 10:22 am
The materials and approach mentioned by Mr. Philips are unpublished and unavailable based my literature searches. I would appreciate if instead of claiming prior knowledge that he makes this information available.
This paper was peer reviewed by several leading academics and industry practioners, none of whom felt this proposed analytic method was “ancient”.
May 1st, 2006 at 6:44 am
As another long-time communication practitioner, I am with David on this. Content analysis and measurable objectives have been around for a long time and used successfully by many practitioners. It doesn’t necessarily take “really smart people” to analyze coverage using these methods but it does take professionals who, at the beginning of a media campaign, know how to set measurable objectives and determine what messages are to be looked for in resulting media coverage. Message analysis, accuracy of content and even tone have been available since the early 80s. I know because we used them at Western Electric. As for literature searches, I suggest Klaus Krippendorf, “Content Analysis: An Introduction to Its Methodology” (1980).
May 1st, 2006 at 8:19 am
Based on first hand experience with media analysis, I am throwing my lot in with Wilma and David on this topic.
At FCC, we use Carma International (this is not a plug for Carma) to benchmark our media favourability score using a global index. Carma scores the media content by key messages carried, spokespersons quoted, positive, neutral or negative tone, placement of the article in the publication or on the broadcast (i.e.) front page versus tucked away in some unnoticed location), volume, key reporters and key publications (which are targeted in our national media relations strategy. Then, our score is compared to companies worldwide on a global media favourability index.
The measurement is credible among our senior management and is used on our balanced scorecard as a leading measure against corporate reputation. With the media favourability score alongside the other financial and performance indicators, we are able to position communication as a contributing business driver.
May 2nd, 2006 at 8:02 am
Interesting feedback, everyone. But again, I would encourage you to just read the paper. Clearly, I’ve done a lousy job of summarizing the kind of in-depth contextual analysis discribed there. I was late to catch a plane, what can I say?!
Wilma - it’s fun to see you refer to Klaus Krippendorf! He and a couple Harvard programmers built our Ai engine for media analysis.
May 30th, 2006 at 10:23 am
This thread is very interesting to me for one main reason - the assertion that saying something is new when it isn’t somehow disqualifies the information from consideration.
Out here in corporate-land, we still are debating how to measure PR effectiveness — or at least, how to explain it to people who have no idea what we are talking about.
Everyone wants numbers — preferably with the requisite currency designator in place. That’s how we get the misuse of AVE (imo, okay for product stories, but not general reputational matters). It’s even easy to use Web trends and path analysis — story appears in WSJ/FT - track traffic post-story and compare to pre-story; works best for products that can easily be bought online…
But even the content analysis is still descriptive in nature, rather than more outcome oriented… or am I mistaken?
In the paper Angie references, the revolutionary part (again, imo) is that it’s focused on errors - how “Six Sigma” is that!?! You could get someone’s attention saying you’re going to “reduce news media errors and misstatements about our company and what it is doing” as a strategy — and could link it up with market research on reputation. Even without a currency symbol, it’s measureable and impactful, no?
- - take care all
May 30th, 2006 at 11:05 am
God bless you, Sean! Yes — it’s very much about figuring out contextual accuracy as opposed to whether or not any of our “five key message points” got into a story. David Michaelson tried to clarify in a recent email:
The example in the long-term care analysis was that the reporters assumed the readers knew what long-term care insurance was and never explained it in the article. A subsequent consumer survey showed significant confusion by consumers over what the product was and how it worked. Follow up media efforts included a strong emphasis on making sure reporters did not make this presumption in their articles.
Angie